Competitor Content Tracker

Competitor content tracker — marketing automation on custom code that monitors selected competitors' publications and delivers a concise weekly report on topic coverage. The tool replaces manual review of 10–20 websites and closed newsletters with a single structured digest. Uncovered topics become visible — a direct source of ideas for your own content plan.

The AI agent workflow consists of six steps:

1. The scheduler triggers data collection on a schedule — once a day for fast-updating sources or once a week for long-form formats.
2. The AI agent crawls the list of sources: competitor blogs, LinkedIn pages and posts, YouTube channels, podcast feeds, email newsletters, knowledge bases.
3. For each piece of content found, the title, publication date, main thesis, format (article, video, short post), and key facts are extracted.
4. The AI agent groups publications by topic, determines the frequency of each topic in the sample, and flags atypical formats.
5. Results are compared against the archive of previous weeks — new topics, persistent trends, and topics competitors have stopped covering are highlighted.
6. The finished digest is delivered to Slack, a CMS, or email with sections: 'write often', 'write rarely', 'new this week', 'dropped from the agenda'.

What the competitor content tracker does not do:

• Does not write content for you. The AI agent builds a backlog of ideas, but final articles, videos, and scripts are created by the editorial team.
• Does not track closed sources. Publications behind a paywall, private Telegram chats, and personal DMs remain out of scope — the agent works only with the open web.
• Does not evaluate the quality of competitors' content. The agent counts frequencies and identifies topics; the subjective assessment of 'strong / weak / good production' is made by a human.

The digest contains 15–30 cards per week and takes 10–15 minutes to read. The editor tags interesting topics in a CMS, Notion, or Linear, and they automatically enter the content plan backlog. Market knowledge stops living in one person's head — it sits in a shared repository and is accessible to the entire team.

The tracker's technical stack is built on a combination of a scraper, LLM summarizer, and notification router. Custom code is required because each source has its own structure: RSS, HTML blog, social media API, YouTube transcript — a universal tool cannot handle this without additional logic.

The data flow looks like this:

1. The scheduler (cron, Vercel Cron, workflow engine timer) initiates the run.
2. The scraper module reads a pre-configured list of sources. For blogs — RSS with fallback to HTML parsing; for LinkedIn — the official API or a verified connector; for YouTube — Data API and transcripts via Whisper; for podcasts — RSS feeds and transcription.
3. New items (by published_at and identifier) are stored in the database — Postgres or Supabase — with raw text and metadata.
4. An AI agent running on an AI model processes batches and generates a card for each item: topic, thesis (2–3 sentences), format, key facts, tags.
5. A second summarization pass aggregates the week's cards: grouping by topic, frequency counting, highlighting unusual formats, comparison with previous periods.
6. The notification router sends the finished digest to the marketing Slack channel, duplicates it to email, and (optionally) creates a draft in the CMS with the tag «competitor-digest».
7. An editor reads the digest and flags topics that go into the backlog. Flags feed back into the database as a signal — the agent learns which topics matter specifically to you.

Typical configuration options

• Minimum (2 weeks): 5–8 sources, RSS + HTML, one digest per week in Slack, no archive.
• Medium: 10–15 sources with different content types, daily collection, weekly digest + daily alerts on key topics, archive in Postgres.
• Extended: 20+ sources, including YouTube transcripts and podcasts; integration with CMS (Payload, Contentful) for auto-creating topic drafts; semantic search across the archive via pgvector.

Alternative approaches

• Ready-made media monitoring tools (BrandMentions, Brand24) cover brand mentions but handle content marketing topic analysis poorly.
• Feedly + manual summarization — cheaper to start, but runs into 3–5 hours of content marketer work per week and loses scale beyond 10 sources.
• A flow orchestrator without custom code works for RSS sources but breaks on LinkedIn, YouTube, and non-standard sites — which is why the final solution is built on custom-code.

Security and compliance

The agent accesses only public pages and official APIs. Scraping respects robots.txt and rate limits. Competitor data is stored as quotes with source attribution — this minimizes legal risks and leaves a trail for fact-check.

To launch a competitor content tracker, you need a minimal set of inputs, access credentials, and internal team agreements.

Data and access:

• An agreed-upon list of 5–20 competitors with URLs of their main channels (blog, LinkedIn page, YouTube, podcast feed).
• A Slack workspace or email sender where digests will be delivered.
• A database for the archive — Postgres, Supabase, or equivalent. For a simple setup, a managed instance from Supabase is sufficient.
• Access to an AI model API or another LLM for summarization.
• For LinkedIn and YouTube — separate API keys and agreed-upon quotas.

Team readiness:

• A marketing lead or content editor who defines the list of sources and decides which topics go into the backlog. Approximately 1–2 hours per week to work with the digest.
• An engineer (in-house or contractor) familiar with Python or Node.js, to set up scrapers and deploy the pipeline. At least 20–30 hours for the MVP.
• An agreement on scope: which sources count as competitors, what to do with friendly resources, which topics are out of scope.

Potential pitfalls

• Sources that block scrapers or frequently change their layout — budget 10–15% of your time for fixes.
• Unlabeled videos and podcasts without transcripts require an additional step with Whisper and may increase the cost of LLM runs.

Launch timeline: 2–4 weeks for a basic setup with 5–10 sources and a weekly digest. Adding complexity — YouTube transcripts, CMS integration, semantic search over the archive — adds another 2–4 weeks.