News Pipeline

GDELT publishes machine-readable article files every minute. The pipeline consumes them in two modes off one code path: a realtime poller working a rolling window, and a historical backfill that walks a date range backwards and checkpoints as it goes, so a crash resumes exactly where it stopped.

Everything after fetch is shared — parse and country enrichment, then two-stage deduplication that drops exact URL matches before paying to embed anything and near-duplicates after, then batched embedding into sharded vector indexes. What used to be keyword hunting is now search by meaning.

A Python client library on top made the store self-service for other internal teams: semantic search with date, country, language and source filters, streaming bulk export for large pulls, and an agentic mode that answers questions with citations.

I own pipeline reliability end to end: observability and alerting, so failures surface instead of passing silently.