Norm, the night shift

Nightly research and grooming · Layer 4

"Reads journals all night."

What Norm does

A knowledge base goes stale when nobody feeds it. Norm runs overnight, looks for new material on the topics you configured, and files it so your answers are current by morning.

Scheduled runs

Runs on a nightly schedule or on demand through the API. Only one run is in flight at a time.

arXiv and Semantic Scholar

Each topic can switch either source on, with its own search terms, arXiv categories and direct URLs.

Topics are config

Adding a topic means editing config: seeds, sources, on or off. No code change.

Fresh material, same collections

New papers are chunked, embedded and added to the collections that Annie retrieves from.

Job history

Every run is recorded as a job in a local database, so you can see what ran, when, and how it ended.

Backlog grooming

The same nightly pipeline also grooms the ticket backlog.

How it works

Each run follows the same five phases, in order.

  1. Discover new papers from the sources each topic enables.
  2. Filter by topic relevance and skip anything already fetched.
  3. Extract and chunk the content.
  4. Embed and index it into the collection.
  5. Verify the result and write a run summary.

Where it fits in the four layers

1 Infrastructure & roles Vera, Grant, the bus, the device registry
2 Data pipeline & state Flo, DynaRAG, the carousel
3 Reasoning & inference Rick, Annie, Max, Hank
4 UI & collaboration Marco, Noted, Norm, the graph

Norm is a layer 4 worker. It writes into the collections that layer 3 retrieves from, and its runs show up as jobs you can inspect in the web app.