Architecture

bAInquet is a modular-monolith API on Koa, a BullMQ worker pipeline, a set of exporter modules, object storage fronted by a CDN, and an MCP read server. This page describes the system shape and the data flow, and marks which pipeline stages are real versus stubbed today.

The system shape

                         +---------------------------+
  connectors  --HMAC-->  |  Koa API (modular         |
  (push side)            |  monolith, 12 modules)    |
                         |  auth, organization,      |
  dashboard --JWT--->    |  website, verification,   |
  / admin                |  connector, knowledge,    |
                         |  exporters, publisher-    |
                         |  trigger, analytics-read, |
                         |  billing, admin,          |
                         |  ingestion                |
                         +-------------+-------------+
                                       |
                    enqueue jobs       |   read/write
                                       v
        +----------------+    +------------------+    +-------------+
        | Redis (BullMQ, |<-->| PostgreSQL 16    |    | ClickHouse  |
        | replay nonce,  |    | (graph + ops +   |    | (access /   |
        | idempotency,   |    | pgvector)        |    | job logs)   |
        | rate limit)    |    +------------------+    +-------------+
        +-------+--------+
                |
                v
        +----------------+        +------------------+        +-----------+
        | BullMQ worker  | -----> | object storage   | <----- | CDN       |
        | (pipeline)     | export | (MinIO / S3):    |  serve | (node     |
        +----------------+        | versions/ + cas/ |        | files)    |
                                  | + latest/        |        +-----------+
                                  +------------------+
                                                                   ^
                                  +------------------+             |
   AI agent / Claude Desktop ---> | MCP read server  | ------------+
                                  | (apps/mcp, stdio)|  read knowledge service
                                  +------------------+

Components

The API (Koa modular monolith)

A single Koa application composed of 12 internal modules with lint-enforced boundaries (a module's internal/ code may not be imported across module seams). The modules are: auth, organization, website, verification, connector, knowledge, exporters, publisher-trigger, analytics-read, billing, admin, and ingestion.

The composition root (apps/api/src/app.ts) applies middleware in order: request-id, error-envelope, body parser, pre-auth rate limit (IP-keyed), auth (only when a JWT signer is configured), post-auth rate limit (per-tenant), shell routes, then the module routers. See REST API.

The worker pipeline (BullMQ)

A separate process (apps/worker) that runs one BullMQ worker per job name, all on Redis. Each job is idempotent (a deterministic idempotencyKey collapses duplicates to skipped_idempotent) and has its own retry policy with a Redis dead-letter queue on exhaustion.

The live path is content.ingest to content.normalize to relationship.resolve to node.publish. The deterministic entity, fact, chunk, and QA extraction happens inside the normalize engine, not in separate jobs.

Exporter modules

Self-registering modules that project the knowledge graph into static files. They stream from the graph, run in a deterministic order, and write manifest.json last. See the exporter-module pattern and Node files.

Object storage and CDN

Publishing writes to object storage (MinIO locally, S3-compatible in production) under a per-site root: immutable versions/{ulid}/ copies, a content-addressed cas/ dedupe store, and a mutable latest/ serving copy. A CDN serves these files. The atomic flip writes latest/manifest.json as the single last mutating PUT, so a failed publish leaves the previous latest/ intact.

The MCP read server

apps/mcp is a hand-rolled MCP server (stdio, newline-delimited JSON-RPC 2.0, protocol version 2024-11-05) that exposes three read-only tools: search_website_knowledge, get_entity_facts, and list_node_files. Every query is tenant-scoped. This is the read side: agents and Claude Desktop query it directly.

Data flow (push to read)

1. A connector POSTs signed content to POST /v1/ingest/batch. The API verifies the HMAC, the timestamp window, the nonce (replay guard in Redis), the connector secret (Argon2id at rest), and scope, then persists raw_content_item rows and enqueues content.ingest.
2. content.ingest confirms the row and emits content.normalize.
3. content.normalize runs deterministic extraction, writes entities, facts, chunks, and QA into the graph, and emits relationship.resolve and node.publish.
4. relationship.resolve resolves dangling edges and emits node.publish if any were resolved.
5. node.publish runs the exporter registry, writes the immutable version, and flips latest/.
6. AI consumers read the published files from the CDN, or query the MCP server.

Pipeline stage status

This is the honest status of each worker job, from the maintainer notes. Stubbed jobs are marked Planned because they log and return without doing the work; the real extraction is inside the normalize engine, not these jobs.

Job	Status
content.ingest	Real
content.normalize	Real (loads the normalize engine via a dynamic-import seam)
relationship.resolve	Real (loads resolveRelationships via the seam)
node.publish	Real (loads the publisher via the seam)
analytics.aggregate	Real (ClickHouse to Postgres rollup; scheduled)
verification.check	Real (scheduled; ages out lapsed verification grace windows)
connector.heartbeat.check	Real (scheduled; flags connectors that stopped heart-beating)
org.transfer.execute	Real (single-transaction ownership re-association)
entity.extract	Stub (work currently happens inside the normalize engine)
fact.extract	Stub (work currently happens inside the normalize engine)
chunk.generate	Stub (work currently happens inside the normalize engine)
qa.generate	Stub (work currently happens inside the normalize engine)

The dynamic-import seam

content.normalize, relationship.resolve, and node.publish do their real work by dynamically importing the API's compiled dist/. The API must be built before the worker can run those stages; if the dist is absent, those jobs return a no-op success.

No scheduler is wired

There is no repeatable scheduler today. verification.check, analytics.aggregate, and connector.heartbeat.check exist in the job catalogue with retry policies, but nothing enqueues them on a timer. Wiring a scheduler is an operations task.

Datastores

• PostgreSQL 16 with pgcrypto and pgvector: the knowledge graph, operational tables, and the migration ledger.
• Redis: BullMQ queues, the ingestion replay-nonce cache, request-level idempotency, rate limiting, and per-site publish locks.
• ClickHouse: CDN access logs and terminal job outcomes, rolled into Postgres analytics_rollup by analytics.aggregate.
• Object storage (MinIO / S3): published node files.

Related

• Core concepts: the graph the system is built around.
• REST API: the mounted route surface.
• Node files: what node.publish produces.