Ingestion and signing

This is the authoritative reference for pushing content into bAInquet: the five ingestion endpoints, the HMAC signing scheme every request must carry, and the IngestItem schema. It documents the real server behavior, and the SDK signer mirrors the server verifier byte for byte.

If you use the official SDK or a native connector, you do not implement signing yourself. This page is for connector authors and for anyone debugging a 401.

Endpoints

All ingestion routes are POST and rooted at /v1/ingest. The calling principal is always a Connector (scoped bearer + HMAC), never a user.

Endpoint	Purpose
POST /v1/ingest/batch	submit 1 to 500 content items in one call
POST /v1/ingest/item	submit exactly one item (convenience)
POST /v1/ingest/delete	tombstone item ids (graph rebuild and republish)
POST /v1/ingest/heartbeat	liveness plus cursor and state report
POST /v1/ingest/schema	advertise connector field mappings (self-config)

Responses use the standard envelope (Envelope and errors). A route that returns per-item outcomes is always ok: true at the envelope level even when some items fail; item-level failures live inside results[].

Headers (every ingest request)

Authorization: Bearer <connectorId>.<secret>
Content-Type: application/json
X-Signature: <lowercase hex HMAC-SHA256 over the canonical string>
X-Timestamp: <unix seconds; replay window +/-300s>
X-Nonce: <per-request unique value; Redis replay guard>
X-Body-Sha256: <lowercase hex sha256 of the exact body bytes>
Idempotency-Key: <unique per logical request>     # REQUIRED
X-Site-Domain: example.com                          # item url host must match this
X-Connector-Type: wordpress                         # used by /ingest/schema
X-Connector-Version: 1.0.0

A missing Idempotency-Key is rejected 400 validation.missing_idempotency_key. The X-Site-Domain, X-Connector-Type, and X-Connector-Version headers are read by the service where relevant.

The HMAC signing scheme

This is the exact bq.connector.hmac.v1 scheme implemented by both the SDK signer and the server verifier. Any drift produces a 401.

The connector token issued by POST /v1/websites/:id/connectors is connectorId.secret. Split it on the first dot; the secret is base64url and contains no dot.

token         = `${connectorId}.${secret}`
signingKey    = HKDF-SHA256(ikm = secret,
                            salt = websiteId,
                            info = "bq.connector.hmac.v1",
                            length = 32 bytes)
bodySha256    = sha256(rawBody) as lowercase hex
canonical     = METHOD + "\n" + path + "\n" + bodySha256 + "\n"
                       + issuedAt + "\n" + nonce + "\n" + websiteId
X-Signature   = lowercase hex HMAC-SHA256(signingKey, canonical)
X-Timestamp   = issuedAt          (unix seconds)
X-Nonce       = nonce             (per-request unique; replay guard)
X-Body-Sha256 = bodySha256

The canonical string is six lines joined with \n, in this exact order:

1. METHOD, uppercased (always POST for ingestion).
2. path, the request pathname only, for example /v1/ingest/batch. No query string, no base URL.
3. bodySha256, lowercase hex sha256 of the exact body bytes you send. Any body change invalidates the signature.
4. issuedAt, the same unix-seconds value sent as X-Timestamp, as a string.
5. nonce, the same value sent as X-Nonce.
6. websiteId, the website the connector is bound to (also the HKDF salt).

Server verification pipeline

For each request the server, in order:

1. Reads the exact raw body and verifies the HMAC over the canonical string. A mismatch is 401 auth.invalid_signature.
2. Checks |now - issuedAt| <= 300s; outside the window is 401 auth.timestamp_skew. The check accepts exactly at the edge; one second beyond is rejected.
3. Checks the nonce against Redis with an atomic SET NX EX; a duplicate nonce is a replay and is rejected. The guard is fail-closed: a Redis error surfaces as a retryable 503 service.unavailable rather than skipping the check.
4. Resolves the connector row and Argon2id-verifies the secret at rest (with a decoy verify for unknown connectors so timing does not reveal existence). A revoked or suspended connector, or a stale tokenVersion, is rejected 401 auth.token_revoked.
5. Enforces scope: the body website_id must match the connector's bound website, and each item type must be in the connector's allowed source types. A mismatch is 403 auth.scope_violation.

Signing example (Node, no SDK)

A bare curl cannot derive the HKDF key, so signing happens in code; the example below emits the headers and posts the body. It is runnable with Node 18+ and no dependencies.

import { createHash, createHmac, hkdfSync, randomUUID } from "node:crypto";

const HKDF_INFO = "bq.connector.hmac.v1";
const sha256Hex = (s) => createHash("sha256").update(s, "utf8").digest("hex");

const signIngest = ({ token, websiteId, path, body }) => {
  const dot = token.indexOf(".");
  const secret = token.slice(dot + 1);
  const signingKey = Buffer.from(
    hkdfSync("sha256", Buffer.from(secret), Buffer.from(websiteId),
             Buffer.from(HKDF_INFO), 32));
  const issuedAt = String(Math.floor(Date.now() / 1000));
  const nonce = randomUUID();
  const bodySha256 = sha256Hex(body);
  const canonical = ["POST", path, bodySha256, issuedAt, nonce, websiteId].join("\n");
  const signature = createHmac("sha256", signingKey).update(canonical).digest("hex");
  return {
    Authorization: `Bearer ${token}`,
    "Content-Type": "application/json",
    "X-Signature": signature,
    "X-Timestamp": issuedAt,
    "X-Nonce": nonce,
    "X-Body-Sha256": bodySha256,
  };
};

const BASE = "https://api.bainquet.online";
const token = process.env.BNQ_CONNECTOR_TOKEN;     // connectorId.secret
const websiteId = process.env.BNQ_WEBSITE_ID;      // site_123
const path = "/v1/ingest/batch";

const body = JSON.stringify({
  website_id: websiteId,
  partial: true,
  items: [
    {
      type: "product",
      id: "p_1",
      url: "https://example.com/product/a",
      title: "Product A",
      json: { price: 16.99, currency: "EUR", sku: "A-001" },
      language: "en",
      checksum: "sha256:" + sha256Hex("Product A|16.99|EUR|A-001"),
      updated_at: "2026-06-11T08:30:00Z",
    },
  ],
});

const headers = signIngest({ token, websiteId, path, body });
headers["Idempotency-Key"] = randomUUID();
headers["X-Site-Domain"] = "example.com";
headers["X-Connector-Type"] = "custom";
headers["X-Connector-Version"] = "1.0.0";

const res = await fetch(BASE + path, { method: "POST", headers, body });
console.log(res.status, await res.json());

A retry of the same logical operation reuses the same Idempotency-Key but re-signs with a fresh nonce and issuedAt.

The IngestItem schema

The item shape is validated by a strict schema; unknown fields are rejected.

Field	Type	Required	Constraint
type	SourceType enum	yes	must be in the connector's allowed source types
id	string (1 to 256)	yes	connector-stable, opaque id (not a uuid); unique per (website_id, type)
url	string (URI)	yes	absolute http(s); host must match X-Site-Domain (or a verified alias)
canonical_url	string (URI)	no	when present, overrides url for source dedupe
title	string (<= 1024)	no
html	string	no (see below)	raw HTML
text	string	no (see below)	plain text
json	object	no (see below)	structured CMS fields or schema.org; highest-trust input (cms_field)
language	BCP-47 string	no	defaults to the website default
checksum	string	yes	must match sha256:<64 lowercase hex>; drives idempotency
updated_at	RFC3339 UTC Z	no	source-side last-modified
attributes	object	no	open bag resolved via /v1/ingest/schema mappings

At least one of html, text, json

The validator requires at least one of html, text, or json to be present (an OR). A connector may legitimately send both html and json. (Some older spec prose says "exactly one"; the enforced rule is at least one.)

SourceType enum

page, post, product, category, collection, faq, review, event, location, profile, doc, media, custom.

This is the wire contract. The server maps each wire value to a storage-legal column value before persisting; you only ever send the values above.

checksum format

checksum must match sha256:<64 lowercase hex>. It is the item-level idempotency key: an unchanged checksum for the same (website_id, id) is skipped, and a changed checksum is treated as an update. Compute it over whatever stable representation of the item your connector chooses (its serialized fields), as long as it changes when the content changes.

Request bodies

POST /v1/ingest/batch

{
  "website_id": "site_123",
  "partial": true,
  "items": [
    {
      "type": "product",
      "id": "p_1",
      "url": "https://example.com/product/a",
      "title": "Product A",
      "json": { "price": 16.99, "currency": "EUR", "sku": "A-001" },
      "language": "en",
      "checksum": "sha256:9f2b...64hex...",
      "updated_at": "2026-06-11T08:30:00Z",
      "attributes": { "brand": "Acme", "in_stock": true }
    }
  ]
}

• items: 1 to 500 (the cap is 500; over it is 413 ingest.batch_too_large).
• partial (default true): when true, an invalid item errors per item and the rest still process; when false, any invalid item rejects the whole batch (ingest.batch_rejected).

POST /v1/ingest/item

{ "website_id": "site_123", "item": { "...": "one IngestItem" } }

POST /v1/ingest/delete

{ "website_id": "site_123", "ids": ["p_1", "p_9"], "reason": "source_deleted" }

reason is one of source_deleted (default), gdpr_erasure, connector_resync, or takedown. gdpr_erasure forces a hard delete rather than a soft tombstone. Unknown ids are reported back, not treated as an error.

POST /v1/ingest/heartbeat

{
  "website_id": "site_123",
  "cursor": "page=42",
  "connector_state": "syncing",
  "stats": { "items_seen": 12044, "items_pending": 30 }
}

The response carries a next_action of continue, pause, or backfill, telling the connector how to proceed.

POST /v1/ingest/schema

{
  "website_id": "site_123",
  "mappings": [
    { "source_path": "acf.price", "target": "attribute", "attribute_name": "price" }
  ]
}

target is one of title, text, language, canonical_url, updated_at, or attribute (when attribute, attribute_name is required). A provenance_method of llm_inferred is rejected; ingestion is deterministic. A mapping that conflicts with an existing one is 409 ingest.schema_conflict.

Responses and per-item status

POST /v1/ingest/batch success (partial)

{
  "ok": true,
  "data": {
    "website_id": "site_123",
    "received": 1, "accepted": 1, "skipped": 0, "errored": 0,
    "results": [
      {
        "id": "p_1",
        "status": "accepted",
        "checksum": "sha256:9f2b...",
        "raw_content_id": "f3a1c2de-...uuid",
        "job_id": "content.ingest:01J...ULID"
      }
    ]
  },
  "meta": { "requestId": "req_01J..." }
}

Invariant: received == accepted + skipped + errored == results.length.

Per-item status is one of:

• accepted: queued for the content.ingest pipeline.
• skipped: an idempotent dedupe hit. The reason is idempotent_replay, unchanged_checksum, or tombstoned.
• error: carries an error.code plus a message for that item.

Idempotency

Two independent layers:

1. Request level: the Idempotency-Key header caches the full response. A replay with the same key returns the cached envelope and does not re-enqueue. A replay with the same key but a different body is 409 ingest.duplicate. If the original is still in flight, the replay gets a retryable 503 service.unavailable.
2. Item level: a (website_id, item.id, checksum) already present returns status: "skipped" without enqueuing. Changing the checksum for the same (website_id, id) is treated as an update.

Error and replay edge cases

Situation	Server response
HMAC mismatch or missing X-Signature	401 auth.invalid_signature
X-Timestamp outside +/-300s	401 auth.timestamp_skew
Replayed nonce	rejected as a replay
Redis unavailable during the replay check	503 service.unavailable (retryable, fail-closed)
Token revoked or stale tokenVersion	401 auth.token_revoked
website_id not the connector's website, or type out of scope	403 auth.scope_violation
Missing Idempotency-Key	400 validation.missing_idempotency_key
Same key, different body	409 ingest.duplicate
Body fails the schema	422 validation.failed (with details.fields[])
Batch over 500 items	413 ingest.batch_too_large

See Envelope and errors for the full code catalogue and the details shapes.

Conformance

Every connector should pass the black-box conformance suite, which asserts the envelope shape, idempotency, HMAC and the replay window, token scoping, delete/tombstone, and batch semantics. See Conformance.