Generic API connector

The generic API connector is not a package: it is the raw HTTP plus HMAC recipe for ingesting from any language or platform. Map your content to items, serialize, sign with the bq.connector.hmac.v1 scheme, and POST to the ingestion endpoints. The SDK is the reference implementation of exactly this recipe.

Use this path when no prebuilt connector fits your stack. For JavaScript or TypeScript, prefer the SDK; it implements everything below.

The four steps

1. Map each content unit to an IngestItem.
2. Serialize the batch body to JSON once.
3. Sign those exact bytes with the HMAC scheme.
4. POST to /v1/ingest/batch with the full header set, then read the per-item results.

Step 1: map content to items

Build one IngestItem per content unit. Field names are snake_case on the wire.

Field	Required	Notes
type	yes	A SourceType: page, post, product, category, collection, faq, review, event, location, profile, doc, media, custom. Must be in the token's allowed source types.
id	yes	Connector-stable, unique per (website_id, type), at most 256 chars. Opaque.
url	yes	Absolute http(s); host must match X-Site-Domain (or a verified alias).
checksum	yes	sha256:<64 lowercase hex> over canonicalized content (see step 3).
html / text / json	one of	At least one must be present; you may send more than one.
canonical_url, title, language, updated_at, attributes	no	title at most 1024 chars; language BCP-47; updated_at RFC3339 UTC Z.

The content checksum is sha256: plus the lowercase hex SHA-256 of a canonicalized object built from type, url, and whichever of html / text / json you send. Canonicalize by deeply sorting object keys, treating an absent field as null, then serializing deterministically. The checksum drives item-level dedupe, so identical content must always produce the identical checksum. Do not include updated_at in the checksum.

Step 2: serialize the batch

The batch body is:

{
  "website_id": "site_xyz",
  "partial": true,
  "items": [ /* 1 to 500 items */ ]
}

Keep each batch to at most 500 items and 5 MiB. Serialize the body to a single JSON string, then hash, sign, and send those exact bytes. Do not re-serialize after signing.

Step 3: the HMAC signing scheme

The scheme name is bq.connector.hmac.v1. The connector token is connectorId.secret (split on the first .).

signingKey  = HKDF-SHA256(ikm = secret, salt = websiteId, info = "bq.connector.hmac.v1", 32 bytes)
bodySha256  = sha256(rawBody)            // lowercase hex of the EXACT bytes you send
canonical   = METHOD\npath\nbodySha256\nissuedAt\nnonce\nwebsiteId
X-Signature = lowercase hex HMAC-SHA256(signingKey, canonical)

Field rules, all of which must match the server's verifier exactly:

• METHOD is uppercased (POST).
• path is the request pathname only, including /v1, with no host and no query string (/v1/ingest/batch).
• bodySha256 is the lowercase hex SHA-256 of the exact body bytes.
• issuedAt is unix seconds, sent as X-Timestamp; the server accepts a plus-or-minus 300s window.
• nonce is unique per request, sent as X-Nonce; the server rejects a reused nonce (Redis replay guard).
• websiteId is the website the token is scoped to; it is also the HKDF salt.
• The signature is compared in constant time.

Worked example

These values are reproducible. Given:

secret    = s3cr3t-base64url-value
websiteId = site_xyz
method    = POST
path      = /v1/ingest/batch
issuedAt  = 1700000000
nonce     = fixed-nonce
body      = {"website_id":"site_xyz","items":[{"id":"p1"}]}

the derivation yields:

signingKey (hex) = f37786546be4fb63a127845bbc3d42327849ca53f82672fab09e0de37cc97eb7
bodySha256       = f66cfb586eb72ad387d83ed02d0e321040b6ec08952eadf844cdcd64d09457fc
canonical        = POST\n/v1/ingest/batch\nf66cfb586eb72ad387d83ed02d0e321040b6ec08952eadf844cdcd64d09457fc\n1700000000\nfixed-nonce\nsite_xyz
X-Signature      = 7449cfa0b2bf8d1cceae8b8e7ec81d65e3c6c2ac881d514816c90c3e8499f6f8

If your implementation produces the same X-Signature for these inputs, it is byte-compatible with the server verifier.

Reference: HKDF and signing in Python (stdlib only)

import hashlib, hmac

def hkdf_sha256(ikm: bytes, salt: bytes, info: bytes, length: int) -> bytes:
    prk = hmac.new(salt, ikm, hashlib.sha256).digest()                 # extract
    okm, prev, counter = b"", b"", 1                                   # expand
    while len(okm) < length:
        prev = hmac.new(prk, prev + info + bytes([counter]), hashlib.sha256).digest()
        okm += prev
        counter += 1
    return okm[:length]

def sign(secret: str, website_id: str, method: str, path: str, body: bytes):
    key = hkdf_sha256(secret.encode(), website_id.encode(), b"bq.connector.hmac.v1", 32)
    body_sha = hashlib.sha256(body).hexdigest()
    import time, uuid
    issued_at = str(int(time.time()))
    nonce = str(uuid.uuid4())
    canonical = "\n".join([method.upper(), path, body_sha, issued_at, nonce, website_id])
    sig = hmac.new(key, canonical.encode(), hashlib.sha256).hexdigest()
    return sig, issued_at, nonce, body_sha

Step 4: POST with the full header set

Required headers on every ingestion request:

Header	Value
Authorization	Bearer <connectorId.secret>
Content-Type	application/json
X-Signature	lowercase hex HMAC-SHA256 (above)
X-Timestamp	issuedAt (unix seconds, plus-or-minus 300s window)
X-Nonce	unique per request
X-Body-Sha256	the body sha256 (also bound into the canonical string)
Idempotency-Key	required; a 400 is returned if missing
X-Site-Domain	the site host (every item url host must match)
X-Connector-Type	your connector identifier
X-Connector-Version	your connector version

curl example

#!/usr/bin/env bash
set -euo pipefail

BASE="https://api.bainquet.online"
TOKEN="$BAINQUET_CONNECTOR_TOKEN"      # connectorId.secret
WEBSITE_ID="site_xyz"
SITE_DOMAIN="example.com"
PATH_="/v1/ingest/batch"

BODY='{"website_id":"site_xyz","partial":true,"items":[{"type":"product","id":"p_1","url":"https://example.com/product/a","title":"Product A","html":"<h1>Product A</h1>","language":"en","checksum":"sha256:..."}]}'

SECRET="${TOKEN#*.}"                   # everything after the first dot
ISSUED_AT="$(date +%s)"
NONCE="$(uuidgen)"
BODY_SHA="$(printf '%s' "$BODY" | openssl dgst -sha256 -hex | awk '{print $2}')"

# signingKey = HKDF-SHA256(secret, salt=websiteId, info="bq.connector.hmac.v1", 32)
# (openssl 3.x supports HKDF via the `kdf` command; or precompute in your language)
SIGNING_KEY_HEX="$(openssl kdf -keylen 32 -kdfopt digest:SHA256 \
  -kdfopt "key:$SECRET" -kdfopt "salt:$WEBSITE_ID" \
  -kdfopt "info:bq.connector.hmac.v1" -binary HKDF | xxd -p -c 256)"

CANON=$(printf 'POST\n%s\n%s\n%s\n%s\n%s' "$PATH_" "$BODY_SHA" "$ISSUED_AT" "$NONCE" "$WEBSITE_ID")
SIG="$(printf '%s' "$CANON" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$SIGNING_KEY_HEX" -hex | awk '{print $2}')"

curl -sS -X POST "$BASE$PATH_" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -H "X-Signature: $SIG" \
  -H "X-Timestamp: $ISSUED_AT" \
  -H "X-Nonce: $NONCE" \
  -H "X-Body-Sha256: $BODY_SHA" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "X-Site-Domain: $SITE_DOMAIN" \
  -H "X-Connector-Type: my-connector" \
  -H "X-Connector-Version: 1.0.0" \
  --data-raw "$BODY"

WARNING

--data-raw "$BODY" posts the body unchanged. Hash and sign the exact bytes you send. If a tool re-encodes the JSON (reorders keys, changes whitespace) after you compute X-Body-Sha256, the server returns 401 auth.invalid_signature.

Read the response

Every response is the standard envelope. Branch on ok, never the HTTP status alone (a 200 can carry a partial-batch result):

{
  "ok": true,
  "data": {
    "received": 1, "accepted": 1, "skipped": 0, "errored": 0,
    "results": [{ "id": "p_1", "status": "accepted" }]
  },
  "meta": { "requestId": "req_..." }
}

Walk data.results[]: each item is accepted, skipped (already current), or error. The invariant is received == accepted + skipped + errored == results.length. For an error item, read error.code, fix that item, and re-send only the failed ones with a fresh Idempotency-Key; good siblings dedupe to skipped.

Retry only 5xx and 429 (honor Retry-After), re-signing with a fresh nonce and timestamp but reusing the Idempotency-Key. Do not retry 400, 403, 409, 413, or 422.

Other endpoints

The same signing and headers apply to every ingestion endpoint:

• POST /v1/ingest/item: single upsert.
• POST /v1/ingest/delete: tombstone by id with a reason.
• POST /v1/ingest/heartbeat: report a cursor; honor next_action.
• POST /v1/ingest/schema: advertise field mappings for custom attributes.

See Ingestion and signing for the full endpoint reference, and the SDK as the reference implementation of this recipe.