# brain-knowledge The knowledge service holds one entry for each thing a Cloud Brain found. An entry records its source and its enterer. The entries are in a database of their own. No other Worker is bound to it. It is not the database the Brain runs on. An entry holds text to search, not a file. A paper's PDF or a page's HTML stays in `brain-static`. The entry has its path and its SHA-256. A search finds an entry by its words, or by its words and its meaning together. Each entry has one vector in one Vectorize index. `knowledge.search` with `semantic=true` puts the entries nearest a question and the entries that have its words into one order. An agent uses this to find the module, the paper or the note for a task. ## Methods | | | |---|---| | Worker | `brain-x-knowledge` | | Methods | `knowledge.put`, `get`, `search`, `list`, `delete`, `stats`, `embed`, `observe`, `facts` and `submit`. Each method's lexicon names what it takes, what it answers and its errors: `getSource?worker=brain-x-knowledge&part=lexicons`. | | Refusals | The wrapper refuses a call that does not match its lexicon with 400 `InvalidRequest`. The message names the field, as `input.public: a boolean`, and never its value. The handler checks the lengths and patterns below. | | Pages | `knowledge.list` answers the last entered first. `stale=true` answers the longest stale first, in one page with no cursor. `knowledge.embed` takes the last entered first. | | Filters | `source`, `kind`, `by`, `tag`, and `since`. `since` is compared with `at`. It takes milliseconds, or a date from `YYYY-MM` on. The service refuses four digits alone. | | Storage | `database` holds `entries`, `entries_words` (their words), `facts` and `searches`. `vectors` is the index `bge-base-en-768`, with one vector for each entry. | | Calls | `ai.v1/embeddings` on `brain-x-ai`, as this Worker, charged to the call's origin, or to this Worker for a caller with no account | ## An entry | field | | |---|---| | `id` | `a-z 0-9 . _ : -`, 1 to 100 characters. Default: the SHA-256 of the kind, the URL and the title. One thing has one id, whoever enters it: `arxiv:2610.08077`. | | `kind` | A name, `a-z 0-9 -`, 40 characters at most. In use: `paper`, `article`, `post`, `finding`, `note`, `module`, `doc`, `cell`, `product`, `offer`, `search`, `other`. A new source of entries can make up its own. | | `public` | `true`: anyone who may call a method reads the entry. `false`, or left out of a new entry: only the side of its enterer reads it (see Who reads an entry). | | `title` | 1 to 500 characters | | `text` | The abstract, the summary or the finding, 16000 characters at most. A search reads this and the title. | | `url` | Where the enterer found it: an `http` or `https` address that parses | | `source` | The service it came from, `a-z 0-9 -`, 40 characters at most: `arxiv`, `hn`, `simon-willison` | | `published` | The date the source gives, as text, 40 characters at most | | `file`, `sha256` | The path of the kept copy in `brain-static`, 200 characters at most, and the SHA-256 of its bytes | | `method` | How the enterer entered it, 100 characters at most: `snapshot`, `digest:research-YYYY-MM-DD`, `manual` | | `cites` | The ids of the entries this one is drawn from, 50 at most. A finding cites its sources. | | `tags` | `a-z 0-9 -`, 40 characters each, 16 at most | | `by`, `at` | Who entered it, and when. `by` is the caller the core names (`x-brain-caller`): `owner`, `worker:brain-x-NAME` or a DID. The service ignores a value in the body. A call with a token enters as its maker's call. For a call with a token, `by` is `"owner"` when a token the owner made enters, and the member's DID when a token a member made enters. The service does not keep the delegation and the token's name (`x-brain-via`, `x-brain-holder: secret:NAME`). `changed.by` is set the same way. | | `staleSince` | `null`, or the time in milliseconds when the source of the entry first differed from it. Whoever enters the entry sets it. A put of `{ id, staleSince: 0 }` clears it. The service does not look at a source. A search or a get still answers an entry that has `staleSince`, with the date. | | `changed` | `null`, or `{ by, method, at }` of the last put after the first, whatever it changed. A put of `staleSince` alone, or of `sha256` alone, counts. Its `method` is the one that put gave, or `null`. | | `wordsAt` | The time in milliseconds of the last write of the title or the text. `changed.at` does not say this, because a put that writes no word also sets `changed.at`. For some entries it is the time the entry was first entered. | With `unlessPublic: true`, the service leaves alone an entry whose kept copy is public. It counts that entry in the answer's `skipped`. The entry is also in `kept` when the put has `ifAbsent`. `brain-x-library` sends `unlessPublic` with the cards of a notebook that is not public. A card that anyone reads is then not overwritten from a file they cannot open. With `ifAbsent: true`, the service leaves a kept entry whole, whoever reads it. The answer's `kept` has its `{ id, url, file, sha256, staleSince, public }`. A writer uses this to tell whether its source has moved on, without writing the entry again. `brain-x-library` uses it for module cards (see brain-library). A put of a kept id changes the entry. `by`, `method` and `at` stay as first entered. A field that is left out, or `null`, keeps its value. An empty text clears `text`, `url`, `source`, `published`, `file` or `sha256`. `title` and `method` cannot be cleared. `tags` and `cites` are only added to: the given values join the kept values. To remove one, delete the entry and put it again. Two puts of one id at once both hold. A put reads each entry, merges, and writes it only while the row is the one it read. The guard is on the row's `at` and `changed.at` in the upsert itself, so the service uses no `rows` op. An id that another put wrote in between is read and merged again. The setting `putTries`, 3 by default, bounds the reads of one id. Past it the put answers 409 `Contended` and names the ids; the other ids are written. A 403 or 400 on a read again names the ids the call wrote before it. Each change makes `changed.at` greater than before, even within one millisecond. ```js await client.procedure("knowledge.put", { id: "arxiv:2610.08077", kind: "paper", title: "Self-Retrospection Distillation", text: "…", url: "https://arxiv.org/abs/2610.08077", source: "arxiv", method: "manual" }) await client.query("knowledge.search", { q: "hindsight training", kind: "paper" }) ``` ## A product and its offers A product is an entry of kind `product`, with the id `product:SLUG`. Its text says what makes it that product, and lists its offers. An offer is an entry of kind `offer`, with the id `offer:HOST:ITEM`. It is one shop's page of one product. Its `cites` holds its product, and its `url` is the page. Its `source` is the host with dots as dashes, and its `file` and `sha256` name the last kept copy. A search is an entry of kind `search`, with the id `search:YYYY-MM-DD:SLUG`. Its text holds the question, the results seen and why each kept one was kept. An offer found by a search cites the search too. The service checks none of these shapes. ## Facts A fact is one value a page showed, read at one time. It belongs to one entry. Any entry may have facts; an offer has most. | field | | |---|---| | `id` | The entry's id. The caller must be able to change the entry, as for a put. | | `field` | What the value is: `title`, `price`, `stock`, `lead-time`, `ship-de` (the shipping to Germany), `read` (`ok`, or why the page was not read) and `spec:NAME` for a line of a spec table. The pattern is `^[a-z0-9][a-z0-9:-]{0,59}$`. | | `shown` | The text as the page showed it, 1 to 500 characters | | `number`, `unit`, `currency` | The number in the text, its unit (40 characters at most), and its currency in three capitals. Each may be null. | | `variant` | The option the value belongs to, as shown, 200 characters at most, or null | | `pack`, `qtyFrom` | The units one price buys, and the least quantity of its tier. Each is 1 when left out. | | `readAt` | When the page was read, in milliseconds. A time over a minute ahead of the service's clock is refused. | | `file`, `sha256` | The kept copy the fact was read from | | `by` | The caller the core names. The service ignores a value in the body. | `knowledge.observe` only adds facts. No call changes a fact. Deleting the entry deletes its facts. A read again is a new fact beside the old one. The current fact of an entry, field, variant and tier is its newest reading. `knowledge.facts` answers the current facts, or with `history=true` every reading, newest first. It answers the facts of one entry (`id`), or of one `field` across entries. `min`, `max`, `unit`, `currency` and `kind` narrow it, and a bound holds for the current fact. A fact with no number passes no bound. The order is by field, then number, then entry. Each fact answered carries its entry's `kind` and `title`, its `readAt`, its `ageDays` in whole days, and `old`. `old` is true when `ageDays` exceeds the setting `factsOldDays`, 30 by default. With `qty=N`, `field=price` and `currency`, the answer is `offers`, one for each entry and variant that has a price tier at or under N. Only the entry's newest reading with a price counts, as the greatest `readAt` of its current prices. An entry whose newest reading has no tier at or under N has no offer. Each takes the tier with the greatest `qtyFrom` at or under N. `packs` is N over `pack`, rounded up, and `total` is `packs` times the price. `ship` is the current `ship-de` fact of the same currency, of the same variant or of none. `landed` is `total` plus its number, and `perUnit` is `landed` over N. Without such a fact, `ship`, `landed` and `perUnit` are null. The cheapest per unit comes first, and the offers with no `perUnit` come last. A price of another currency is not converted and is not in the answer. The service reads 1000 prices at most for one answer. A caller reads the facts of the entries they read, by the entry's reader now. A page holds 1 to 100 facts, with no cursor. ```js await client.procedure("knowledge.observe", { facts: [{ id: "offer:123bearing.eu:f3-8m", field: "price", shown: "€7.24", number: 7.24, currency: "EUR", readAt: Date.now() }] }) await client.query("knowledge.facts", { field: "price", currency: "EUR", qty: 10 }) ``` ## Submit `knowledge.submit` takes `{ url, note }`. The service reads the address and enters it as the Brain's entry. Anyone calls it by a post through `brain-x-delve`. - **An arXiv address** is `arxiv.org/abs|pdf|html/` (also `www.` and `export.`) or `huggingface.co/papers/`. It enters `arxiv:`, with the id's version left out, of kind `paper` and public. - The title, the abstract and the date come from the `citation_` meta of `export.arxiv.org/abs/`. When that page is not read, they come from the first entry of `export.arxiv.org/api/query?id_list=`. - An old arXiv id (`cs/0112017`) has a `/`, which no entry id holds. Its address enters as any other address. - **Any other address** enters `url:`, the id `brain-x-snapshot` gives a page. It is of kind `article` and private to the Brain. - Its title is the page's first ``, else its `og:title`, else the address. Its text is the rest of the page's text, 16000 characters at most. - A judge reads it and makes it public with a put of `{ id, public: true }`. A put that leaves `public` out keeps a private entry's reader. - **Each entry** has the source `town-submit`, the method `submit`, and the tags `submitted` and the poster's handle. The handle is lower-cased, with every character outside `a-z 0-9 -` made a `-`. A call over HTTP has no handle tag. - A stranger's new entry is embedded at this Worker's cost, 0.00005 USD, with an empty `x-brain-context`. A member's or the owner's is charged to their account. - The note, 300 characters at most, is the first line of the text. `by` is the caller the core names: `anonymous` for a stranger's post. - **A kept id** that is public or the Brain's is left whole. It gets the tag `submitted` when it has room, and its address is not read. The answer `kept` tells any caller that the Brain holds that id. - **An entry put under the id while the address is read** is left whole, as the put's `ifAbsent` leaves it. The answer is `kept` or `private` by its reader, as for an id kept before. - **The tag of a kept id** is written after its read. An entry its enterer makes private between the two still gets the tag `submitted`, and no other change. - **An id a member keeps privately** stays theirs. The service reads the address as for a new id, writes nothing, and answers `private`. So the answer does not show the member's entry; an arXiv id answers `private` in place of `entered`. - **The answer** is `{ id, outcome, public, title, heading }`, with `title` for a public paper alone. `outcome` is `entered` for a public paper, `private` for a page kept until it is judged, and `kept` for an id kept before. A reply by post shows `heading` as its sentence. - **A read** takes 10 s at most (the setting `submitMs`) and 5000000 bytes at most (`submitBytes`). - The service answers 400 with the reason for an address that does not parse, a scheme other than http or https, or a page past either bound. It does the same for a page that answers no 2xx, or an arXiv id with no paper. It answers 502 when arXiv answers neither its page nor its API. - **byPost**, the request post, is trusted only from the core's call by post whose `x-brain-holder` is `post:<byPost.did>`. Any other caller who gives it gets 403. - **The markup** is read by a scanner of tags and texts (`knowledgeMarkup`), not by patterns. The text of `script`, `style`, `noscript`, `template` and `textarea` is left out. ``` @cloudbrain.delve.town knowledge.submit url: https://arxiv.org/abs/2611.01234 note: a swarm that learns one policy ``` ## Tests No SQL engine runs in a page. `knowledgeFakeSql` answers for the statements of `knowledgeSql` from one map, and refuses any other statement. Its search finds a word as written, with no stemming, and orders by how many of the words an entry has. Under a use, each word found adds the use's weight of each field that has it. The fake answers a statement by its name and cannot know what an engine refuses. A test can therefore pass on a statement that no database takes. Run the statements on a real D1 database before a deploy. `test_knowledge_put_guard_on_a_sqlite` runs `knowledgeSchema`, `knowledgeColumns` and `knowledgeSql.put` over `bun:sqlite`. It checks the guard of the upsert by its changes count. A page has no `bun:sqlite`, and the test passes over it there. Vectorize and a model do not run in a page either. `knowledgeFakeVectors` is the binding as a map. The rig's embedding of a text is a count of its words.