Labels, extractors and review

An annotator is a question InboxQL asks of your mail, by name, and remembers the answers to. There are two kinds:

• a label answers yes or no — is this a purchase? — or picks a level — how important is this conversation?
• an extractor pulls values out — what was the amount, the merchant, the order number?

The answers become query terms: label:purchased, label:importance, extract:receipts.amount>100. Nothing else about them is special, which is the point — once a question has been answered, it is as cheap to ask as from:.

Open Intelligence → Labels & extractors to see them.

These are the labels a model or a rule decides. The labels you put on mail — Inbox, Receipts, the ones you made in Gmail — are the other kind, covered in the Desk. label: answers for both, and the rail lists both under Labels.

Somewhere to start

A new mailbox offers the elements: the things worth knowing about mail later, answered by your AI harness reading each conversation once. Each says how much it has answered.

They replaced an earlier pack built on Laya and GLiNER, which read a few hundred words of one message — too little to know that a receipt line is a DeWalt battery, or that “I’ll get back to you Monday” leaves the next move with you. Installing replaces those starters only where they never answered anything.

Installing runs nothing. Reading with your AI harness is how they get answered.

The engines

An annotator’s engine is what answers the question. They are good at different things, and choosing well matters more than anything else here.

Your AI harness — reading whole conversations

The elements’ engine. Nothing in InboxQL runs it: the AI you already use reads conversations through iql work next and answers through iql work submit, and InboxQL checks every answer — levels and fields as declared, a quote that is really in the conversation — before keeping it. It reads everything: the whole conversation and the text of its attachments. See Reading with your AI harness.

Laya — judgement, on this machine

Answers a yes-or-no question, or picks a level, by scoring each possible answer. It cannot invent an answer that was not one of the options, and it runs on this computer: about two to three seconds a message, nothing sent anywhere. The first annotator that needs it asks to download it (about 600 MB, becoming 1.2 GB once prepared), or run iql laya install.

Write its instruction as a statement about the message — “this message is a receipt or an invoice” — not a question or a prompt.

It reads the subject, the sender and the first few hundred words. For a per-conversation label it reads a one-line summary of the conversation, then the newest message.

GLiNER — values, found in the text

Pulls values out by finding them in the message: every value it returns is a piece of the message’s own text, with its position. That makes it the right engine for extraction — it cannot make up an order number that is not there. It reads the body and any attached files whose text has been read, which for receipts is usually where the numbers are.

Its labels are the field names, so name fields plainly: amount, merchant, due date. Up to twelve at once. About 15–20 seconds a message on a laptop, and also entirely local; iql gliner install downloads it once.

LLM — understanding

A prompt sent to a language model, through the AI provider configured in Settings → AI Configuration. Use it when the question needs understanding rather than locating — what is the sender actually asking for? — and expect it to be slower and, for extraction, less trustworthy: a generative model can return a value that is nowhere in the message.

If the provider is on another machine, running an LLM annotator sends every message in its scope there. InboxQL refuses to do that until you have agreed to it for that annotator, and Plan says how much text would leave.

Rules — a query

A rule annotator is just a query, evaluated by the database in one pass. It is free and exact, and if a query can say what you mean, a saved query is usually the simpler way to say it. Rules are available from the command line (iql annotate create … --engine rule).

Reading the answers

label:purchased              said yes
extract:importance.level=important
                             picked that level
label:purchased@0.9          said yes with a score of 0.9 or more
-label:purchased             looked, and said no
unlabeled:purchased          has not looked yet
extract:receipts             found at least one record
extract:receipts.amount>100  found one with an amount over 100

A levelled label’s level is a value like an extractor’s, so it is filtered and grouped the same way: extract:importance.level=important, and label:importance | count by importance.level for how your mail divides up. A per-conversation label is recorded on each conversation’s newest message; add | thread to see the whole conversation.

A label has three states. -label:x is evaluated, no — it does not include mail the annotator has never looked at. Before concluding “there are no more of these”, check how far it has got: the card shows how many messages it has evaluated out of how many are in scope.

Its scores order; they do not measure

Laya’s scores rank messages well — a 0.95 is more likely to be right than a 0.6 — but out of the box they are not percentages. Its published model is not calibrated, so label:purchased@0.9 is “the more confident ones”, not “ninety per cent of these are right”. Calibration, below, fixes that from your own rulings.

Teaching it: rulings

When a label is wrong, say so. Every label appears as a chip under the sender of a message; right-click it (or focus it and press the menu key):

• Right — confirm it.
• Wrong — it should have said the opposite.
• Should be … — for a levelled label, the level it should have picked.
• Clear my ruling — go back to what the engine said.

A ruling outranks the engine. It survives every re-run and every change to the annotator’s instruction, and the message never goes back in its queue. On a per-conversation label, the ruling lands on the conversation’s newest message, which is the one the label is about.

Review

Rulings made while reading are biased: you notice the mistakes. To measure an annotator honestly, use Review on its card. It draws ten messages it has answered — spread across its whole range of scores, not just the doubtful ones — and hides its answer until you have given yours. Press y or n, or 1–4 for a level.

The card then shows a score line: how often it agreed with your reviewed rulings, and for a yes-or-no label the score cut-off that would have been right most often. Ten rulings is too few to trust, and it says so; around fifty starts to mean something. iql annotate score <name> gives the same answer in a terminal.

Two annotators asking the same question with different engines, each reviewed, is how you find out which engine to keep.

Calibration

Once an annotator has at least twenty reviewed rulings,

iql laya calibrate purchased

fits its scores to them, so that afterwards 0.9 means roughly nine in ten. Calibration never changes which way an answer goes — it cannot make the model more accurate, only more honest about how sure it is. It uses reviewed rulings only, because a set made of corrections would drag every score toward a coin toss.

Making your own

New annotator on the Annotators tab, or:

iql annotate create waiting-on-legal --engine laya \
  --instructions "this message is waiting on a contract or legal review"
iql annotate run waiting-on-legal --scope "after:30d" --dry-run
iql annotate run waiting-on-legal --scope "after:30d"

Changing the instruction makes it a new version. Earlier answers are kept but marked stale, and the next run revisits only what the new version has not answered. Your rulings carry over.

Try a new annotator on a narrow --scope first. A whole mailbox at three seconds a message is a long time to find out an instruction was badly worded.

When it runs

A pass handles at most 200 messages per annotator and leaves the rest for the next. On a laptop running on battery, the automatic pass after a sync waits for mains power unless you allow it in Settings → System → Heavy work on battery.

Switching off, and deleting

Switching an annotator off stops it running — triggers skip it, Run is disabled — and keeps everything it has already said. label:purchased keeps matching. That is almost always what you want.

Deleting removes the annotator and every answer it ever gave, including your rulings, which nothing can rebuild. The confirmation says how many answers that is. An annotator another one is scoped on — purchased, behind receipts — cannot be deleted without switching the dependent one off.