ORION DB August 2026 Michael Korneev

We Made the Primary Key the Index

How Orion DB turns an object ID into a zero-maintenance search structure — and the silent 32-bit wall we hit along the way

OID-as-index: classic index vs Orion DB OID index

How Orion DB turns an object ID into a zero-maintenance search structure — and the silent 32-bit wall we hit along the way

Most databases store a key in a column and then build a side structure — a B-tree, a hash table, an inverted index — to find rows by that key. The key lives in one place; the lookup mechanism lives in another. You pay for both: storage for the data, storage for the index, CPU to maintain it on every write, and complexity to keep them in sync.

We took a different path.

In Orion DB, every object has an OID — an object identifier inside a collection. Normally it auto-increments on insert: first object gets OID 1, next gets 2, and so on. But Orion DB exposes a feature called OID_IN that lets the caller specify the OID:


INSERT <Template> INTO <Collection>
  OBJECT &Obj OID_IN &vOid OID_OUT &oids

This changes everything. Hash a word to a number. Insert the posting at that OID. Looking it up is a point read — not a search through a side structure.


Classic index:   key stored in a column → search a B-tree → get a pointer → read the row
OID index:       key IS the address     → read it

"Is message X unread?" — one read. If the OID exists, yes. If absent, no. No scan, no filter, no index maintenance on write.

What an OID actually is

An OID is an object's unique identifier within a collection. Objects in Orion DB are stored in a 16-way trie — a tree structure walked one nibble (4 bits) at a time. This means lookup by OID is O(log₁₆ N) by construction. For a million objects, that's 5 hops. For a billion, 8 hops.

There is no separate index structure to maintain because the storage structure itself is the index. The trie is not a side-car bolted onto a heap file — it is how objects are stored.

And because OIDs ascend with insertion order, a collection whose OIDs are message IDs is already sorted newest-first. Reading "the 50 newest unread messages" is a walk down from the last ID. No ORDER BY. No sort buffer. No filesort. Just a descending trie walk.

What I designed before this work

This is important context. The OID-as-index concept was not invented during the indexation work — it was designed into the platform years earlier.

The concept: "We use OID like index, because Orion DB has insert by any definite OID." This was the starting design principle.

OID_IN — the user-facing twin of OID_OUT for INSERT operations, explicitly intended for content-addressed records where a hash becomes an OID.

InsertWithOid / InsertWithOidLL — the internal functions that thread a caller-supplied OID through the insert path. Critically, they only ratchet the collection's auto-increment cursor forward — they never rewind it. This means arbitrary hashed OIDs (which can be any number) can never break the auto-increment sequence for regular inserts.

The pattern was already in production. AIRecords and OrnGit were already using OID-addressed indexes before this work began.

Per-collection format versioning — a field called ObjVersion was written into the database format years before it was needed. The comment said: "in case the size of object headers also changes, and it can vary per list." This turned out to be exactly what made the 64-bit OID upgrade a non-event — existing databases opened unchanged under the new engine because the version field told the reader which header format to expect. This is the strongest proof of forward-looking design discipline in the whole story. A field nobody needed for years made a breaking change into a seamless upgrade.

The retrieval design was also pre-existing: a mailbox is a collection. Virtual folders (Unread, Read, tags) are their own index collections. Topics are stored as a set of rare-word hashes rather than exact phrases — so a user who misremembers the wording ("that email about the contract") still finds the message, because the individual word hashes match.

What the AI agent built

Working from the architecture above, the AI agent (Claude) implemented the following:

Descending reads with cursor paging. The list verb previously materialised all objects and sorted them in memory. For a mailbox with 954 messages, that meant loading 954 rows per page request: 543 milliseconds. The new implementation walks the trie in descending order and stops after a page of 200 rows: 94 milliseconds. The cursor tracks position, so the next page resumes from where the last one stopped — no gap, no repeat, no re-scan.

Widened OBJ_ID to 64 bits. This was the critical fix for hash-as-OID to work at scale (more on why below). The changes touched:

Test harnesses for every claim. All tests run offline against copies of real production databases — never against a live one.

The 32-bit wall

Then we hit something nobody saw coming.

One line in the codebase:


typedef unsigned int OBJ_ID;

That's 32 bits. The entire pipeline around it — the trie, the page allocator, the transaction engine — was 64-bit capable. But this one typedef truncated every OID to 32 bits before it reached storage.

Below 2³²: everything worked perfectly. No symptoms.

At 0xFFFFFFFF (2³² - 1): the value was silently dropped and an auto-incremented ID was assigned instead. The insert "succeeded" — but the object landed at the wrong OID. No error. No warning. The caller had no way to know.

Above 2³²: the insert failed outright.

Silent truncation is the worst failure mode in a database. A loud failure you catch in testing. A silent one hides until production data depends on the correctness of OID placement — which is exactly what hash-as-index requires.

It hid because nobody had asked for an ID that large before. The existing OID-addressed patterns (AIRecords, OrnGit) used OIDs well below the 32-bit ceiling. It was only when we started hashing words into OIDs — which can produce any number in the 64-bit range — that the wall became reachable.

Why the bit width matters

This is not academic. It is the correctness argument for the entire pattern.

When you hash words into OIDs and use a 32-bit space:

When you widen to 64 bits:

The fix was one typedef. The impact was the difference between "works" and "works correctly."

Sparse OIDs are free

A natural concern: if you hash "invoice" to OID 7,482,901 and "receipt" to OID 2,891,447,003 — don't you waste enormous amounts of space for all the OIDs in between?

No. We measured this explicitly.

First insert costs one 8 KB page. After that, inserting at OID 2²⁰ (1 million), then 2³⁰ (1 billion), then 2³¹ — zero additional file growth. The trie allocates pages on demand. An empty branch of the trie is a null pointer, not a pre-allocated block. Sparse IDs are as cheap as dense ones.

This is what makes hash-shaped OIDs practical. You can hash any word to any number and the storage cost is proportional to the number of objects you actually insert, not to the range of OIDs you use.

The test results

Every claim above is backed by a test that runs offline against a copy of a real production database.

OID-index probe test: Insert at deliberately sparse OIDs (77, 500, 964). Each lands at exactly the requested OID. Point read by OID finds the row. An absent key (123456) reads as absent. Descending walk starts at the highest key and proceeds in order.

Ceiling bisect test: Exact boundary identified: 2³¹ lands correctly, 2³² - 1 is silently dropped, anything ≥ 2³² fails. This is how we found the typedef.

Descending verb test: 6 out of 6 checks over 954 real messages. Same message set as ascending order. Strictly descending. Exact reverse of ascending. Newest-N (not oldest-N). Cursor paging with no gap and no repeat. Legacy ascending path unchanged (backwards compatibility).

Real index build test: An Unread index over a live mailbox: 949 messages scanned once → 257 postings created. The newest 50 then read straight off the index, strictly ordered, with no filter and no sort. That's 50 reads instead of 949.

64-bit upgrade test: OIDs at 2⁴⁰, 2⁵⁰, and 2⁶² all land correctly and read back. Existing databases open unchanged under the new engine — the ObjVersion field handles the transition.

Numbers worth remembering:

Four things the AI got wrong

This is the section that matters most. The technical details above are clean in retrospect. The path to get there was not.

1. The AI built an "index" that was a scan

The first implementation put the key in Vocab and Value columns — then loaded every row and filtered client-side. This is the exact thing the platform's own architecture forbids. Worse, the implementation created three postings per message (sender, recipient, and topic), so the index was larger than the table it indexed.

It benchmarked slower than having no index at all. And the conclusion drawn was: "the index pattern doesn't pay here."

Wrong. The pattern was correct. The implementation was a full table scan wearing an index costume. The platform owner's correction was blunt and accurate: "It's not index!"

2. The AI trusted documentation over the compiler

Our own documentation said EPL's int type is 32-bit. The compiler source says:


typedef uint64_t EL_UINT;

Two separate analyses of the OID ceiling were wrong because the AI assumed 32-bit integers where the runtime actually uses 64. The lesson: when the docs and the compiler disagree, the compiler is right.

3. The AI blamed the storage engine without evidence

After measuring a boundary at an API level, the conclusion was announced: "The Orion DB engine is fundamentally 32-bit." It wasn't. The engine's design was 64-bit-ready throughout. The bottleneck was a single typedef in the API layer. The AI blamed the engine for a problem it didn't have.

4. The AI guessed five times before instrumenting once

On a separate but related bug, five hypotheses were shipped. All five were wrong. Then two print statements were added — and they localised the issue immediately.

The engineering discipline says: instrument first, hypothesise second. The AI did it sixth.

The takeaway for all four: the fix was one line. Finding it took measurement. The detours all came from the AI assuming instead of measuring. Every shortcut that skipped instrumentation cost more time than the instrumentation would have taken.

Where the indexation system ships

The OID-as-index mechanism is not a standalone feature. It is a primitive that multiple products build on. The postings carry TargetCid (collection ID) and TargetOid (object ID), so the index does not care what it points at — it is a general-purpose lookup structure.

BizOS Mail — Unread/Read folders, tags, saved search filters, search by topic, sender, recipient, date range. The virtual folder system is entirely index-backed: "show me unread from this week" reads an intersection of two indexes.

BizOS Connect — Message search across team channels. Same mechanism, different target collections.

BizOS Tables — Row search across user-defined table schemas. The index adapts to whatever columns the user creates.

AIRecords — Where the pattern originated. Per-user AI analysis results, retrievable per message. This was the first production use of OID-addressed indexing.

OrnGit — Code and repository search. File contents, commit messages, and branch metadata indexed for fast lookup.

Cross-product search — Because targets are addressed uniformly (CID + OID), a query for "find anything about X" can span mail, chat, tables, and code from a single index. The search does not need to know which product the result lives in — the posting points to it directly.

Local AI integration — The AI model's job is to compose the query, not to read the corpus. A natural language request becomes a set of word hashes and constraints. The index returns ~50 candidates. The model ranks them. Retrieval stays bounded regardless of mailbox size. The model never sees the whole mailbox — it sees 50 pre-filtered results.

The design principle

The OID-as-index pattern works because Orion DB is not a general-purpose database bolted onto an application. It is a purpose-built storage engine inside a platform we control end-to-end.

The compiler (EPL) knows the database schema at compile time. The protocol (TProtocol) carries typed messages. The HTTP server (HSRV) reads directly from the database. The AI fleet queries the same indexes the application uses.

When you own the database at the engine level, you can make the primary key do things that other databases need a separate subsystem for. An OID becomes an address. A collection becomes an index. A point read becomes a search.

No B-tree maintenance. No index rebuild. No vacuum. No REINDEX. The key is the address, and the address is the object.

Orion DB is the database engine inside the Elastic Platform. We built the compiler (EPL), database, protocol, HTTP server, and GUI framework from scratch. 9 products ship on it. 20 AI agents build it daily.

More about the platform: elastcode.com/investors