Skip to content
Explanation

Case Ledger Synchronization

The specification governs

This page explains the design of the Vultron Protocol. It is informative: the normative text is the Vultron Protocol Specification. Where this page and the specification differ, the specification governs.

Organizations that do not share a database coordinate a vulnerability case. Each one keeps its own copy of the case. This page explains how those copies stay in agreement, and it answers two questions:

  • Who decides the order in which things happened?
  • What happens when messages arrive in the wrong order?

Vultron answers the first question with a single writer. It answers the second by holding an early message until the message it depends on arrives.


Before you read this page

This page builds on The CASE_MANAGER and the Case Ledger. That page explains why the CASE_MANAGER is the case's only writer, why the ledger holds only admitted entries, and how each new entry is sent to every participant. This page takes those as given and explains ordering and recovery.


What a ledger entry carries

Each entry is a small, immutable record. These are the fields that carry the ordering and integrity guarantees this page is about.

Field Meaning
case_id The case this entry belongs to
log_index Position in the case history; starts at 0 and counts up
published When the CASE_MANAGER stamped the entry, by its own clock
prevLogHash Hash of the previous entry
entryHash Hash of this entry's own content
payloadSnapshot A copy of the assertion the entry records

prevLogHash and entryHash link the entries into a chain, like the teeth of a zipper. Each entry names its predecessor, so a receiver can tell whether an entry belongs at the end of the history it already holds. The first entry in a case names the genesis hash, a value derived from the case object itself, so the chain is anchored to the case it describes. The hash is computed from the case id, its creation time and its owner (attributed_to). So a case its owner creates and one the CASE_MANAGER creates from the owner's proposal start from the same anchor (CLP-08-002, ADR-0117).

entryHash also lets a receiver check the entry's content, not just its place. Before a replica holds, applies or stores an entry, it recomputes the hash from the entry's fields and compares it with entryHash (SYNC-03-006). An entry whose content does not reproduce its hash is refused and logged as an error (SYNC-03-012). The replica sends no Reject(CaseLedgerEntry) for it, because a replay would resend the same entry. It sends the CASE_MANAGER a Create(ProcessingFault) with failure class LedgerEntryRefused/HashMismatch instead, naming the refused Add(CaseLedgerEntry) (SYNC-03-007). The CASE_MANAGER does not try to repair it, since a replay or a fresh case copy would resend the same content (SYNC-03-008). This implementation logs the fault as an error and rechecks its own stored copy of the entry, so the log says which side is broken (SYNC-03-010).

The CASE_MANAGER makes the same check before it sends an entry, whether it is sending a new entry to every participant or resending entries a participant is missing (SYNC-03-009). An entry that fails the check is not sent and is logged as an error. No later entry of that send goes out either, because the replica could hold those only as a gap. So a broken entry in the CASE_MANAGER's own store is found where it starts. In this implementation, a send that holds an entry back fails, and the act it belongs to commits nothing. The one exception is the catch-up of a participant whose updates were paused: that participant is skipped, and everyone else still gets the new entry (BT-14-003). Otherwise one broken old entry would stop the case from committing anything. The replica's check still catches a fault in rendering or parsing the wire form, which only shows up after the entry is sent.


Log index is the causal order

The log_index sequence is the causal order of the case. If the CASE_MANAGER observed event A before event B, then A has the lower log_index (CLP-14-001, ADR-0079). Anything that needs to know what happened first compares log indexes. It does not compare clocks.

This matters because clocks disagree. Two organizations on different continents, running different software, will not agree on the millisecond at which an event occurred. Sorting a shared history by wall-clock time would make the order depend on whose clock was fast. Sorting by log_index does not.

Timestamps are still recorded, and they are still useful — as supporting evidence, and to find a participant whose clock is wrong. They are not the order.

What the ledger claims, and what it does not

The ledger is authoritative from the CASE_MANAGER's point of view. It is a postmark, not a forensic reconstruction. It records the sequence in which the CASE_MANAGER observed and processed events. It makes no claim about when those events happened inside another organization (CLP-15-005).

  • For events the CASE_MANAGER generates itself, causal order is known by construction: the entries are written one after another on a single path.
  • For events reported by participants, the CASE_MANAGER's own receive-and-commit order is the order. The CASE_MANAGER cannot verify a causal claim about something it did not see.

The CASE_MANAGER never reorders entries to match timestamps supplied by someone else. Doing so would let external data override the one thing the CASE_MANAGER knows first-hand.

Rules the CASE_MANAGER follows

Rule Requirement
Order is never reversed: an event observed earlier never gets a higher log_index CLP-14-001
Every entry carries a timestamp; none may be empty CLP-14-002
Timestamps on consecutive recorded entries never move backwards as log_index increases CLP-14-003
Every entry in a ledger belongs to the same case CLP-14-004
No two entries in a case share a log_index CLP-14-005
No entry predates the case it belongs to CLP-14-006
Index numbers have no holes: the genesis entry is 0 and each later entry is one more than the entry before it CLP-14-010

Because the index run has no holes, a hole means an entry is missing. Suppose a receiver gets an entry that does not extend its chain, and whose log_index is greater than its tail index plus one. The receiver asks the CASE_MANAGER to replay what it missed, and it may hold the early entry aside until the missing ones arrive (SYNC-14-001, SYNC-14-002). A replica must hold every entry from genesis through the position it has acknowledged before it takes new protocol-significant actions on the case (SYNC-10-004). Refused assertions do not use up an index, because the ledger records only admitted entries.

The CASE_MANAGER should also refuse an assertion whose own timestamp is far from its clock. By default, that is more than five minutes ahead or more than seven days old (CLP-14-014, CLP-14-015). A deployment that knows its own clock conditions may tune both thresholds (CLP-14-009). These are sanity checks, not proof of honesty. A participant with a badly wrong clock can still produce a well-formed but misleading assertion.

A retry of an assertion the ledger already holds gets the entry recorded for it as its answer, however old its timestamp. That is because the retry records nothing new (OX-06-002).

Two timestamps, two different rules

An entry carries two timestamps, and the rules above do not all apply to the same one:

Timestamp Written by Rules it answers to
published on the entry the CASE_MANAGER, at commit time CLP-14-002, CLP-14-003, CLP-14-006 read across the whole ledger
published inside payloadSnapshot the asserting participant CLP-14-013 at the commit boundary, CLP-14-007, CLP-14-008, CLP-15-003, CLP-15-006

The distinction matters most for the "never move backwards" rule. Across the CASE_MANAGER's own commit timestamps, the rule holds because one writer stamps every entry from one clock. Applied to participants' timestamps, the rule would be wrong, because two participants' clocks are not synchronized. An assertion that one participant records later can legitimately claim an earlier time than an assertion that the other records earlier.

So the CASE_MANAGER compares a participant's claimed timestamp only against that same participant's previous assertion. It allows a small tolerance when it compares the timestamp against the case's creation time.


What participants must do

The CASE_MANAGER can only record the order it sees. So each participant has an obligation: send events in the order they happened.

  • If A caused B, send A first (CLP-15-001).
  • Do not collect several related events and send them in an arbitrary order (CLP-15-002).
  • Give B a timestamp no earlier than A's (CLP-15-003).
  • Timestamp an event with when it happened, not when the batch went out or a retry was attempted (CLP-15-004).

A participant that breaks these rules produces a malformed assertion. That is a fault on the sending side, not a CASE_MANAGER failure.

The CASE_MANAGER cannot enforce these rules at the moment an assertion arrives, and it must not try. It may not reconstruct an order it did not observe (CLP-15-005). It sees the order messages arrived in, which is not the order they were sent in — the transport makes no such promise.

So when an assertion's claimed time falls behind the same participant's previous assertion, the CASE_MANAGER records it and says so, and does not refuse it. It reports a small regression quietly, and one beyond the configured tolerance as a warning. A refusal would throw the message away. The ledger commit happens before any of the receiving work, so a refusal loses the whole message. Also, the CASE_MANAGER cannot tell a participant that really did misorder its events from two messages that simply arrived out of order.

The two ordering rules are visible only across a whole case. They show when the causal links a scenario declares are compared against the order the ledger recorded. The conformance suite checks them there.

The CASE_MANAGER does refuse one thing outright: an activity that arrives with no published time at all. Because the field is optional in ActivityStreams, a receiver could fill in the blank with its own clock. That receiver could no longer tell what the sender claimed from what it invented. Every check above would then compare the CASE_MANAGER's clock against itself. So the CASE_MANAGER refuses a message without a claimed time at the door (CLP-15-008).

A field that is present but empty carries no claimed time either. A blank published — an empty string, or one holding only whitespace — is refused the same way as an omitted one, and reports the same reason (MV-03-002).

A case status is held to the same rule wherever it sits in a message, except inside a case preview. Every case status carries the time its author made it. The CASE_MANAGER gives each new status a later time than the one before it, so time order and the order of the history agree (CM-29-005). A receiver therefore refuses an activity that carries a case status with no published (CLP-15-010). Inside a case preview it reads the status as the sender showed it, because a preview shows only what its sender chose (MV-12-006). The same holds for every other whole object a message carries, at any depth, except an actor profile, a collection or an embargo consent row; a typed reference is not a whole object, and the case preview it may carry is a projection of the object it names (CLP-15-011). A receiver also refuses a case that arrives with no case statuses at all, because starting one would put the receiver's own clock on the case's first status (CM-29-004).

A value that is present and not blank but unreadable as a timestamp is a different fault. The CASE_MANAGER reports it as malformed data, not as a missing field, so a sender is never told to supply something it already sent.


When entries arrive out of order

Ledger entries travel over ordinary message transport, which gives no ordering guarantee. Vultron may also not be the only implementation on the wire. So a replica can receive entry 7 before entry 6.

The naive response is to reject entry 7, because it does not extend the chain, and wait for the CASE_MANAGER to send it again. That does not work. The replay travels the same unordered transport and can arrive out of order again. So under repeated reordering, an entry can be lost indefinitely. In an earlier version of this implementation, that is what stalled a late-joining vendor at case closure.

Instead, a replica keeps an early entry (ADR-0037, SYNC-14-001). It puts the entry in a holding area and waits for the predecessor. Convergence then no longer depends on delivery order at all.

What gets held, and what does not

A replica holds an entry when the entry does not extend its chain but belongs later in the history. That is, its log_index is more than one past the end of what the replica holds. That is a forward gap: the predecessor has not arrived yet.

Two kinds of entry are not held:

  • An entry at or behind the end of the chain. That is a duplicate or a stale replay, not a gap.
  • An entry exactly one past the end of the chain whose prevLogHash does not match the hash of the replica's last recorded entry. Nothing is missing in front of it, so it is not a gap — it is a fork, a claim about a history the replica does not share.

Both go to the ordinary duplicate and divergence handling instead.

Even when it does hold an entry, the replica still sends Reject(CaseLedgerEntry) naming the last entry it applied (SYNC-14-002). Holding solves reordering, but it cannot solve loss. If the missing entry was never delivered at all, the reject is what prompts the CASE_MANAGER to send it again.

The CASE_MANAGER also does not replay for just anyone. It acts on a Reject(CaseLedgerEntry) only from an active participant in the case, and reports REFUSED to any other sender (SYNC-03-005).

The CASE_MANAGER does not replay on demand without limit. If a peer keeps rejecting from the same position, the CASE_MANAGER waits out a short cooldown before replaying to it again (SYNC-15-003). Without that bound, a peer that cannot anchor its chain rejects every entry it is sent. Each reject triggers another full replay, and the two feed each other into a storm.

Each further replay from that same position doubles the wait, up to a ceiling, so a peer that never catches up is retried less and less often (SYNC-15-018). A reject from a position that has advanced always triggers a replay, so a peer making progress is never held back. A reject from genesis for an entry the last replay from genesis already sent asks for nothing new, so the CASE_MANAGER may send nothing for it until the ceiling has passed (SYNC-15-019).

The holding area should also be bounded, so that a hostile or broken peer streaming far-future entries cannot exhaust the replica's storage. When it is full, the replica discards the held Add farthest ahead of the gap and logs a warning (SYNC-14-006, SYNC-14-013). Discarding is safe because the replica already sent the reject for that gap. But the eviction itself sends nothing.

Recovery therefore waits on the replay that reject triggers, or on the next entry that fails to match the tail. The cooldown above may delay either.

The holding area is deliberately not the ledger (SYNC-14-005). Across Vultron, the presence of an entry in an actor's own store means this entry is committed and its effects are applied (SYNC-13-001). A held entry is neither, so storing it would break that meaning.

Entries that arrive before the case

The same holding area covers a related race: an entry can arrive before the participant has the case object at all. Without the case, the replica cannot compute the genesis hash and so cannot anchor the chain.

A replica holds these entries too, with no gap comparison (SYNC-15-004). There is no chain yet to compare against, so the replica holds every entry for the unknown case, including entry 0. While it waits, the replica asks the CASE_MANAGER for the case and its whole ledger once, not once per held entry (SYNC-15-016). If a further entry arrives after ten minutes and the case still has not, the replica asks again, as a new request (SYNC-15-017). When the case is later delivered, the replica drains the holding area for that case (SYNC-15-005). Seeding the case is enough, because the genesis hash is derived from the case object. The genesis ledger entry does not have to be re-sent (ADR-0059).

Draining the held entries

Each held entry is filed under the hash of the predecessor it is waiting for. So after the replica commits a new last entry, it finds the successor with a single lookup. The successor is the held entry whose prevLogHash equals the hash of the entry just committed.

The drain then repeats until no held entry extends the chain (SYNC-14-003). Each step is a delivery of its own. The replica replays the Add(CaseLedgerEntry) that carried the successor through its own inbox (SYNC-14-009). That Add is the one it received and held as it arrived. The replica never builds an Add of its own for a held entry, so nothing is attributed to a sender that did not send it.

The replica does not process the successor inside the delivery that committed its predecessor. When that delivery commits the successor, it replays its successor in turn. A run of ten entries that arrived in reverse order clicks into place in one cascade, in log_index order.

Three properties make the drain safe:

  • Same path. A drained entry goes through the same processing as an entry that arrived in order — no separate code path with its own behavior (SYNC-14-008). In this implementation that holds by construction, because each held entry reaches the announce receive behavior tree as an ordinary inbox delivery.
  • Effects before storage. The entry's consequences — an embargo ending, a participant joining, a status changing — are applied first, and the entry is stored only if they all succeed (SYNC-12-001, SYNC-12-004, SYNC-14-004).
  • Applied once. An entry already in the local ledger is skipped entirely, so a replay or a duplicate cannot apply the same effects twice (SYNC-12-003, SYNC-14-007).

Because held entries are replayed through the ordinary path, the reject-and-replay recovery becomes order-tolerant as well, at no extra cost.


Why state checks wait for the drain

Holding an entry has a consequence that is easy to get wrong. Case State (CS) has ordering rules of its own:

  • An ephemeral state cannot be a resting place. So it has exactly one permitted successor and must be followed by it.
  • The recorded history must remain a valid sequence throughout.
  • A move to public awareness must be recorded before the embargo termination that it triggers.

If a replica checked those rules at the moment an entry arrived, an out-of-order entry would look like a violation. An entry naming a state that is only reachable from its predecessor would appear to arrive from nowhere.

That is why the checks run at drain time instead (CSB-19-001). A held entry is parked before any case-state effect is applied, so the state machine is never touched at arrival time. When the gap closes, the entries replay in log_index order, which is causal order. So the guards see them in the sequence they were meant to be seen in.

So an entry waiting in the holding area raises no state violation at all (CSB-19-002). An entry naming an ephemeral state is perfectly well-formed on its own. It would only be a real violation if the entry that resolves it never arrived. Flagging it on arrival would be a false alarm, and rejecting it would prevent the replica from ever converging.

The public-awareness-before-embargo-termination ordering survives reordering for free (CSB-19-003). The CASE_MANAGER commits the public-awareness entry first, so it holds the lower log_index. The drain works in log_index order, so it can never present the termination entry first. No receiver-side re-check of the timing is needed, because the order in the ledger is sufficient.


What a replica guarantees

Taken together, the ledger rules and the holding area give each participant these properties:

Property Meaning
Append-only A committed entry is immutable and identified by its content hash
Deterministic projection The same run of entries produces the same state in every conforming implementation
Idempotent replay Re-processing entries already seen changes nothing
Monotonic progress A replica never regresses the position it has acknowledged, and a late copy of the case cannot move it back
Reject on divergence An entry that does not extend the chain is not silently applied
Order-independent convergence Delivery order does not affect the final state

The result is eventual consistency with a strong bound. Given the same entries, every replica reaches the same state as the CASE_MANAGER, no matter what order those entries arrived in.

The holding area is kept in the replica's store, apart from its ledger, so a restart keeps what it holds. Nothing in it is committed, so it never stands in for an applied entry (SYNC-14-005). An entry it does not hold comes back the same way an evicted one does. The next entry that does not match the replica's tail produces a Reject, and the CASE_MANAGER replays from the position that reject names.

A Reject needs a later entry to arrive. When the lost entry is the last one before a quiet period, nothing follows it, so the replica has no gap to see and sends no Reject. The replica closes that gap itself by stating its position: Read(<case>/ledger), a Read of the case's ledger, states the last entry it applied, or index -1 when it holds none (SYNC-17-001). The CASE_MANAGER records that position and replays every entry after it, under the same cooldown as a Reject replay (SYNC-17-004). The reference participant sends one for each case it holds after every restart, the moment it is most likely to have missed an entry. A participant also states its position once a case copy first seeds its replica, so the CASE_MANAGER learns the replica is ready (SYNC-17-008). Whenever a participant states a position the CASE_MANAGER's ledger holds, the CASE_MANAGER resends each Invite that participant still owes an answer to, such as an embargo Invite or the full-case Invite it dropped before its case copy arrived (SYNC-20-001). It does so for any participant still on the roster, even one that has not consented to the embargo in force and whose position it otherwise refuses, because a resent Invite carries no withheld case content. A resend is the Invite as first sent, with its original id, ledger position and RSVP deadline, and it is rate-bounded per Invite and participant (SYNC-20-003, SYNC-20-005). Every other case message the participant sends states the same position as its ledgerPosition, so the CASE_MANAGER answers that position the same way whenever the participant acts (SYNC-18-007). A position the CASE_MANAGER's ledger does not hold means the participant's chain has split, and the CASE_MANAGER sends it a fresh copy of the case (SYNC-18-005, SYNC-18-012). Neither answer changes how the CASE_MANAGER judges the message itself.

A replica that already holds the case also receives fresh copies of it, as Announce(VulnerabilityCase). Each copy names the ledger position it reflects, the last entry its case file is a projection of. So the replica can tell a current copy from a stale one (SYNC-16-005). A copy the replica's chain is already past changes nothing (SYNC-16-020). A copy that names a different entry at a position the replica holds is refused as a conflict (SYNC-16-021).

Any other copy replaces the replica's case. When that copy is ahead of the replica's chain, the replica asks for the missing entries with a Reject. It also stores each entry the copy already reflects, without applying it again or moving its position back (SYNC-16-023, SYNC-16-024). An entry at the copy's position that is not the entry the copy names comes from a different chain. So the replica refuses it as a conflict and does not store it (SYNC-16-025).


Eventual consistency is the guarantee

Eventual consistency is the ordering guarantee the protocol makes, and the only one (SYNC-19-002, ADR-0161). The rule that goes with it is that no decision may depend on the order messages arrive in, except the order the CASE_MANAGER observes, which the ledger records (SYNC-19-001).

The rule covers more than ledger entries. The CASE_MANAGER also sends some messages directly, outside the ledger. A participant can receive one of these before, between, or after the ledger entries it relates to. Whatever the order, the participant must end with the same case file and give the same answer (SYNC-19-001).

The CASE_MANAGER cannot see the order in which one participant sent two messages either. So a decision must not depend on that order (SYNC-19-001).

A sender may deliver its messages to each recipient in order, and this implementation does (OX-01-005). That is an optimization, not a guarantee another implementation must give, and no participant may rely on it (SYNC-19-007, SYNC-19-003, SYNC-19-005, SYNC-19-006).

Messages the CASE_MANAGER sends outside the ledger

Most direct messages are only stored, and the participant takes their effect from the ledger entry, so their order cannot matter. The table lists each one and what makes its arrival order safe. Where the implementation still breaks the rule, the table names the open issue.

Message Related ledger entries What makes the order safe
Relayed embargo Invite(EmbargoEvent) The proposal entry, the consent rows, and the entry that records the Invite's emission The participant stores the Invite and the embargo it carries, and its answer needs only the case and the embargo. The owner's automatic answer to a first proposal reads the case without that proposal's own entry while the proposal is still open, so it does not depend on the entry either. An Invite that arrives after its proposal is decided is answered but not recorded as an open proposal, so the record ends the same in either order. The participant answers one Invite once, so a redelivered Invite sends no second answer.
Embargo Invite relayed to a joiner, or to a reinstated participant The joiner's copy of the case and the entries replayed to it An Invite that arrives before the copy waits for it, for a bounded time.
Copy of the case, Announce(VulnerabilityCase) Every entry at or before the ledger position the copy names The replica judges the copy by that position, and stores without effect each entry the copy already reflects.
Full-case Invite(whole VulnerabilityCase)[target=Actor] Its own entry The participant stores it, and its reply waits until its replica reaches the Invite's ledger position.
Stub Invite(VulnerabilityCase preview)[target=Actor] The invitee's participant entry The CASE_MANAGER refuses an answer to an Invite that a later one replaced.
Refusal notice Refuse(activity) and embargo refusal None It only closes the participant's pending assertion.
Removal and reinstatement notices The removal or reinstatement entry The participant stores the notice and applies the entry.
Embargo ending notices to a participant the ledger no longer reaches The termination or shorter-revision entry A termination notice that names a revision the participant has not applied yet means that revision replaced its embargo: the revision's own notice is still on its way, or none was owed because the revision was longer. The participant applies the revision first and then ends it. A late revision notice then finds no embargo in force and changes nothing.
Forwarded offers: a recommended participant, a role, an ownership transfer The offer's own entry The receiver stores the offer and answers it by trigger, and the CASE_MANAGER judges the answer against the ledger.
Answers to a case proposal, and the Create(VulnerabilityCase) that starts a case The case's first entries Entries that arrive before the case are held until it arrives.

Further reading