Skip to content
Explanation

Timing Metrics Within a Case

This page is not normative

This page is not considered a core part of the Vultron Protocol as proposed in the main documentation. Although within the page we might provide guidance in terms of SHOULD, MUST, etc., the content here is not normative.

This page proposes metrics for the time a Coordinated Vulnerability Disclosure (CVD) case takes between its events, not only the order of those events. The measures in this section so far are about ordering: whether one event precedes another.

The case ledger records the time at which the CASE_MANAGER committed each event. These times make intervals, durations, and margins measurable within a single case. The page is a draft research proposal for people who study the CVD process. No part of the page is normative, and no code calculates these metrics today.

A desideratum is an order of two events that a case should achieve. For example, the fix should be ready before the vul is public. Desirable Histories gives the twelve original desiderata. Desiderata from the Process Models adds desiderata about report, embargo, and consent events.

The page uses the events and symbols of What a Case Records:

Symbol Event Belongs to
\(\evR_i\) Participant \(i\) receives the report (report receipt) Participant \(i\)
\(\evVa_i\) Participant \(i\) decides that the report is valid (report validation) Participant \(i\)
\(\evAc_i\) Participant \(i\) accepts the report for work (report acceptance) Participant \(i\)
\(\evC_i\) Participant \(i\) closes the report (report closure) Participant \(i\)
\(\evJ_i\) Participant \(i\) joins the case (joining) Participant \(i\)
\(\evS_i\) Participant \(i\) agrees to the active embargo (embargo agreement) Participant \(i\)
\(\evV_i\) Vendor \(i\) becomes aware of the vul (vendor awareness) Vendor \(i\)
\(\evF_i\) Vendor \(i\) has a fix ready (fix readiness) Vendor \(i\)
\(\evD_k\) Deployer \(k\) deploys a fix (fix deployed) Deployer \(k\)
\(\evP\) The public becomes aware of the vul (public awareness) The case
\(\evX\) An exploit becomes public (exploit public) The case
\(\evA\) Someone observes attacks that use the vul (attacks observed) The case
\(\evE\) An embargo starts on the case (embargo start) The case
\(\evEx\) The embargo ends (embargo exit) The case

The subscript \(i\) names one participant or one vendor, and the subscript \(k\) names one deployer. Without a subscript, a symbol means the event for any participant, or for the one vendor of a two-party case. The notation \(a \prec b\) means that event \(a\) occurs before event \(b\). On the Measuring CVD pages, hover over an event symbol to see its name and meaning.

Each metric on this page is about the case, or about one participant. Some events belong to the case and occur at most once in the case. These events are public awareness, a public exploit, observed attacks, and the start and end of the embargo.

Other events belong to one participant and can occur once for each participant. These events are report handling, fix readiness, deployment, and embargo consent. What a Case Records describes these two kinds of event, the ledger that stamps each event, and the gaps in the ledger.


Ordering is the sign of an interval

Open Questions names the gap this page addresses: the Case State (CS) model has a sense of order and no sense of time. The measures of the Foundations count only whether each desideratum held in a case. Consider two cases where fix readiness occurs before public awareness (\(\evF \prec \evP\)). In one case, defenders had a month of warning, and in the other case, one minute. The measures of the Foundations give the two cases the same score.

Every desideratum has a matching interval. Let \(\sigma_1\) and \(\sigma_2\) be the two events of a desideratum \(\sigma_1 \prec \sigma_2\). Let \(t(\sigma)\) be the time at which an event \(\sigma\) occurred. The margin of the desideratum is

\[m(\sigma_1 \prec \sigma_2) = t(\sigma_2) - t(\sigma_1)\]

A positive margin means the desideratum held, and its size says by how much. A negative margin means it failed, and its size says how badly. The measures of the Foundations keep only the sign of the margin \(m\). Thus each timing metric below adds detail to a Foundations measure, and does not replace it.


Which clock to use

The CASE_MANAGER writes one case ledger entry for each change that it admits to the case. Thus the ledger is a history of the case in time order, and almost every metric on this page uses the ledger as its data.

Each ledger entry carries two times: the commit stamp of the CASE_MANAGER, and the claimed time of the sender. What a Case Records describes the two times. Every interval on this page uses the commit stamp by default, because all commit stamps in a case come from one clock. A commit stamp shows when the CASE_MANAGER learned of an event, and the CASE_MANAGER can learn of an event late.

Some intervals start and end at entries that the same participant sent, for example the time to fix of vendor \(i\). For these intervals, both claimed times come from the clock of that participant, and the claimed time should be the time that the event occurred (CLP-15-004). A fixed offset in the clock of that participant cancels out of the interval. Thus the claimed times are often the better data for an interval inside one participant. An event of a participant is not always in an entry that the participant sent. For example, the CASE_MANAGER or the Case Owner can report a deployment for a deployer, and the CASE_MANAGER records the join of an Invitee. An interval with ends from two senders mixes two clocks.

The clock skew of a participant, listed under process health below, is not by itself a test of its claimed times. A large skew can mean only that the participant reported its events late, and then its claimed times are the better data. Some signs do show a clock that is wrong:

  • A negative skew: a claimed time after the commit of its entry.
  • Claimed times that decrease across activities that depend on each other (CLP-15-003).
  • A skew that changes much from one entry of the participant to the next.

An interval on claimed times and the same interval on commit stamps can differ. A large difference is itself worth reporting. It shows that the two ends of the interval reached the ledger with different delays, or that the clock of the participant drifted. The delay can be on the side of the participant, or on the side of the CASE_MANAGER, for example when the CASE_MANAGER was down.

A few metrics use a third time: the time on an object inside an activity, such as the time on a report. The CASE_MANAGER carries this time as the sender received it (CLP-15-007), so it comes from the clock of the author of the object.

This page does not choose a clock for each metric. An implementer of the metrics can use commit stamps, claimed times, or both.


Kinds of measure

Five shapes cover the proposed metrics.

Shape Definition Example
Interval Time from one event to another Time from awareness of vendor \(i\) (\(\evV_i\)) to \(\evF_i\)
Margin A signed interval for an ordering desideratum \(m(\evF \prec \evP)\)
Dwell time Total time the case spends in a state Time with an embargo revision open
Spread Time between the first and the last participant to reach the same event Fix readiness of the first vendor (\(\evF\)) to fix readiness of the last vendor (\(\evF\))
Count or rate How often something happens in one case Embargo Invites that end in a pocket veto

Intake and triage

These intervals measure how fast a report becomes a case and a commitment. They are per-participant, because Report Management (RM) is per-participant.

Invitees and their two Invites

An Invitee is a participant that the case invites to join. The case first sends the Invitee a Stub Invite, which asks the Invitee to join and carries only a summary of the case. After the Invitee joins, the case sends a Full-Case Invite, which carries the full case and asks the Invitee to judge the report (CM-11-011). The Glossary defines both Invites.

Metric Definition
Time to validate Report receipt by participant \(i\) (\(\evR_i\)) to report validation by participant \(i\) (\(\evVa_i\))
Time to accept \(\evVa_i\) to report acceptance by participant \(i\) (\(\evAc_i\))
Time to join, for an Invitee Its Stub Invite to joining by participant \(i\) (\(\evJ_i\))
Time to judge, for an Invitee Its Full-Case Invite to its answer, which is \(\evVa_i\) when it accepts (CM-11-011)
Report to case The time on the report that created the case to case creation, when the report carries a time

For an Invitee \(i\), \(\evR_i\) and \(\evJ_i\) are the same event. The reason is that acceptance of the Stub Invite puts the Invitee at RM Received (CM-11-001). For an Invitee, then, time to validate is time to judge plus the wait between joining and receiving the Full-Case Invite.

The Case Owner's \(\evR\) and the reporter's \(\evAc\) are set when the case is created (What a Case Records). The validation and acceptance by the Case Owner come later, each in its own ledger entry. Thus time to validate is measurable for the Case Owner from case creation, and time to accept is measurable from the validation. The reporter's \(\evAc\) is not a decision, so the reporter has no time to accept. The time that the Case Owner held the report before it created the case is not on the ledger.

Report to case is a bound on that time. The ledger entry that adds the report to the case carries the report as received, with the time that the reporter put on the report (CLP-15-007, CLP-15-011). The Case Owner received the report after that time and before it created the case. Thus, when the reporter's clock agrees with the CASE_MANAGER's clock, report to case is an upper bound on how long the Case Owner held the report. A reporter's clock that runs fast makes report to case too small, and it can make the value negative. The interval uses two clocks, so it is a research measure. A report that the Case Owner adds to the case later has no report to case.


Embargo

These metrics describe how an embargo is negotiated, how long it lasts, and how it ends. Embargo Management (EM) is per-case, and Participant Embargo Consent (PEC) is per participant and per embargo.

Embargo terms used in this table

Term Meaning
Proposal A participant proposes an embargo, and the Case Owner accepts or rejects the proposal
Revision, EM Revise A proposal to change the active embargo; the EM state is Revise while a revision is open
RSVP deadline The time by which an invited participant must answer an embargo Invite
Pocket veto An embargo Invite that reaches its RSVP deadline without an answer, so the consent row moves to TIMED_OUT
Lapse A participant lapses when it agreed to an earlier embargo, but has not agreed to the revision that is now active
Threat signal Public awareness, a public exploit, or observed attacks, each of which ends an active embargo early

The Glossary defines each term.

Metric Definition
Time to embargo Case creation to the embargo start (\(\evE\)), usually zero because a case starts with an active embargo
Decision latency A proposal to the Case Owner's decision on it
Revision count and time in Revise How many revisions were proposed, and the dwell time in EM Revise
Planned and actual length The embargo's end time minus \(t(\evE)\), against the time of the embargo exit (\(t(\evEx)\)) minus \(t(\evE)\)
Time left at exit The embargo's end time minus \(t(\evEx)\); zero when the end time was reached
Exit cause Whether the end time was reached, or the embargo ended early, and for an early end its recorded cause
Response latency Each participant's time from an embargo Invite to its answer
Pocket-veto rate The share of embargo Invites that move to TIMED_OUT at their RSVP deadline, not counting Invites that time out because the Case Owner closes the case
Lapse count How many participants lapsed when a revision with longer terms came into force

Time left at exit and exit cause together separate the two ways an embargo ends. An embargo that ends early because every fix is ready ends on its own terms. An embargo that ends early because of a threat signal has a margin of about zero against that threat signal. With commit stamps, this margin is slightly negative, because the CASE_MANAGER commits the termination after the status that reported the threat signal. The time left at exit of such an embargo is the cost of the threat.


Fix and deployment

Metric Definition
Time to fix \(\evV_i\) to \(\evF_i\), per vendor
Time to deploy Fix readiness of the first vendor (\(\evF\)) to fix deployment by deployer \(k\) (\(\evD_k\)), per deployer
Notification spread Awareness of the first vendor (\(\evV\)) to awareness of the last vendor (\(\evV\))
Fix spread Fix readiness of the first vendor (\(\evF\)) to fix readiness of the last vendor (\(\evF\))
Trailing-vendor gap Fix readiness of the last vendor (\(\evF\)) to \(\evEx\); negative when the embargo ended before every fix was ready

Time to deploy starts at the first \(\evF\) of any vendor in the case. The reason is the protocol rule for deployment (CSB-15-004). A deployer that is not also a vendor can move to \(\evD\) after any vendor has a fix ready. The case does not record which vendor made the fix that a deployer deployed. Thus the case cannot give a time to deploy for each vendor.

The spreads are Multi-Party CVD (MPCVD) measures. The Foundations model has one vendor, so it has no spreads. Benchmarking MPCVD combines the outcomes of all vendors into one score, but does not decide how to combine them. The fix spread and the trailing-vendor gap show the cost of the slowest vendor directly. The fix spread shows how long the slowest vendor holds the case, and the trailing-vendor gap shows whether the embargo waited for the slowest vendor.


Disclosure margins

Each desideratum gives a margin. These are the margins most directly tied to defender outcomes.

Margin What a positive value measures
\(m(\evF \prec \evP)\) Lead time between a fix being ready and the vul being public
\(m(\evD \prec \evP)\) Lead time between deployment and the vul being public
\(m(\evP \prec \evX)\) Time between the vul being public and an exploit being public
\(m(\evX \prec \evA)\) Time between a public exploit and observed attacks

Exposure

Some states are bad to be in. For these states, the time spent in the state matters more than the order of entry. Exposure metrics are dwell times in those states.

Metric Definition
Public exposure Time during which \(\evP\) holds and \(\evD\) does not, per deployer
Exploit exposure Time during which \(\evX\) holds and \(\evD\) does not, per deployer
Unfixed exposure Time during which \(\evP\) holds and \(\evF\) does not, per vendor

Open Questions proposes a test of the vulnerability management of a deployer. The test compares two intervals that start at \(\evF\). The interval to \(\evD\) should be shorter than the interval to \(\evA\). Exploit exposure measures a related race as a duration: the time during which an exploit is public and the deployer has not deployed the fix.


Closure and case administration

Metric Definition
Case duration Case creation to the case being fully closed
Time open after disclosure \(\evP\) to the case being fully closed
Embargo Invites timed out at close Embargo Invites still awaiting a reply that the owner's close moves to TIMED_OUT (CM-23-014)
Ownership transfer latency An offer to transfer case ownership to its acceptance, rejection, withdrawal, or expiry

A case is fully closed when the Case Owner leaves the case and the CASE_MANAGER closes its own report (CM-23-002).


Process health

These metrics describe the protocol's own operation in a case, not the CVD outcome. They help separate a slow case from a case whose messages went missing.

Metric Definition
Gap rate RM status updates from participants that skip a step, which the CASE_MANAGER accepts and notes on the case (RSH-06-004)
Regression count RM status updates from participants that move backward, which the CASE_MANAGER refuses and notes (RSH-06-002)
Refusal count Assertions the CASE_MANAGER refused rather than admitted
Replay count Requests from participants to send missed ledger entries again, because their replica fell behind
Clock skew Commit stamp minus claimed time, per participant

A gap is a forward move that skips a step the CASE_MANAGER never saw. Either a message was lost, or the participant took the step locally and reported only its latest state. Thus a rising gap rate is a reason to look for lost messages, but it does not prove that messages were lost. A large clock skew alone does not mean that the claimed times of a participant are wrong (Which clock to use).


Gaps in the data

Most of the metrics above can be computed from the case ledger as it is. Some metrics are not exact, because the ledger stamps some events late, stamps some events with the creation time, or does not record some events. What a Case Records lists these data gaps. A metric that uses an event with a data gap has the same gap.


Intervals to events that have not happened

Many cases never see \(\evX\) or \(\evA\), and an open case has not yet closed. An interval that ends at an event that has not occurred is censored. A censored interval is not missing: the interval is at least as long as the time from its first event to now, or to the close of the case. A measure that treats censored intervals as missing biases every average toward the cases that finished.

The close of a case censors its intervals for good. After the case closes, the ledger cannot record \(\evX\) or \(\evA\), even if a public exploit or attacks occur later. A margin whose second event never occurred is censored in the same way. For example, a closed case with \(\evF\) and no \(\evX\) on its ledger achieved \(\evF \prec \evX\), by at least the time from \(\evF\) to the close of the case. This result is only as good as the reports: a public exploit that no participant reported is not on the ledger.

This page does not choose a method for censored data. Survival-analysis methods, such as a Kaplan–Meier estimate of time to fix, are one idea, as are medians rather than means.


Relation to service level expectations

Open Questions and Policy Formalization describe Service Level Expectations (SLEs). An SLE is a timer that starts and stops on transitions, such as a limit on the time from \(\evV\) to \(\evF\). An SLE is a threshold on one of the intervals above. When the intervals have definitions first, a program can state an SLE as a number against a metric that already has a definition. The program can also report the distribution of an interval across its cases, rather than only how often it met a threshold. An SLE uses the same clock as its interval, so the program must say whether the SLE is on commit stamps or claimed times (Which clock to use).


In a two-party case

A two-party case has one reporter and one vendor. A two-party case is a special case of the same model, not a different model. The table below shows how each group of metrics changes in a two-party case.

Metrics In a two-party case
Intake and triage for the vendor, all embargo metrics, time to fix, closure, process health No change
Disclosure margins without \(\evD\) No change
Notification spread, fix spread Always zero, because the case has one vendor
Trailing-vendor gap The interval from the \(\evF\) of the one vendor to \(\evEx\)
Time to deploy, margins with \(\evD\), public exposure, exploit exposure Not available, unless the vendor also has the Deployer role

The deployment metrics need a participant with a deployment state. A vendor has a deployment state only when the vendor also has the Deployer role (SDO-03-002).


A starting set for a CVD program

The page lists many metrics. A CVD program that wants a short set can start with the metrics below.

Metric Why it is in the starting set
Time to fix The vendor's response, and the interval that SLEs most often limit
\(m(\evF \prec \evP)\) The warning that defenders get before the vul is public
Planned and actual embargo length Whether embargoes last as long as they were planned to last
Time left at exit and exit cause Why an embargo ended, and what an early end cost
Time to deploy How fast a fix reaches the deployers, where the risk is
Notification spread How fast the case reaches every vendor in an MPCVD case
Case duration The total effort and time that a case takes

The other metrics are mainly for research or for tuning the protocol. These metrics include the process-health metrics, the pocket-veto rate, the lapse count, the exposure dwell times, and report to case.


Summary

Area Representative metrics
Intake and triage Time to validate, accept, judge, join; report to case
Embargo Decision latency, revisions, planned and actual length, time left at exit, exit cause, response latency, pocket-veto rate
Fix and deployment Time to fix and deploy; notification spread, fix spread, trailing-vendor gap
Disclosure margins \(m(\evF \prec \evP)\), \(m(\evD \prec \evP)\), \(m(\evP \prec \evX)\), and the margin of every other desideratum
Exposure Time with \(\evP\) or \(\evX\) and no \(\evD\)
Closure Case duration, time open after disclosure, embargo Invites timed out at close, transfer latency
Process health Gap rate, regressions, refusals, replays, clock skew

Further reading