Desiderata from the Process Models
This page is not normative
This page is not considered a core part of the Vultron Protocol as proposed in the main documentation. Although within the page we might provide guidance in terms of SHOULD, MUST, etc., the content here is not normative.
This page proposes new desiderata for Coordinated Vulnerability Disclosure (CVD) cases. A desideratum is an order of two events that a case should achieve, for example a fix that is ready before the vul is public.
The original research gives twelve desiderata over six events of the Case State (CS) model. Since then, the protocol has added detailed models of Report Management (RM), Embargo Management (EM), Participant Embargo Consent (PEC), and multi-party participation. These models give more preferences about which events should occur before which other events.
The page is a draft research proposal for people who study the CVD process. No part of the page is normative, and no one has tested the new desiderata against case data.
The page uses the events and symbols of What a Case Records, which also tells how the case ledger records each event. The table below repeats these events:
| Symbol | Event | Belongs to |
|---|---|---|
| \(\evR_i\) | Participant \(i\) receives the report (report receipt) | Participant \(i\) |
| \(\evVa_i\) | Participant \(i\) decides that the report is valid (report validation) | Participant \(i\) |
| \(\evAc_i\) | Participant \(i\) accepts the report for work (report acceptance) | Participant \(i\) |
| \(\evC_i\) | Participant \(i\) closes the report (report closure) | Participant \(i\) |
| \(\evJ_i\) | Participant \(i\) joins the case (joining) | Participant \(i\) |
| \(\evS_i\) | Participant \(i\) agrees to the active embargo (embargo agreement) | Participant \(i\) |
| \(\evV_i\) | Vendor \(i\) becomes aware of the vul (vendor awareness) | Vendor \(i\) |
| \(\evF_i\) | Vendor \(i\) has a fix ready (fix readiness) | Vendor \(i\) |
| \(\evD_k\) | Deployer \(k\) deploys a fix (fix deployed) | Deployer \(k\) |
| \(\evP\) | The public becomes aware of the vul (public awareness) | The case |
| \(\evX\) | An exploit becomes public (exploit public) | The case |
| \(\evA\) | Someone observes attacks that use the vul (attacks observed) | The case |
| \(\evE\) | An embargo starts on the case (embargo start) | The case |
| \(\evEx\) | The embargo ends (embargo exit) | The case |
The subscript \(i\) names one participant or one vendor, and the subscript \(k\) names one deployer. Without a subscript, a symbol means the event for any participant, or for the one vendor of a two-party case. The notation \(a \prec b\) means that event \(a\) occurs before event \(b\). On the Measuring CVD pages, hover over an event symbol to see its name and meaning.
The page also uses the skill coefficient (\(\alpha_d\)) of Discriminating Skill and Luck. The skill coefficient compares how often a desideratum \(d\) holds in observed cases with how often \(d\) holds by chance. The process models that the page uses are RM, EM, and their interactions with CS.
The twelve original desiderata
The original research orders six events. The six events are vendor awareness (V), fix readiness (F), fix deployment (D), public awareness (P), a public exploit (X), and observed attacks (A). The table below gives its twelve desiderata, from the most desirable to the least desirable. Desirable Histories calls this set of twelve desiderata \(\mathbb{D}\), and What Does Success Mean in CVD? explains each desideratum.
| Ordering Preference | Notation |
|---|---|
| Fix Deployed Before Public Awareness | \(\evD \prec \evP\) |
| Fix Ready Before Public Awareness | \(\evF \prec \evP\) |
| Fix Deployed Before Exploit Public | \(\evD \prec \evX\) |
| Fix Deployed Before Attacks Observed | \(\evD \prec \evA\) |
| Fix Ready Before Exploit Public | \(\evF \prec \evX\) |
| Vendor Awareness Before Public Awareness | \(\evV \prec \evP\) |
| Fix Ready Before Attacks Observed | \(\evF \prec \evA\) |
| Public Awareness Before Exploit Public | \(\evP \prec \evX\) |
| Exploit Public Before Attacks Observed | \(\evX \prec \evA\) |
| Public Awareness Before Attacks Observed | \(\evP \prec \evA\) |
| Vendor Awareness Before Exploit Public | \(\evV \prec \evX\) |
| Vendor Awareness Before Attacks Observed | \(\evV \prec \evA\) |
The twelve desiderata treat each event as one event for the whole case. In a case with more than one vendor or deployer, vendor awareness, fix readiness, and fix deployment occur once for each vendor or deployer. The new desiderata below use the subscripts of the event table to name each vendor and each deployer.
A desideratum is useful only if a case can fail it
A desideratum is useful as a measure of skill only if a case can fail it. The skill coefficient compares how often an order holds with how often the order holds by chance. A conforming case is a case in which every participant follows the protocol. If the protocol enforces an order, the order holds in every conforming case. Then the observed frequency of the order is 1, and its skill coefficient tells nothing about skill.
The process models contain rules of two kinds:
- A MUST is a requirement that the implementation enforces. An order that a MUST sets is a conformance ordering. A history that breaks a conformance ordering shows a broken implementation, not a case without skill.
- A SHOULD, or a preference that a model states without a requirement, is not enforced. A conforming case can break it, so it is a candidate desideratum.
The candidates below come from the second kind of rule. The end of the page lists the conformance orderings, because a check of these orderings can still find defects.
Events beyond the six
The process models add the events below to the six original events. The table gives the step of the process model at which each event occurs.
| Event | Process model step |
|---|---|
| Report receipt (\(\evR_i\)) | RM Start → Received |
| Report validation (\(\evVa_i\)) | RM → Valid |
| Report acceptance (\(\evAc_i\)) | RM → Accepted |
| Report closure (\(\evC_i\)) | RM → Closed |
| Embargo start (\(\evE\)) | EM → Active, the first time on the case |
| Embargo exit (\(\evEx\)) | EM → eXited |
| Joining (\(\evJ_i\)) | Acceptance of the Stub Invite to the case (CM-11-001) |
| Embargo agreement (\(\evS_i\)) | PEC → AGREED |
The CASE_MANAGER records each of these events as an entry in the case ledger. One exception is the statuses that the case gets when it is created. These statuses show the creation time, not the time of the decision (What a Case Records).
The embargo start (\(\evE\)) and the embargo exit (\(\evEx\)) belong to the whole case, because a case has at most one active embargo. When an embargo is active, \(\evJ_i\) and \(\evS_i\) are one event. The reason is that acceptance of the Stub Invite both joins the case and agrees to the embargo (CM-11-001). Joining and embargo agreement are separate events only for a participant that joins when no embargo is active, or that answers a later embargo.
Splitting embargo exit by cause
The same change of state can be a success or a failure, depending on its cause. Thus the embargo exit is split into three outcomes by cause.
| Split | Meaning |
|---|---|
| Embargo exit on its own terms (\(\evExOwn\)) | The embargo ends because its end time is reached, or ends early because fixes are ready or deployed |
| Embargo exit on a threat signal (\(\evExThreat\)) | The embargo ends because of public awareness, a public exploit, or observed attacks |
| Other early embargo exit (\(\evExOther\)) | The embargo ends early for another reason, or for no recorded reason |
The protocol records this split.
A terminated embargo records its end: END_TIME_REACHED or EARLY.
When the cause is known, the terminated embargo also records a structured cause: THREAT_SIGNAL, FIX_READY, FIX_DEPLOYED, or OTHER (EMB-13-010).
END_TIME_REACHED, FIX_READY, and FIX_DEPLOYED are \(\evExOwn\).
THREAT_SIGNAL is \(\evExThreat\).
An early end with the cause OTHER, or with no recorded cause, is \(\evExOther\).
Only the CASE_MANAGER can record THREAT_SIGNAL, when a case status update sets public awareness, a public exploit, or observed attacks (EMB-13-005).
Thus \(\evExOther\) is never a recorded threat signal.
Public awareness is one event
Public awareness (\(\evP\)) is not split by its source. A coordinated publication by a participant and a leak are different causes, but the case cannot tell them apart.
A participant reports public awareness with a case status update that sets P. The status update asserts that information about the vul is public. It does not say why the information is public. A participant can explain the cause in the free text of the activity, but nothing structured records it.
The same is true of a public exploit (\(\evX\)). A status update that sets X does not say who published the exploit. Thus a measure cannot tell an exploit that a participant published from any other public exploit.
Embargo desiderata
An embargo is a means, not an end. An embargo exists to give vendors time for fix readiness (\(\evF\)) and fix deployment (\(\evD\)). The embargo lets these events occur before \(\evP\), a public exploit (\(\evX\)), and observed attacks (\(\evA\)). Thus the embargo desiderata rank below the fix desiderata of the twelve original desiderata. A case that achieves \(\evF \prec \evP\) without an embargo is not a worse case.
| Ordering preference | Meaning | Source |
|---|---|---|
| \(\evE \prec \evP\), \(\evE \prec \evX\), \(\evE \prec \evA\) | An embargo is active before the vul is public, exploited, or attacked | A case starts in EM Active at creation, under a published default or the protocol default (Default Embargoes) |
| \(\evExOwn \prec \evP\), \(\evExOwn \prec \evX\), \(\evExOwn \prec \evA\) | The embargo ends on its own terms, not because a threat signal ended it | An embargo ends early at \(\evP\), \(\evX\), or \(\evA\) (CVD Case State Interactions with the RM and EM Process Models) |
| \(\evF_i \prec \evExOwn\) for every vendor \(i\) | Every vendor's fix is ready before the embargo ends on its own terms | VP-14-009 asks participants to extend for trailing vendors; VP-09-007 permits ending at a quorum |
An embargo before public awareness
The prototype assumes that every case starts with an embargo. The embargo is a default that the participants publish, or the protocol default. Under this assumption, a new case starts in EM Active, unless \(\evP\), \(\evX\), or \(\evA\) already occurred. In that condition, the case stays in EM None (EP-04-008, EP-04-020). Thus \(\evE \prec \evP\) fails only when the report arrives, or the case is created, after \(\evP\), \(\evX\), or \(\evA\) already occurred.
An implementation that does not start every case with an embargo can fail this desideratum in more ways. For example, the participants can be slow to agree to an embargo, and the vul can become public before they agree. Thus the desideratum is itself an argument for default embargoes. A default embargo is a mechanism that makes the case achieve the desideratum more often.
An embargo that ends on its own terms
The desideratum of an embargo exit on its own terms before public awareness, a public exploit, and observed attacks is the most useful of these. Public awareness and a public exploit each end an active embargo (VP-09-001). Thus any \(\evEx\) after \(\evP\) or \(\evX\) is \(\evExThreat\). Observed attacks SHOULD also end an active embargo (VP-09-003), and the CASE_MANAGER does not commit an active embargo beside observed attacks (CSB-18-004). Thus \(\evEx\) after \(\evA\) is also \(\evExThreat\).
The desideratum \(\evExOwn \prec \evP\) shows that the embargo ended as planned. Disclosure did not force the end of the embargo. An embargo that ends as \(\evExOther\) fails the desideratum, because it did not end as planned. But \(\evExOther\) is not a threat signal, so the measure does not count it as \(\evExThreat\).
This desideratum also covers two preferences that the measure cannot observe directly. Participants SHOULD NOT publish while an embargo is active (VP-05-006, VP-05-014), and exploit publishers SHOULD NOT release exploit code while an embargo is active (VP-14-012). The case cannot tell who made the vul public or who published the exploit. But a participant that publishes during the embargo sets \(\evP\) or \(\evX\), and the embargo ends on a threat signal. Thus the case fails \(\evExOwn \prec \evP\) or \(\evExOwn \prec \evX\).
Every vendor's fix before the embargo ends on its own terms
The desideratum \(\evF_i \prec \evExOwn\) for every vendor is stronger than the protocol requires. VP-09-007 lets an embargo end when a quorum of vendors has a fix ready, even if some vendors do not. A vendor without a fix ready at that time is a trailing vendor.
The desideratum gives the higher goal: a fix ready from every vendor before the embargo ends. A case that ends its embargo at a quorum fails the desideratum once for each trailing vendor. The measure should count each of these failures.
A lesser outcome is also worth observing.
The Case Owner can end the embargo early because a quorum of vendors has a fix ready.
The terminated embargo then records the cause FIX_READY, with the vendors whose readiness made the quorum (EMB-13-010).
But the protocol does not define a quorum.
The FIX_READY cause is a claim of the Case Owner, and the CASE_MANAGER records it as given (EMB-13-005).
Thus a measure can count the cases that ended at a claimed quorum, but it cannot judge whether the quorum was sufficient.
A rule that this page does not propose
A participant SHOULD NOT close its report while an embargo is active, unless it first tells the others that it is closing and whether its Embargo Consent continues (RMB-14-003). This page does not make \(\evEx \prec \evC_i\) a desideratum. The desideratum would reward a bad pattern: a participant publishes early, the embargo ends, and then the participant closes its report.
The other desiderata already cover the purpose of the rule. A participant that publishes early makes the case fail \(\evExOwn \prec \evP\). A vendor that closes its report without a fix fails \(\evF_i \prec \evExOwn\). Also, a closed participant that agreed to the embargo is still bound by the embargo.
The Case Owner cannot close the case while an embargo is active, because the CASE_MANAGER refuses the close (CM-23-011). Thus the rule that the Case Owner does not close the case while an embargo is active is a conformance ordering, not a desideratum.
Report-handling desiderata
The twelve original desiderata treat vendor awareness as the first good event. The RM model separates awareness from engagement: a vendor can know of a report and still not validate it, accept it, or work on it.
The desiderata below count only the vendors whose reports reach Valid. The case does not record which vendors the vul affects. A vendor that the vul does not affect decides that the report is invalid, and a measure cannot tell this decision from a wrong decision about a real vul. Thus these desiderata measure only the timing of the decisions. Whether a decision of Invalid was correct is a different preference, validation accuracy, in Preferences about the path.
| Ordering preference | Meaning | Source |
|---|---|---|
| \(\evAc_i \prec \evP\), \(\evAc_i \prec \evX\), \(\evAc_i \prec \evA\) for every vendor \(i\) whose report reaches Valid | The vendor has committed to work on the report before the vul is public, exploited, or attacked | Reward Functions prefers RM paths that pass through Accepted |
| \(\evVa_i \prec \evP\) for every vendor \(i\) whose report reaches Valid | A vendor that decides the report is valid does so before the vul is public | Validation is the first RM decision |
| \(\evAc_i \prec \evF_i\) for every vendor \(i\) | The vendor accepts the report before its fix is ready, rather than deferring or closing it first | CSB-18-001 also permits a fix from Deferred or Closed |
The desideratum \(\evVa_i \prec \evP\) is a weaker form of \(\evF_i \prec \evP\). A vendor usually decides that the report is valid before its fix is ready. Thus a vendor with a fix ready before public awareness usually also has a validation before public awareness.
There is one exception. A vendor can report fix readiness when its RM state is Closed (CSB-18-001), and the RM state can go from Invalid to Closed without Valid. Thus a vendor that decides the report is invalid, closes it, and later reports a fix has a fix without a validation. The protocol permits this history, but it is rare.
The desideratum \(\evAc_i \prec \evP\) implies awareness of vendor \(i\) before public awareness (\(\evV_i \prec \evP\)). The reason is that a vendor cannot accept a report that it has not received. Thus report acceptance before public awareness is the stronger desideratum. It rewards a vendor that does work on the report, not a vendor that only knows of the report. Report acceptance also measures the period between \(\evV\) and \(\evF\), where the twelve original desiderata have nothing to measure.
The desideratum \(\evAc_i \prec \evF_i\) is only partly enforced. A vendor can report fix readiness only when its RM state is Accepted, Deferred, or Closed (CSB-18-001). But the RM state can go from Valid to Deferred, or from Received to Closed, without Accepted. A vendor that defers a report and then releases a fix follows the protocol, but fails the desideratum. The measure should count this failure.
Multi-party desiderata
Benchmarking MPCVD applies the twelve original desiderata to each vendor separately, and then combines the results. The desiderata below relate the events of one participant to the events of a different participant. Thus a history of one vendor cannot express these desiderata.
| Ordering preference | Meaning | Source |
|---|---|---|
| \(\evV_i \prec \evF_j\) for every vendor \(i\) whose report reaches Valid and every vendor \(j\) | Every vendor whose report reaches Valid learns of the vul before any vendor's fix is ready | Keeps the trailing vendor's time to fix from starting late |
| \(\evF_i \prec \evP\) for every vendor \(i\) | Every vendor's fix is ready before the vul is public | \(\evF \prec \evP\) from \(\mathbb{D}\), quantified over vendors |
The desideratum \(\evF_i \prec \evP\) for every vendor states one choice for how to combine the results of the vendors. Benchmarking MPCVD does not choose how to combine these results, and gives a majority rule and a risk-weighted rule as options. The rule "every vendor" is the strictest rule, and the embargo desiderata above already use this rule.
Every vendor aware before any fix is ready is contested
The desideratum \(\evV_i \prec \evF_j\) for every vendor \(i\) whose report reaches Valid and every vendor \(j\) is contested. The planning interview for issue #6450 found three objections:
- It is a proxy for an interest the per-vendor desiderata already measure. The trailing vendor's own \(\evF_i \prec \evP\) and \(\evD_i \prec \evP\) measure what matters to that vendor. A fix that is ready but not released costs the trailing vendor nothing in itself.
- It is backwards in supply-chain cases. A downstream vendor often cannot fix until the upstream fix exists. The downstream vendor often learns of the vul because of the upstream fix.
- Its chance baseline is 393/400, so it holds almost always by chance. Its skill coefficient tells little.
The desideratum remains in the baseline table because the research may still use it. Issue #6501 proposes an optional dependency graph between participants, which could replace it. Move advice does not use \(\evV_i \prec \evF_j\) (ADR-0200).
Deployers
This page does not propose a desideratum about when deployers join the case. Fix deployment can be a better measure of the vulnerability management that follows CVD than of CVD itself.
Some cases have deployers anyway. For example, a vendor that runs a cloud service also deploys its own fix, and so does an organization that runs its own vulnerability disclosure program. When a case has deployers, a case in which they join before public awareness (\(\evJ_k \prec \evP\)) is better than a case in which they join after. A deployer that joins only after \(\evP\) cannot achieve \(\evD_k \prec \evP\), however quickly it acts.
But this order is often not achievable, for three reasons:
- It is difficult to tell deployers from the public.
- It is difficult to be fair: a case that includes some deployers must decide which deployers to exclude.
- When the group of deployers becomes large enough, telling the group is the same as telling the public.
Even so, a case can include a small number of large deployers, such as service providers, before publication. In past cases, this action has protected critical infrastructure before the fix reached everyone else.
The protocol does not require a case to invite a deployer, and the case does not record which fix a deployer deployed. Thus a measure sees only the deployers that join, and a case that invites no deployers cannot fail an order about deployers. ADR-0182 scores a multi-party case per vendor, and leaves a participant with no vendor path, such as a deployer, out of scope until it has measures of its own.
Preferences about the path
Some preferences are not orderings of two events. These preferences are about the shape of a history: how many steps the history has, or how the participants reached a decision.
| Preference | Measure |
|---|---|
| A participant answers an embargo Invite before its RSVP deadline, rather than leaving it to a pocket veto | For each participant, the Invites answered before they time out, out of those Invites and the Invites that time out at their RSVP deadline, not counting the Invites the case's close times out before their deadline (CM-23-014) |
| A case revises its embargo few times | For the case, the number of embargo revisions that the Case Owner activates, not counting the replacement of a default embargo |
| An embargo is as short as the fixes allow (EM Principles) | For the case, the embargo duration divided by the time from embargo start until the last vendor's fix is ready |
| A report is judged correctly: a real vul is Valid and a bogus report is Invalid | For each participant, the validation decisions known to be correct out of those whose correctness is known |
Reward Functions defines each measure and the events that it reads. Each measure is a simple count or ratio from the event history, and no measure combines them into one reward function (ADR-0182). The preferences come from the criteria for reward functions over RM and EM histories in the original research, so this page lists them to make the list complete, not as new proposals. Order desiderata and path preferences need different measures. An order desideratum fits the skill coefficient (\(\alpha_d\)), which counts how often the order holds. A path preference is not an order of two events, so it has a count or ratio of its own instead.
Conformance orderings
The orderings below are enforced by the protocol. Conformance orderings are not desiderata, because a conforming case cannot fail them. But a check of case data against the conformance orderings is still useful. A history that breaks a conformance ordering shows a defect in an implementation.
| Ordering | Requirement |
|---|---|
| \(\evV_i \prec \evF_i\) for every vendor \(i\) | The vendor fix path (History Constraints) |
| \(\evF_i \prec \evD_k\) for some vendor \(i\), for every deployer \(k\) that is not also a vendor | CSB-15-004 |
| \(\evF_i\) only once participant \(i\)'s RM state is Accepted, Deferred, or Closed | CSB-18-001 |
| A report is never closed from Valid | VP-02-004, RMB-14-008 |
| The case is initialized before anyone else is invited and before any embargo change | CM-14-007, CM-14-014 |
| An embargo end time is set before an embargo-eligible case reaches RM Valid | DUR-07-004 |
| \(\evS_k\) before any case content sent to \(k\), while an embargo is active | CM-10-004, CM-10-005 |
| The Case Owner does not close the case while an embargo is active | CM-23-011 |
Case content includes case ledger entries, case announcements, status broadcasts, and embargo announcements. Participants SHOULD NOT share the report by other routes with a participant that has not agreed to the embargo either (VP-08-006, EMB-11-001). The case cannot observe information shared by other routes, so a measure of the case cannot check this preference.
Chance baselines
The skill coefficient needs a baseline: the frequency (\(f_d\)) with which each desideratum holds by chance. For the twelve original desiderata, the baseline comes from a random walk over the six-event CS model, in which each permitted next event has the same probability. The new events are in the RM, EM, and CS models together, and rules connect these models. A walk that ignores the rules gives baselines for histories that cannot occur.
ADR-0182 takes each baseline from the same walk over the smallest model that holds the events of the desideratum. Thus each baseline below is a one-vendor chance baseline, and the baseline of \(\evV_i \prec \evF_j\) is a two-vendor chance baseline. The baseline measures skill against a case with one vendor (or two) that chance handles, not against a chance version of the case at hand. A desideratum that must hold for every vendor is scored for each vendor against the one-vendor baseline. Thus one table serves every case, whatever its number of participants.
The models
The one-vendor model holds one vendor's RM state and fix path, the case's public awareness, exploit, and attack events, and the embargo. The two-vendor model holds two vendors with the same shared events and embargo. The models do not hold embargo consent or joining, because no ordering desideratum on this page names those events.
The walk obeys every rule that the protocol enforces:
- The RM, CS, and EM transition tables.
- Vendor awareness and report receipt are one event (Vendor Notification).
- A fix is ready only after the vendor's report is Accepted, Deferred, or Closed (CSB-18-001).
- No embargo is active beside public awareness, a public exploit, or observed attacks (CSB-18-002, CSB-18-003, CSB-18-004). Thus each of these events ends an active embargo in the same step, as \(\evExThreat\).
- Each of these events also cancels a pending embargo proposal (EMB-16-001), and after any of them no new embargo is proposed.
An embargo that ends without a threat signal ends either on its own terms or otherwise, with the same probability. A preference that the protocol states only as a SHOULD does not constrain the walk, because the desiderata measure exactly those preferences.
The walk is solved exactly, as an absorbing Markov chain, and not by simulation. The same computation, run on the six-event model, gives the published baselines of the twelve original desiderata (A Random Walk through CVD States).
The baselines
| Group | Desideratum | Measured over | Model | \(f_d\) | Exact |
|---|---|---|---|---|---|
| Embargo | \(\evE \prec \evP\) | every case | one vendor | 0.087 | 2/23 |
| Embargo | \(\evE \prec \evX\) | every case | one vendor | 0.087 | 2/23 |
| Embargo | \(\evE \prec \evA\) | every case | one vendor | 0.087 | 2/23 |
| Embargo | \(\evExOwn \prec \evP\) | every case | one vendor | 0.017 | 2/115 |
| Embargo | \(\evExOwn \prec \evX\) | every case | one vendor | 0.017 | 2/115 |
| Embargo | \(\evExOwn \prec \evA\) | every case | one vendor | 0.017 | 2/115 |
| Embargo | \(\evF_i \prec \evExOwn\) | every vendor, each scored alone, in cases whose embargo ends on its own terms | one vendor | 0.046 | — |
| Report handling | \(\evAc_i \prec \evP\) | vendors whose report reaches Valid | one vendor | 0.052 | 5/96 |
| Report handling | \(\evAc_i \prec \evX\) | vendors whose report reaches Valid | one vendor | 0.198 | 16/81 |
| Report handling | \(\evAc_i \prec \evA\) | vendors whose report reaches Valid | one vendor | 0.222 | 2/9 |
| Report handling | \(\evVa_i \prec \evP\) | vendors whose report reaches Valid | one vendor | 0.167 | 1/6 |
| Report handling | \(\evAc_i \prec \evF_i\) | every vendor, each scored alone | one vendor | 0.333 | 1/3 |
| Multi-party | \(\evV_i \prec \evF_j\) | vendor \(i\) whose report reaches Valid, and every other vendor \(j\) | two vendors | 0.983 | 393/400 |
| Multi-party | \(\evF_i \prec \evP\) | every vendor, each scored alone | one vendor | 0.042 | 1/24 |
The column "Exact" gives the baseline as a fraction when the fraction is short.
The rules that connect the models change the baselines, and the change can be large. For example, \(\evF \prec \evP\) has the baseline \(1/9\) in the six-event model. In the one-vendor model, the fix must wait for the vendor to accept the report, so the baseline of \(\evF_i \prec \evP\) is \(1/24\).
The baseline of \(\evV_i \prec \evF_j\) is near 1. By chance, a second vendor seldom has a fix ready before the first vendor learns of the vul. Thus a case can show skill on this desideratum only by a small margin, and a case that fails it is notable.
Open problems
The desiderata on this page are proposals. The problems below remain before these desiderata can have a measure like the skill coefficient of Discriminating Skill and Luck.
The desiderata are not independent
Some desiderata imply other desiderata:
- \(\evAc_i \prec \evP\) implies \(\evV_i \prec \evP\), and also implies \(\evVa_i \prec \evP\).
- Suppose that every vendor has a fix ready before an embargo exit on its own terms (\(\evF_i \prec \evExOwn\)). Suppose also that \(\evExOwn \prec \evP\). Then every vendor has a fix ready before public awareness (\(\evF_i \prec \evP\)).
The twelve original desiderata have the same kind of dependence. For example, \(\evF \prec \evP\) implies \(\evV \prec \evP\), because a vendor is always aware of the vul before its fix is ready.
The skill score in Reasoning Over Histories adds \(-\log(f_d)\) for each desideratum \(d\) that a history meets. A stronger desideratum holds less often by chance than a weaker desideratum that it implies. Thus the stronger desideratum adds more to the score, and the weaker desideratum adds less. The score handles the dependence of the twelve original desiderata in this way, and the same method can apply to the new desiderata.
The method is not sufficient when a desideratum and a desideratum that it implies have nearly the same baseline. Then the score counts nearly the same skill two times. The table below lists each pair in which an extended desideratum implies another desideratum in the one-vendor model. The walk finds each pair by a search of the model, not by an argument. The table leaves out a pair in which the weaker desideratum always holds within the scope of the pair.
The table gives the frequency of each desideratum of the pair in the one-vendor model, among histories in the scope of both desiderata. Thus these frequencies can differ from the baselines in the table above, when the two desiderata have different scopes or one of them is an original desideratum. The column "Share" is the ratio of the two frequencies. It gives the chance that a history that meets the weaker desideratum also meets the stronger desideratum. The table flags a pair whose share is 90% or more as counting the same skill twice.
| Stronger | Weaker | \(f\) stronger | \(f\) weaker | Share | Counts twice (share ≥ 90%) |
|---|---|---|---|---|---|
| \(\evE \prec \evP\) | \(\evE \prec \evX\) (equivalent) | 0.087 | 0.087 | 1.000 | yes |
| \(\evE \prec \evP\) | \(\evE \prec \evA\) (equivalent) | 0.087 | 0.087 | 1.000 | yes |
| \(\evE \prec \evX\) | \(\evE \prec \evA\) (equivalent) | 0.087 | 0.087 | 1.000 | yes |
| \(\evExOwn \prec \evP\) | \(\evE \prec \evP\) | 0.017 | 0.087 | 0.200 | no |
| \(\evExOwn \prec \evP\) | \(\evE \prec \evX\) | 0.017 | 0.087 | 0.200 | no |
| \(\evExOwn \prec \evP\) | \(\evE \prec \evA\) | 0.017 | 0.087 | 0.200 | no |
| \(\evExOwn \prec \evP\) | \(\evExOwn \prec \evX\) (equivalent) | 0.017 | 0.017 | 1.000 | yes |
| \(\evExOwn \prec \evP\) | \(\evExOwn \prec \evA\) (equivalent) | 0.017 | 0.017 | 1.000 | yes |
| \(\evExOwn \prec \evX\) | \(\evE \prec \evP\) | 0.017 | 0.087 | 0.200 | no |
| \(\evExOwn \prec \evX\) | \(\evE \prec \evX\) | 0.017 | 0.087 | 0.200 | no |
| \(\evExOwn \prec \evX\) | \(\evE \prec \evA\) | 0.017 | 0.087 | 0.200 | no |
| \(\evExOwn \prec \evX\) | \(\evExOwn \prec \evA\) (equivalent) | 0.017 | 0.017 | 1.000 | yes |
| \(\evExOwn \prec \evA\) | \(\evE \prec \evP\) | 0.017 | 0.087 | 0.200 | no |
| \(\evExOwn \prec \evA\) | \(\evE \prec \evX\) | 0.017 | 0.087 | 0.200 | no |
| \(\evExOwn \prec \evA\) | \(\evE \prec \evA\) | 0.017 | 0.087 | 0.200 | no |
| \(\evF_i \prec \evExOwn\) | \(\evVa_i \prec \evP\) | 0.027 | 0.423 | 0.065 | no |
| \(\evF_i \prec \evExOwn\) | \(\evF_i \prec \evP\) | 0.046 | 0.145 | 0.317 | no |
| \(\evF_i \prec \evExOwn\) | \(\evF \prec \evX\) | 0.046 | 0.288 | 0.160 | no |
| \(\evF_i \prec \evExOwn\) | \(\evV \prec \evP\) | 0.046 | 0.624 | 0.074 | no |
| \(\evF_i \prec \evExOwn\) | \(\evF \prec \evA\) | 0.046 | 0.301 | 0.153 | no |
| \(\evF_i \prec \evExOwn\) | \(\evV \prec \evX\) | 0.046 | 0.812 | 0.057 | no |
| \(\evF_i \prec \evExOwn\) | \(\evV \prec \evA\) | 0.046 | 0.859 | 0.054 | no |
| \(\evAc_i \prec \evP\) | \(\evAc_i \prec \evX\) | 0.052 | 0.198 | 0.264 | no |
| \(\evAc_i \prec \evP\) | \(\evVa_i \prec \evP\) | 0.052 | 0.167 | 0.312 | no |
| \(\evAc_i \prec \evP\) | \(\evV \prec \evP\) | 0.052 | 0.333 | 0.156 | no |
| \(\evAc_i \prec \evP\) | \(\evV \prec \evX\) | 0.052 | 0.667 | 0.078 | no |
| \(\evAc_i \prec \evX\) | \(\evV \prec \evX\) | 0.198 | 0.667 | 0.296 | no |
| \(\evAc_i \prec \evA\) | \(\evV \prec \evA\) | 0.222 | 0.750 | 0.296 | no |
| \(\evVa_i \prec \evP\) | \(\evV \prec \evP\) | 0.167 | 0.333 | 0.500 | no |
| \(\evVa_i \prec \evP\) | \(\evV \prec \evX\) | 0.167 | 0.667 | 0.250 | no |
| \(\evF_i \prec \evP\) | \(\evVa_i \prec \evP\) | 0.028 | 0.167 | 0.167 | no |
| \(\evF_i \prec \evP\) | \(\evF \prec \evX\) | 0.042 | 0.185 | 0.225 | no |
| \(\evF_i \prec \evP\) | \(\evV \prec \evP\) | 0.042 | 0.333 | 0.125 | no |
| \(\evF_i \prec \evP\) | \(\evV \prec \evX\) | 0.042 | 0.667 | 0.062 | no |
The flagged pairs are equivalent: each desideratum of the pair implies the other. No embargo is active beside \(\evP\), \(\evX\), or \(\evA\). Thus an embargo can start only before all three events, and it can end on its own terms only before all three events. So \(\evE \prec \evP\), \(\evE \prec \evX\), and \(\evE \prec \evA\) are one desideratum, and the three \(\evExOwn\) desiderata are one desideratum. A score should count each group once.
In the model, \(\evExOwn \prec \evP\) holds whenever \(\evExOwn\) occurs. Thus \(\evF_i \prec \evExOwn\) alone implies \(\evF_i \prec \evP\).
The new events are easier to observe than the original events
What a Case Records tells that \(\evX\) and \(\evA\) occur outside the protocol. Open Questions tells that \(\evD\) needs a threshold before anyone can observe it.
Most of the new events are protocol acts. The CASE_MANAGER writes each of these protocol acts to the case ledger when it admits the act. These acts include \(\evAc_i\), \(\evE\), \(\evEx\), \(\evJ_i\), and \(\evS_i\). The exception is the statuses that the case gets when it is created, which show the creation time.
Thus a measure can get a desideratum between two protocol acts from the ledger alone. A desideratum that uses \(\evX\) or \(\evA\) has the same observation problems as the original desiderata.
In a two-party case
A two-party case has one reporter and one vendor. A two-party case is a special case of the same model, not a different model. The table below shows how each group of desiderata changes in a two-party case.
| Desiderata | In a two-party case |
|---|---|
| Embargo desiderata | No change. Vendor \(i\) is always the one vendor. |
| Report-handling desiderata | No change. Vendor \(i\) is always the one vendor. |
| \(\evV_i \prec \evF_j\) | Becomes \(\evV \prec \evF\), which the vendor fix path already enforces |
| \(\evF_i \prec \evP\) for every vendor | Becomes \(\evF \prec \evP\) for the one vendor |
| Embargo agreement before case content (a conformance ordering) | Always true for the reporter. The case makes the reporter a signatory of any active embargo when the case is created (CM-14-012). |
Summary
| Group | Desiderata |
|---|---|
| Embargo | \(\evE \prec \evP, \evX, \evA\); \(\evExOwn \prec \evP, \evX, \evA\); \(\evF_i \prec \evExOwn\) for every vendor |
| Report handling | For vendors whose reports reach Valid: \(\evAc_i \prec \evP, \evX, \evA\); \(\evVa_i \prec \evP\). For every vendor: \(\evAc_i \prec \evF_i\) |
| Multi-party | \(\evV_i \prec \evF_j\) for vendors \(i\) whose reports reach Valid (contested); \(\evF_i \prec \evP\) for every vendor |
| Path shape | Answered invitations, few revisions, short embargoes, accurate validation |
Further reading
- Desirable Histories — the twelve original desiderata and the partial order they induce.
- What a Case Records — how the case ledger records these events, and the gaps in the ledger.
- Timing Metrics Within a Case — measuring the intervals between these events, not only their order.
- Benchmarking MPCVD — per-vendor histories and how they might aggregate.
- CVD Case State Interactions with the RM and EM Process Models — the couplings between the models that these desiderata draw on.