Skip to content
Explanation

Measuring and Benchmarking MPCVD

This page is not normative

This page is not considered a core part of the Vultron Protocol as proposed in the main documentation. Although within the page we might provide guidance in terms of SHOULD, MUST, etc., the content here is not normative.

This page extends the skill measure and the benchmarks of the preceding pages to a case with more than one affected vendor or product. It assumes the set of desiderata \(\mathbb{D}\) from Desirable Histories, the skill coefficient \(\alpha_d\) from Discriminating Skill and Luck, and the benchmark constant \(c_d\) from Benchmarking CVD.

Multiparty Coordinated Vulnerability Disclosure (MPCVD) is the process of coordinating the creation, release, publication, and potentially the deployment of fixes for vulnerabilities across a number of vendors and their respective products. The need for MPCVD arises due to the inherent nature of the software supply chain CERT Guide to CVD. A vulnerability that affects a low-level component (such as a library or operating system API) can require fixes from both the originating vendor and any vendor whose products incorporate the affected component. Alternatively, vulnerabilities are sometimes found in protocol specifications or other design-time issues where multiple vendors may have each implemented their own components based on a vulnerable design.

State Tracking in MPCVD

Applying our state-based model to MPCVD requires a forking approach to the state tracking.

MPCVD Case State Tracking

At the time of discovery, the vulnerability is in state \(vfdpxa\). Known only to its finder, the vulnerability can be described by that singular state.

As it becomes clear that the vulnerability affects multiple vendors' products, both finder/reporters and coordinators might begin to track the state of each individual vendor as a separate instance of the model. For example, if 3 vendors are known to be affected, but only 1 has been notified, the case might be considered to be in a superposition of states \(\{Vfdpxa,vfdpxa,vfdpxa\}\).

Each vendor, in turn, might then ascertain whether they are able to produce fixes for all their available products at once, or if those fixes will be staggered over time. In either case, the vendor might track the case as a superposition of states for each affected product dependent on its fix readiness status. For example, a vendor might have four products affected, with two in the fix-ready state and two fixes still in progress. Should they opt for transparency into their internal process, they might communicate their status as the superposition of \(\{VFdpxa,VFdpxa,Vfdpxa,Vfdpxa\}\). Alternatively, they might choose to only share the lowest state across their products, which in this example would be \(\{Vfdpxa\}\).

The above example implies a need to expand our notation.

MPCVD Cases have Participant-Specific States

In the MPCVD case, we need to think of each state \(q \in \mathcal{Q}\) as a set of states \(q_M\):

\[ q_M \stackrel{\mathsf{def}}{=}\{ q_1,q_2,\dots,q_n \} \]

where \(q_i\) represents the state \(q\) for the \(i\)th affected vendor and/or product. For example, \(\{ {Vfdpxa}_1, {vfdpxa}_2 \}\) would represent the state in which vendor 1 has been notified but vendor 2 has not. We have more to say about this in Model Interactions, but we'll borrow the diagram from there to illustrate the concept here.

---
title: Case State Diagram showing Participant-Agnostic and Participant-Specific Aspects
---
stateDiagram-v2
    direction LR
    CS: Case State Model
    state CS {
        ps: Participant-Specific
        state ps {
            [*] --> vfd
            vfd --> Vfd : V
            Vfd --> VFd : F
            VFd --> VFD : D
            VFD --> [*]
        }
        --
        g: Participant-Agnostic 
        state g {
            [*] --> pxa

            pxa --> Pxa : P
            pxa --> pXa : X
            pxa --> pxA : A

            pXa --> PXa : P
            pXa --> pXA : A

            pxA --> PxA : P
            pxA --> pXA : X

            Pxa --> PxA : A
            Pxa --> PXa : X

            pXA --> PXA : P
            PXa --> PXA : A
            PxA --> PXA : X
            PXA --> [*]
        }
    }
    [*] --> CS
    CS --> [*]

State transitions across vendors need not be simultaneous. Very often, vendor notification occurs as new products and vendors are identified as affected in the course of analyzing a vulnerability report. So the individual events \(\mathbf{V}_i\) in \(\mathbf{V}_M\) (representing the status of all the vendor notifications) might be spread out over a period of time.

Some transitions are more readily synchronized than others. For example, if an MPCVD case is already underway, and information about the vulnerability appears in a public media report, we can say that \(\mathbf{P}_M\) occurred simultaneously for all coordinating vendors.

Regardless, in the maximal case, each vendor-product pair is effectively behaving independently of all the others. Therefore:

Dimensionality of an MPCVD Case

The maximum dimensionality of the MPCVD model for a case is

\[ D_{max} = 5 * N_{vprod} \]

where \(N_{vprod}\) represents the number of vendor-product pairs. The constant is five rather than the six substates of the Case State (CS) model, because the causal constraints \(D \Rightarrow F \Rightarrow V\) and \(X \Rightarrow P \Rightarrow V\) collapse those six substates into five dimensions.

This is undesirable, as it would result in a wide distribution of realized histories that more closely resemble the randomness assumptions of A Random Walk through CVD States than a skillful, coordinated effort. Further discussion of measuring MPCVD skill can be found in MPCVD Benchmarks below.

Reducing Dimensionality of MPCVD Cases is a good thing

For now, we posit that the goal of a good MPCVD process is to reduce the dimensionality of a given MPCVD case as much as is possible (i.e., to the dimensions of the single-vendor CVD case described in CS States). Experience shows that a full dimension reduction is unlikely in most cases, but that does not detract from the value of having the goal.

Reducing Complexity within a Vendor Organization

Vendors may be able to reduce their internal tracking dimensionality---which may be driven by things like component reuse across products or product lines---through in-house coordination of fix development processes. Within an individual vendor organization, PSIRTs are a common organizational structure to address this internal coordination process.

Regardless, the cross-vendor dimension is largely the result of component reuse across vendors, for example through the inclusion of third party libraries or OEM SDKs. Visibility of cross-vendor component reuse remains an unsolved problem, although efforts such as the Software Bill of Materials are promising in this regard. Thus, dimensionality reduction can be achieved through both improved transparency of the software supply chain and the process of coordination toward synchronized state transitions, especially for \(\mathbf{P}\), if not for \(\mathbf{F}\) and \(\mathbf{D}\) as well.

As a result of the dimensionality problem, coordinators and other parties to an MPCVD case need to decide how to apply disclosure policy rules in cases where different products or vendors occupy different case states with potentially contradictory recommended actions. For example, when four out of five vendors involved in a case have reached \(VFdpxa\) and are ready to publish, but the fifth is still in \(Vfdpxa\). Essentially this can be expected to take the form of a weighting function that acts on a vector of case states, and outputs a set of recommended actions derived from those states.

One possible function would be to apply a simple voting heuristic such as waiting for a simple majority of vendors to reach a state before taking action as that state recommends. In our 4/5 \(VFdpxa\) example, the coordinating parties would behave as if the case were in that state for all.

Another function could be to weight vendors and products by some other factor, such as size of the installed user base, or even based on a risk analysis of societal costs associated with potential actions. Here we acknowledge that our example is under-specified: does the fifth vendor (the one in \(Vfdpxa\)) represent a sizable fraction of the total user base? Or does it concentrate the highest risk use cases for the software? Challenges in efficiently assessing consistent answers to these questions are easy to imagine. The status quo for MPCVD appears consistent with defaulting to simple majority in the absence of additional information, with consideration given to the distribution of both users and risk on a case-by-case basis. At present, there is no clear consensus on such policies, although we hope that future work can use the model presented here to formalize the necessary analysis.

Integrating FIRST MPCVD Guidance

FIRST has published Guidelines and Practices for Multi-Party Vulnerability Coordination and Disclosure. Their guidance describes four use cases, along some with variations. Each use case variant includes a list of potential causes along with recommendations for prevention and responses when the scenario is encountered. A mapping of which use cases and variants apply to which subsets of states is given in the table below.

States FIRST Use Case Description
n/a 0 No vulnerability exists
\({VFDp\cdot\cdot}\) 1 Vulnerability with no affected users
\({V\cdot\cdot p\cdot\cdot}\) 1 Variant 1 Product is deployed before vulnerability is discovered or fixed
\({\cdot f\cdot\cdot\cdot\cdot}\) 2 Vulnerability with coordinated disclosure
\({\cdot fdP\cdot\cdot}\) 2 Variant 1 Finder makes the vulnerability details public prior to remediation
\({VFdP\cdot\cdot}\) 2 Variant 2 Users do not deploy remediation immediately
(multiparty sync) 2 Variant 3 Missing communication between upstream and downstream vendors
\({VfdP\cdot\cdot}\) 2 Variant 4 Vendor makes the vulnerability details public prior to remediation
\({Vfd\cdot\cdot\cdot}\) 2 Variant 5 Vendor does not remediate a reported vulnerability
(multiparty sync) 2 Variant 6 Missing communication between peer vendors impedes coordination
\({\cdot fdP\cdot\cdot}\) 2 Variant 7 Coordinator makes vulnerability details public prior to remediation
\({Vfdp\cdot\cdot}\), \({vfdp\cdot\cdot}\) 2 Variant 8 Finder reports a vulnerability to one vendor that may affect others using the same component
\({\cdot fdP\cdot\cdot}\) 3 Public disclosure of limited vulnerability information prior to remediation
\({v\cdot\cdot P\cdot\cdot}\), \({v\cdot\cdot\cdot X\cdot}\), \({v\cdot\cdot\cdot\cdot A}\) 4 Public disclosure or exploitation of vulnerability prior to vendor awareness
\({vf\cdot PX\cdot}\), \({vf\cdot P\cdot A}\) 4 Variant 1 Finder publishes vulnerability details and vulnerability is exploited
\({v\cdot\cdot p\cdot A}\) 4 Variant 2 Previously undisclosed vulnerability used in attacks

MPCVD Benchmarks

A common problem in MPCVD is that of fairness: coordinators are often motivated to optimize the CVD process to maximize the deployment of fixes to as many end users as possible while minimizing the exposure of users of other affected products to unnecessary risks.

Benchmarking MPCVD

The model presented in this section provides a way for coordinators to assess the effectiveness of their MPCVD cases. In an MPCVD case, each vendor/product pair effectively has its own 6-event history \(h_a\). We can therefore recast MPCVD as a set of histories \(\mathcal{M}\) drawn from the possible histories \(\mathcal{H}\):

\[ \mathcal{M} = \{ h_1,h_2,...,h_m \textrm{ where each } h_a \in H \} \]

Where \(m = |\mathcal{M}| \geq 1\). The edge case when \(|\mathcal{M}| = 1\) is the regular (non-multiparty) case.

We can then set desired criteria for the set \(\mathcal{M}\), as in the benchmarks described in Benchmarking. In the MPCVD case, we propose to generalize the benchmark concept such that the median \(\alpha_d\) should be greater than some benchmark constant \(c_d\):

\[ Median({\alpha_d}) \geq c_d \geq 0 \]

In real-world cases where some outcomes across different vendor/product pairs will necessarily be lower than others, we can also add the criteria that we want the variance of each \(\alpha_d\) to be low. An MPCVD case having high median \(\alpha_d\) with low variance across vendors and products involved will mean that most vendors achieved acceptable outcomes.

To summarize:

  • The median \(\alpha_d\) for all histories \(h \in \mathcal{M}\) should be positive and preferably above some benchmark constant \(c_d\), which may be different for each \(d \in \mathbb{D}\).
\[ Median(\{ \alpha_d(h) : h \in \mathcal{M} \}) \geq c_d > 0 \]
  • The variance of each \(\alpha_d\) for all histories \(h \in \mathcal{M}\) should be low. The constant \(\varepsilon\) is presumed to be small.
\[ \sigma^2(\{ \alpha_d(h) : h \in \mathcal{M} \}) \leq \varepsilon \]

The proposal above treats each vendor's history as independent of the others. The next section describes how Vultron scores a multi-party case, and why it summarizes the vendors differently.

Scoring a Multi-Party Case

ADR-0182 scores a multi-party case one vendor at a time. Each vendor's sequence holds that vendor's own vendor-awareness, fix-ready, and fix-deployed events, and the case's public-awareness, exploit, and attack events. That sequence is scored against the baseline of a one-vendor case, as Reasoning over Possible Histories describes. The vendor's report-handling and embargo desiderata are scored in the same one-vendor model. Each vendor's score has an end part and an embargo part, and each part has an expected, a best, and a worst score. The end part holds the original twelve desiderata and the report-handling desiderata. The embargo part holds the embargo desiderata. The vendor's total adds up the two parts. A best or worst score that adds up more than one walk is a bound, because the walks share the events of the case. No sequence for the whole case is built, because no rule yet says which vendor's events stand for the case.

The vendors' scores are not independent. Public awareness, a public exploit, and attacks happen once for the case, and every vendor's sequence holds each of those events that has happened. So the scores do not form a statistical sample, and a variance or standard deviation across them would describe a sampling distribution that the case does not have.

For each of the expected, best, and worst totals, the case reports the unweighted median across its vendors and the spread. The spread is the highest vendor's score minus the lowest vendor's score. It shows how evenly the case served its vendors: a spread of zero means that every vendor got the same score. The spread measures evenness across vendors, not across users. A vendor with few users counts as much as a vendor with many, and Is the multi-party measure fair? shows how that can mislead.

A two-party case has one vendor, so its summary is that vendor's score, with a spread of zero. A vendor whose state the joint model has no place for has no total. Then the case has no median and no spread, because a median over the other vendors would look like a median over all of them.

A participant with no vendor path, such as a deployer that does not hold the Vendor role, is out of scope. The case-state model has no state for a fix path that the participant does not have, so that participant has no sequence to score. The summary lists every participant that has no sequence, with the reason: no vendor awareness, no case-state events of its own, or events that do not form a valid case history.