Assuring Trustworthiness via Structured Assurance Cases
As you can see in figure 12, there is a lot of
white space shown in these two plots of tool
coverage of the test cases for C and Java,
which means that the tools did not find the
things that were in those test cases.
Identifying the right testing capability for the
problem is ideal. The work that the Institute
for Defense Analysis did for the Department
of Defense 31 in their State-Of-the-Art-
Report, looked at testing methods beyond
just tools and the finding was the same, a lot
of white space.
U SING A PPROPRIATE T ESTING AND
A SSESSMENT M ETHODS
Evaluating and assessing software is all-
encompassing.
Reasonable
real-world
solutions require using multiple techniques
that are suited for specific scenarios and
getting wide coverage instead of a one-size-
fits-all model. The diagram shown in Figure
12 shows a large number of test cases of
weaknesses for C and Java 30 , where several
tools were run on the test cases to see which
of the tools could find the weaknesses.
As shown in Figure 13, the appropriate tool
or detection technique is matched with the
artifact so that the weaknesses you care
Figure 12: Coverage of Software Weakness Assessment Tools
30
National Institute of Standards and Technology (NIST), “Software Assurance Reference Dataset (SRD),”
https://samate.nist.gov/SRD/
31
Institute for Defense Analyses, “State-of-the-Art Resources (SOAR) for Software Vulnerability Detection, Test, and
Evaluation,” 2016, https://www.acq.osd.mil/se/docs/P-8005-SOAR-2016.pdf
- 55 -
IIC Journal of Innovation