FW: [scrumdevelopment] Re: [APM] Traceability
Let’s try another path to the same goal.
You’ve built a system for me (this “could happen). It has test coverage as specified. It’s in production doing something like decision support where an answer comes out from the data put in. but there is not a 1:1 connection between input and output. Bayesian decision support is common around here. All the requirements for the input data are there. They’re defined by a federal agency. “You must measure in this way, store in that way, assess in this way, do spectrographic analysis in that way…” With the data coming in and the modeling working, an assessment of the risk of doing some remediation can be asked of the system. “Should we haul the millions of cubic yards dirt somewhere or can I leave in lie for the next 20,000 years?”
Now the omnipotent federal agency comes along with new requirements. They’d like assure that all the old assessment and decisions will come out of the system as before.
(1) What modules will be impacted by these new requirements? Here you can’t change the code without prior permission. Or at least you can’t check it in.
(2) How much will it cost in time and money to get compliant with the new regs? Everything is budget on fixed price fixed delivery. It’s not the way to do it, but them’s the rules.
(3) What test do I need to rerun for SQA and IV&V – can’t run them all it takes months and don’t have the staff or machines
(4) What test data is impacted for calibrating the system? For decision support system “calibrating” them is critical. Do they tell you the same decision over a long period of time for the same input?
Here – and you may do it different – we look at the traceability matrix to see where each requirement is implemented in the code. This code is not object based (useless you consider VB and PL/SQL object based), so we don’t go the method level. But the SV&V folks sue are interested in the details of how the code implements the requirements.
Tedious? Yes. Valuable? Depends if you want to get your closure permit or not. Or if can defend the site closure decision in front of Congress. With questions like:
(1) Can you tell me all of the changes made to the system over the past 10 years and which ones impacted the recommended risk assessment to the public?
(2) Can you assure me that all the calculation code in fact produces the same answers it would have before the changes you made as a result of our new regulation?
With questions like that the managers of the system (scientist that code) look to SQA/V&V and ask “well what could have been changed without us knowing the consequences of the change?”
Configuration Management / Independent verification and Validation / Safety and Safeguards are all processes we (and aerospace, weapons, medical, pharma, and flight controls) live by. For us traceability is a valuable tool and process.
Glen B. Alleman
From: Steven Gordon [mailto:sagordon@...]
Sent: Thursday, March 11, 2004 11:58 AM
Subject: RE: [scrumdevelopment] Re: [APM] Traceability
Let me take you through my thinking.
Assume traceability does not help you decide what modules would be affected by implementing the new requirement. Then, it only helps after you have determined which modules would be affected. Then, you can ascertain what pre-existing requirements might be broken by touching these modules, so that you know where to concentrate your testing where the problems are most likely to occur instead of testing the entire system?
I do not see how the expense and drudgery of maintaining traceability is worth the savings involved with testing less. If the system is safety-critical, I would want to thoroughly test everything anyway.
There must be more to it. Perhaps, we can use traceability to help us decide what modules would be affected by implementing the new requirement. We start from the central module we know we have to change to implement the new requirement, and use the trace to determine all the other modules we might have to make changes to. But, surely, if we change a module, we have to consider the impact on every module that uses this one, and then every module that uses that one - the requirements involved are totally irrelevant to the possible impact. There are tools that can do this kind of impact analysis within the code base quite well.
So, I still do not see the point, even for non-agile, safety-critical systems.