Follow on: Revisiting the Definition of Software Quality

Interesting article and discussion on the definition of quality on StickyMinds.com! Article is dated back to 2001, but it is still very much relevent. Robert L. Glass has done a good job in defining what quality is and what it is not. As you can read through the comments, not everyone agree with his definition - as one would expect. Quality is a FAT word and can be interpreted in zillion of ways. It is therefore important that the project team agrees to one definition of quality and stick to it. Consistency is far more important than the definition itself.

ISO definition of Quality: The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs. (ISO 8402: 1986, 3.1)

This definition captures both funcional and non-functional requirements. And BTW, the official name of all "illities" is Systemic Qualities. And there are a lot more Systemic qualities than what Robert has mentioned - for instance - interoperabiliy, availability, scalability, etc.

Another point I disagree with Robert is that "customers/users must participate" in prioritizing and selecting "illities". Some of these systemic qualities are customer facing and other are company facing. For example, it is in companies best interest to make sure there is flexibilty in the code for future expansion and understandability is important to the company for maintenance purpose! Customer doesn't care if your code is moduler and your architecture is flexible. All he cares is the feature set he wants, when he wants it. Customer cares less about the business requirements. But when we talk about quality, all requirements come into picture:

• customer requiremens
  
• business requirements (capture market requirements and corporate requirements)
  
• legal requirements
• government requirements
• social requirements
• testability requirements
• operations requirements
• engineering requirements
  

I don't think we need to be in accordance with the customer on all these requirements!!

Another interesting topic that was raised in the article is whether quality can be quantified, given the definition by Robert Glass. I find it rather amusing because, I think, Quality can be defined and can even by quantified. Of cosurse, not everyone would agree with your definition and your way of quantifying it, but you can definetely do it. And as I said, consistency is far more important than the definition itself.

Read Quality Index (QI): Measure of Risk for more insight into how you can measure software quality

It all boils down to Metrics!!

Setting up a goal is one thing, but how do we know that we have achieved our goal? Software engineering is becoming more of an art than science. Success is a relative term! A project manager with exceptional artistic and articulative skills can sell a project, which is on road to failure, as a successful investment to the executives. In absence of real numbers, the darkness prevails. And under this darkness, all decisions lead to the path of failure.

Snapshot:

"We have a GA date approaching. PPM calls a Projet-Team meeting and takes a vote of confidence, which decides the fate of the software!!"

"A P1 bug is not a show-stopper if it already exists in production. The release will not degrade the production quality!!"

"QA gives a conditional GO with list of risks. By the time decision propogates to executives, the attachment is dropped and the Conditional-Go turns into a Sure-shot-Go!!"

"PM: The problem is not in our piece of the code. The issue is because of the other component that we are dependent on!!"

Sounds Familiar? Interesting, isn't it?

Let's face it, we need sophisticated tools that can generate real time metrics for anyone to make informative decisions. People often mix product quality with process quality. Even though a high quality process generates a high qualiy product (TQM principle), I believe the metrics for the two should be different. For example, higher percentage of test automation improves QA process quality and doesn't directly improve underlying product quality! Note: the automation of processes in the early ALM cycle would have a more direct impact on the product quality.

Here are the list of questions, that metrics should be able to answer:

• Q1. What is the overall Quality Index (QI) of the product. QI for a particular feature or requirements? What is the QI of different components?
  
• Consistency of the processes and measures is the key here.
• It is easy to fabricate a QI model that concentrates on intrinsic product quality!
• Q2. What's our Release Readiness? What's the risk if we release our product today?
  
• Q3. What's the QI trend for differenet releases and different builds?
  
• Trend is more important than actual QI snapshot.
• Errors in the QI (if any) cancel out when you read trends
  
• Q4. What's the Dependency Matrix (DM)? How does other SOA components impact my product? How does my product impact other offerings in the organization?
  
• Current snapshot from QI perspective
  
• Roadmap overlap for future releases. Cross-project Backlog Management.
  
• Q5. What's the realtime Coverage graphs?
• Test Coverage (test validating requirements)
  
• Code Coverage (tests validating code)
• Q6. Testing Strategy Automation.
  
• When files A, B and C change, which features get impacted. What test cases and configurations should a test lead plan for next build?
• Q7. Process Quality. How productive is my team?
• Measures of test automation.
  
• Comparisons with baseline (and manual testing)
  

With above metrics in hand, I can easily make statements like:

• We are ready for the release!! Our product QI crossed 85% in the last build.
• Because of TTM (time to market) pressures, we have decided to release our product with 65% QI. To mitigate the risk, we have also decided to increase our customer support resources.
  
• Our product is not ready for GA because we have a dependency on products B, C and D, and product B has a QI of only 30%. Since B is tightly coupled with our core, we are not in a position to release our product.
• I can effecively utilized my QA resources to concentrate on only the impacted features in a build. We don't have to regress every build every time. We can validate a build with handfull of fixes in less than two hours, and that too with over 95% confidence!!
• We can now sell SLAs and QLAs around certan metrics because we have a consistent (and automated) way of capturing them.
• I can trace a customer escalation all the way back to requirements, because we have end-to-end integration of ALM tools with excellent search facilities.