Role of a Quality Architect

I found a really interesting article on Application Quality by Allen Stoker. Make sure you read both part 1 and part 2. Best part is that it discusses the need of a Quality Architect:

"Quality begins in the team - not the application. Proper planning, communication and processes are essential to any successful project. Projects that lack these fundamentals will likely produce problematic applications. I'm a firm believer that large teams with diverse skill sets need a Quality Architect - a highly skilled technical person on your team who has no assignment but to support or ‘enable’ the other team resources. Such a resource can mean the difference between project success and failure."

This is even more interesting to me because I spent some time last year just to understand the role. I would agree 101% with Allen that this role can make a world of difference and can be responsible for a project's success or failure, especially in light of the fact that quality is the measure of success! (assuming quality is part of the defined business goals)

The Role of a Quality Architect:

• Get the business, engineering, and QA teams to agree on common quality goals (i.e. define quality!)
  
• Establish QA infrastructure to boost team's efficiency and effectiveness
  
• Establish processes that complement the tools and provides end-to-end traceability
  
• Review product architectures and provide feedback on systemic qualities before development cycle starts
• Understand the ALM process and idenify risk elements from quality perspective
• Standardize processes and procedures to be able to develop SLAs and QLAs
• Work with cross-functional teams to combine elements of project management and business analytics, especially w.r.t. SOA interdependencies
  
• Translate quality metric data into information! Enable inuitive reporting to drive transparency into product's intrinsic quality.
  
• Participate, Review and Approve testing strategies
  

The role touches almost every aspect of ALM , i.e from requirements to requirements. Horizontally, the Quality Architect is responsible for coordination and collaboration across cross-functional teams (from marketing to design & development to QA to operations & customer care) Vertically, the person is responsible to boost team's productivity and at the same time explain quality metric data to executives in layman terms.

The role requires a fine balance between extraordinary people skills and hands-on technical skills!

Trackback URL: Role of a Quality Architect

Follow on: Development vs. QA - Why disagree?

I like the bold questions raised by jason in his blog "Development vs. QA - Why disagree?". For the last half decade (especially after dot-bust), quality has gained an overwhelming visibility in the software industry and the awareness is growing day-by-day. Numberous studies have proven the exponential relationship between the life of bug and associated cost. Sustaining costs are increasing far beyond original development costs. Therefore, companies are trying to crush new bugs, as soon as they find their way into the code. And hence, the pressure is on developers to test their own code!

I don't see the testing goals between development and QA as conflicting. I see the conflict more because of differences in role, availability of test beds, and more importantly the will! Developers generally don't want to do testing - they always write the perfect code!

The QA is way too on the other side of the wall. 99% of QA teams are involved in black-box testing of features as customer sees them. So, Quality organization is always more close to the end-customer as compared to the development.

To me - both practices are inefficient. We all know that by testing in the end, QA cannot build quality in, it's the development team that needs to write a quality code to start with. I am a firm believer of TQM principles and Deming 14 points. To improve quality, all processes must be standardized, engineering principles must be put in place, and there should be tools that ease the adoption of all these processes. Processes without tools create too much work and chaos!.

The solution is to have a QE team, a team that is more close to development (report to the same director, or even the same manager!) , responsible for all the functional testing. QE team can catch bugs early in development cycle and QA team can focus only on non-functional requirements as part of the system testing.

Developers don't want to be QA!! They restrict themselves to Unit testing and some basic functional unit testing. Another complecated issue is the deployment that is generally not automated. Lack of automated deployment breaks down the Continuous Integration cycle and developers' motivation to automate post deployment test scenarios. Using tools like MockStrutsTestCase and CactusStrutsTestCase, developers have started to look into some level of pre-deployment functional unit testing (including in-container testing) - but again, that's streatching the limits of Unit testing, as jason said.

QA is black box. But there is a limit to what QA can test - with limited resources, especially time. Once feature freeze is done and all the code is checked-in, nobody wants to give QA couple of months just to make sure they can complete the test cycle. The complicated nature of software, with all the reuseable code and interdependencies, fixing one bug late in the game has a huge potential to give birth to two or more bugs (it's like the Samuel monster from Helloboy!)

Watch Watts S. Humphrey's video for more info on why testing in the end is a bad bad thing to do.

Trackback URL: Follow on: Development vs. QA - Why disagree?

Security Testing (Application Layer)

There are different layers at which we can test for security - Physical, Hardware, OS, Network, and Application. In this blog, I am only addressing application layer security testing. Therefore, you'll not find items like testing of firewall policy rules, hardened OS, checking for all open ports on every system in the data center, testing of dialup & VPN access to systems, system interconnection vulnerabilities , or Intrusion Detection System (IDS). This blog is just a starting point and does not gaurantee end-to-end security test plan.

• Authorization: Act of identifying an individual, i.e. it is determining whether they are who they claim to be. This testing includes:
  
• Password based authentication
  
• Checking against Denied Parties Restriction List (DRPL)
• Test for unauthorized countries using Reverse DNS (rDNS)
• Test for Login leakage: Test to make sure that user is not revealed whether the userID was wrong or password was incorrect, in case of authentication failures.
  

• Authentication: Act of determining whether a given user is allowed to access a given resource under given circumstances (Role Based Access Privilege).
  
• Test that only authorized administrators with the appropriate privilege are allowed to access each administrative function.
  
• Spoof testing by logging with one role and trying to access non-privileged administrative function (use URL bookmarking)
• Test by accessing restricted URLs without logging in.

• Password Strength. Test for password length and strength, password history, rollover and expiry. Make sure dictionary words are not allowed.
  

• Passwords in clear text.
  
• Check for hard-coded passwords into the software bits or scripts. Run strings on binary code and look for password tags and strings
• Check for password in log files (at all log levels),
  
• Check for password in client side cookies and hidden form fields.

• Encryption. Tests to make sure that all form submissions use encryption to ensure that information such as passwords do not transit on network in clear text form.
  
• Use snoop to capture network packets and make sure no data is transmitted in clear text
  
• Check for SSL Certificates - HTTPS and TLS (for LDAP)
  

• Session Management. Act of maintaining a transaction or a set of transactions from a given user. This involves maintaining the context(some sort of GUID) of an original authentication so that a user does not have to provide a password for every submission.
  
• Test for automatic password protected locking feature on time out.
• Logout action must terminate the active session
• If multiple servers are used, make sure session transfers are secure and work as designed.
  
• Make sure when a session is destroyed, it is destroyed across all systems.
  
• Test for maximum session limit per user (if there is any limit imposed).

• User Profile and Privacy. Make sure that company's privacy policy is communicated to the end-users. Any forms which collect personal information must include a privacy purpose statement explaining why the information is being collected and how it will be used.
  

• Cookies: Cookies are stored in the browser cache generally to manage session state. These can be permanent or session specific, with the difference that session cookies get destroyed when browser is closed. Since these are plain files, they can be edited by any hacker.
  
• Tests for permanent cookies to make sure no user specific information (ID or username or password!!) is saved.
  

• Auditing and Logging: Act of checking a set of actions to ensure that they comply with a given set of expectations.
  
• Check for information protection regulations, such as Sarbanes Oxley, Graham-Leach-Bliley, Data Protection Act, or HIPAA.
  
• Test to make sure security relevant events are getting logged. Events that are logged must include sufficient information, including: Date/Time; System/Subsystem identifier; User/Process ID (if relevant).
  
• Logging events include:
  
• Number of password guessing attempts,
  
• Attempts to use privileges that have not been authorized,
• Denial Of Service attacks
• Login Logs.Test to make sure information logged includes the user name, date and time of login, and any privilege escalations that are requested and are granted or denied.
• Last Login. Test to make sure that at login time, every user is given information reflecting their last login time and date.

• Web Security Threats
• HTTP Get vs. Post. Make sure portal submit form data using HTTP Post. If HTTP Get is used, add data is visible under URL, irrespective of whether HTTP or HTTPS is used.
• Check for password and other customer sensitive data in hidden form fields.
• Test to make sure that web server is not configured to show directory listing.
  
• XSS security threats. Refer to http://sec.drorshalev.com/dev/xss/xssTricks.htm for more details
• Make sure that hidden form fields don't carry sensitive user information.
  
• URL redirections. Test to make sure all form submissions go through HTTPS

Useful Links:

• Potential threats for authentication based on fingerprint systems
• Java Security System Identity Auditor
• Platform for Privacy Preferences

Trackback URL: Security Testing (Application Layer)