Slide 411/16/2007
©2002- cognence, inc.
• We have problems with poorly written Requirements
• Our requirements are not always testable (I.e., complete,
feasible, consistent, and not subject to interpretation)
• There is too much re-work in our product development
• We deliver the product when it’s completely tested and not
before and we know what completely really means
• We don’t look for and track defects until system test
• We always pass from one life-cycle phase to the next clearly
reviewing and satisfying entry and/or exit criteria
• The product is not used/usable by the customer once delivered
True or False?
Slide 511/16/2007
©2002- cognence, inc.
The ability of a set of inherent characteristics of a
product, product component, or process to fulfill
requirements of customers (CMMI-DEV 1.2 Glossary)
ü Conformance to Requirements
ü Rapid Time to Market
ü Low Cost
ü Lots of Features
ü Easy to Use & Useful
ü Zero Defects
ü High Performance
We define quality as many things, but it is up to each
organization to define what it is and how to achieve it
What Is Quality?
Slide 611/16/2007
©2002- cognence, inc.
• Organization policies, standards, development processes
and procedures are not followed consistently
• Defects are introduced starting with the requirements but
not identified and fixed except when found by the customer
• Quality improvement is primarily composed of “testing
quality into the product ”
• Investment in quality is much lower than the cost of poor
quality
– Little invested to prevent problems - plenty spent to fix
• Products are delivered that do not match the customer’s
requirements or expectations
– Poor satisfaction
– Low confidence
So, What’s Your Quality Focus?
Slide 811/16/2007
©2002- cognence, inc.
Start Thinking about Quality Engineering
• Product Quality Engineering includes all technical and
management functions that determine the quality policy,
objectives, responsibilities, and implementation by means such
as:
– Quality Management - Creating and Implementing Policies,
Processes, Procedures and Standards
– Quality Functions - Quality Control and Quality Assurance
– Quality Planning - Specifying and Measuring Quality
– Quality Requirements - Clear, Accurate and Complete
– Quality Assurance - Process and Product QA
What Do We Do?
Slide 1011/16/2007
©2002- cognence, inc.
Product Quality Engineering
Product Quality Engineering
Quality Control
Quality Assurance
Adapted from
“Software
Quality
Engineering”by
Michael Deutsch
& Ron Willis
Product Quality
Process Quality
Contributes to
Implemented through
Contributes to
What People Produce
•Requirements
•Design
•Code
•Test Plans/Cases
How People Do It
•Standards
•Procedures
•Project Level Processes
Is defined by
Is defined by
Quality Control vs. Quality Assurance
Slide 1111/16/2007
©2002- cognence, inc.
Quality Control evaluates the
products
• Product quality checks:
– Are the requirements
testable?
– Is the product within
tolerance?
– Will the product meet
acceptance criteria?
• Quality Control Techniques:
– Reviews of work product
(peer review)
– Inspections
– Tests (unit, integration, etc.)
Quality Control vs. Quality Assurance - 2
Quality Assurance evaluates
the process
• Process quality checks:
– Is the process being
followed?
– Are there effective Quality
Control activities?
– Is the process working for
the organization?
• Quality Assurance
Techniques:
– Process Reviews
– Work Product Audits
– Assessments
Slide 1311/16/2007
©2002- cognence, inc.
PPQA does:
ü Reinforce correct
process use
ü Objectively
measure process
compliance
ü Support process
implementation &
improvement
ü Operate
collaboratively
ü Provide process
feed-back
PPQA does not:
× Focus on compliance with
the model (CMMI)- that’s an
appraisal!
× Verify the “goodness” of
work products
× Require the same level of
auditing on all projects
× Need to be organizationally
independent
PPQA Objectives
Slide 1411/16/2007
©2002- cognence, inc.
• Understand the organization and project processes as well
as the project’s needs
• Drive creation of the project’s Quality Plan
– To match project size, risk and criticality factors
• Assist in tailoring the Project’s Process so it aligns and
supports the Quality Plan
– Assist in setting up peer reviews for selected work products
• Provide mentoring and other input as to the efficiency of the
software process as used by project members
– Discussions with project members and the project leader
– Analyze the process in the context of which it is being used
• Perform quality audits to ensure quality goals are met
PPQA Responsibilities
Slide 1611/16/2007
©2002- cognence, inc.
...and the #1 challenge = Management Support
• Problems:
– Management sees
no added value
– Expectation to fund
and resource out of
current headcount
– Active subversion
of a quality process
• Solutions:
– Begin a measurements
program early – start with
measuring prevention
cost vs. cost of poor
quality (or keep it simple
and measure cost related
to rework)
– Ensure adequate
communications and
focus
– Provide frequent feedback
to the management team
in a variety of forms
Slide 1811/16/2007
©2002- cognence, inc.
• Problems:
– Staff see no value
in PPQA
– Staff assume
PPQA is
measuring
individual/group
performance
– Staff perceive
PPQA as
“outsiders” or “the
process police”
• Solutions:
– Select appropriate PPQA
staff that are collaborative
in style and respected
(knowledgeable across
the product development
lifecycle)
– Educate staff (PPQA
orientation) on benefits
and “what’s in it for me”
– Again, choose respected
PPQA representatives
from within the
organization
Credibility
Slide 1911/16/2007
©2002- cognence, inc.
Problem:
• Difficult ensure that
PPQA activities are
conducted objectively
and consistently
Solutions:
• Use pre-defined
organizational checklist
and tailor to include the
project’s specific
processes – but be careful
of the “checklist mentality”
(i.e., audit mentality)
• Establish standard
escalation criteria
• Use standard results
reporting mechanisms that
tie results to process and
not teams and/or people
Objectivity
Slide 2011/16/2007
©2002- cognence, inc.
Problems:
• More resources needed
early on during process
adoption
• Resources (PPQA
staffing)
• Need PPQA of PPQA
• PPQA role may change
as organization matures
(e.g., measurements)
Solutions:
1. What we typically see – create a new
functional team to perform PPQA
2. Alternative 1 - Utilize the test team
for independent and objective PPQA
functions
3. Alternative 2 - Establish a flexible
cross-organizational “virtual” PPQA
pool - cycle all technical and
management resources through this
role – cover PPQA of PPQA reviews
Scope & Resource – last but not least!
Slide 2111/16/2007
©2002- cognence, inc.
The PPQA Approach Decision
Things to Think About – no matter which implementation
you choose:
• Know your organization and current culture
– Size of the organization
– Geographical dispersion
– Current knowledge/experience level of the development and test
– Outsourcing initiatives
– Current level and capability to understand/use defect data
• Empower your PPQA Representatives!
– They must have the “last word” on phase-end deliveries
• Have sound escalation procedures
• “Go/No Go” for Exit/Entry Criteria from/to all life-cycle phases based on
process & product defects (especially for the “front-end”)
• Adopt a risk based approach to scope extent of PPQA work
Slide 2211/16/2007
©2002- cognence, inc.
• They understand the what and why of quality functions
– They see the value added in quality-related activities
and do not perceive themselves as costly overhead
– Understanding that a defect, is a defect, is a defect
• They can help to prevent defects from being introduced
into the system starting with the requirements (if
included/invited as key reviewers and approvers)
• Can be easily trained or hired with Quality Engineering
training and/or background
PPQA à PPQA Function: Possible Benefits
Slide 2311/16/2007
©2002- cognence, inc.
• They are not part of the development team
and probably will be viewed as “process
police” having difficulty in getting cooperation
from the project team
• They are very often not invited or included in
planning and key review activities
• May not know the product or stay technically
involved/knowledgeable, therefore creating an
“ivory tower” mentality or perspective
• Resources chosen for this function are
sometimes not trained or qualified - “warm
bodies”
PPQA à PPQA Function: Drawbacks
Slide 2411/16/2007
©2002- cognence, inc.
• They are part of the development team and are not
typically viewed as “process police”
• They understand the what and why of quality functions
– They see the value added in quality-related activities
and do not perceive themselves as costly overhead
– Understand that a defect, is a defect, is a defect
• They can help to prevent defects from being introduced
into the system starting with the requirements (as key
reviewers and approvers)
• They are the last line of defense, as such, keenly
interested that products are delivered to match the
customer’s requirements or expectations
PPQA à Test: Possible Benefits - 1
Slide 2511/16/2007
©2002- cognence, inc.
• By participating in up-front planning activities, they
can ensure quality plans are in place
• They can effectively oversee that organization
policies, standards, processes and procedures are
followed
• They understand the product, participate in work
product reviews, and can directly contribute to the
improvement of work products
• They can better use process and product defect data
to enlist management’s support of quality engineering
concepts (now, across the lifecycle, not just test)
PPQA à Test: Possible Benefits - 2
Slide 2711/16/2007
©2002- cognence, inc.
• They are part of “a” development team (i.e., “one of
us”) and are not typically viewed as “process police”,
even if they are from a different product/application
group
• They can help to prevent defects from being
introduced into the system starting with the
requirements (as key reviewers)
• They understand the product, participate in work
product reviews, and can directly contribute to the
improvement of work products
• As team members/managers are cycled through this
role, they begin to understand the negative or positive
effects the process may have
PPQA à Virtual: Possible Benefits
Slide 2911/16/2007
©2002- cognence, inc.
Planning for quality
Designing in quality factors (e.g.,maintainability, reliability)
Establishing the use of standards and procedures
Reviews
Testing (Dev only unit test)
Audits
Setting quality goals
Providing visibility into the process and product quality for
management (Reporting)
Ensuring non-compliance issues are resolved before the
product is delivered to the customer
PPQA Test Virtual
Responsibility Overlap – “…..with one stone”
Leveraging existing skills and activities
• When planning the project, use a pre-determined risk
based PPQA Planning Checklist to decide level of detail
and frequency of PPQA oversight and involvement for
each project. Possible considerations:
√ Project size
√ Complexity
√ Project Duration
√ Business risk
√ Technology risk
√ Quality Objectives/Requirements
• This can appropriate PPQA resources where most needed
Risk-Based PPQA Planning - 1
• Customize PPQA review and audit approach to fit different
project needs:
– Artifact Audit
+ Ease of scheduling, minimizes overhead & subjectivity
― Reduced opportunity for discussion, questions/answers
– Questionnaires
+ Minimizes overhead
― Low response rate, permits interpretation issues, reliability
is questionable
– Interviews
+ Enables discussion, questions/answers and clarifications
― Difficult to schedule meetings, can be subjective
Risk-Based PPQA Planning - 2
Slide 3211/16/2007
©2002- cognence, inc.
Toll Gate Checkpoints for Approval
1. Approval/Funding for Analysis and Design
effort
2. Detailed requirements approval (RR)
Architecture Review approval
3. Approval/Funding for Construction to begin
(CDR)
4. Change request approval as project phases
progress
5. System Testing sign offs (TRR)
6. Client approval for implementation, Risk and
Change Management approval
Key Project Management Product Deliverables
1. High-level project schedule & budget estimate (planning)
2. Detailed requirements, Architecture Review
3. Detailed design, project schedule, budget (commitment)
4. Product build, test, training and contingency plans, project
control documents.
5. Post-implementation review reports
OPERATE
BUILD
PLAN
1
4
Initiation Analysis Design Construction
System
/QA
Testing
Post
Implementation
ImplementationUAT
Project Management Processes
3
1
6
4
3
Project Management Processes
Estimates
1. Planning estimate - high-level
2. Commitment estimate - detailed
3. Revised cost estimates -
throughout project life cycle
Re-PLAN
5
2
2
5
1
2
3
Example Project Life Cycle with PPQA Checkpoints
PPQA Reviews
Slide 3411/16/2007
©2002- cognence, inc.
• Quality Control (QA testers) evaluates the product
• Quality Assurance (PPQA) evaluates the process
• Marriage of these two provides full cover of product
and process creating one robust function – “The
whole is greater than the sum of the parts”
• Leverage resources and skill set of the team to review
process and project artifacts
• Facilitate better quality planning activities
• Use risk-based approach for PPQA planning activities
and resources on projects
• Ensure product quality at each development phase
and before the customer sees it
Summary
