Cover
Audit Report
Report Number
IT-AR-15-006
Software
Development
Processes
July 13, 2015
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Background
Organizations spend signicant resources developing,
acquiring, and maintaining applications that manage critical
information. To ensure proper governance over software
development, the U.S. Postal Service uses development
processes to ensure proper design, development, and testing of
each new or modied application. The Postal Service has one
of the country’s largest retail networks and has developed over
2,200 software applications to manage its business activities.
The Postal Service uses various processes to ensure each
new or modied application is properly designed, developed,
and tested. To remain competitive, it must use technology that
continues to meet customer needs and achieve business goals.
Currently, there are about 100 applications under development
to optimize the value of the postal infrastructure and leverage
technology to drive business value.
Our objective was to determine whether the Postal Service’s
software development processes are adequate to manage
development risk and reect best practices.
What The OIG Found
The Postal Service does not consistently manage software
development risk or properly develop and maintain
documentation for applications in accordance with current
Postal Service policies. We found that project teams are not
always executing the required phases of the development
process. Also, non-national (eld) applications do not always
adhere to the approved development processes and are not
included in the governance and compliance process.
Further, we found the current governance and compliance
review process does not ensure all software development
complies with Postal Service policies. Finally, management
is not consistently maintaining application status and proper
documentation in the required repositories. We determined
that management did not maintain 1,100 of the 3,451 required
documents for the 71 applications we sampled.
These issues exist because current policy does not clearly
dene roles and responsibilities for documenting system
requirements and testing system functionality. In addition,
software development processes do not address non-national
application development. Finally, management does not conduct
quality reviews or follow-up to ensure all phases of the process
are complete.
Without an adequate software development process, the
Postal Service risks developing applications that do not meet
customer needs or achieve business goals. In addition, there
is a higher risk of cost overruns and project delays, which limit
the organization’s ability to optimize infrastructure and leverage
technology to drive business value. We identied potential
schedule delays and cost overruns of about $4.5 million.
Highlights
The Postal Service does not
consistently manage software
development risk.
Software Development Processes
Report Number IT-AR-15-006
1
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
What The OIG Recommended
We recommended management dene specic roles and
responsibilities for the requirements and testing phases and
ensure that all system requirements are documented and
tested prior to migration to production. We also recommended
management train personnel to test correctly. Finally, we
recommended management revise policies to require quality
reviews, update application status, and upload documentation
at the completion of each development phase. Because of the
2014 cyber intrusion, management disallowed non-national
application development; therefore, we are not recommending
further action on this issue.
Software Development Processes
Report Number IT-AR-15-006
2
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Transmittal Letter
July 13, 2015
MEMORANDUM FOR: JUDITH A. ADAMS
ACTING VICE PRESIDENT,
INFORMATION TECHNOLOGY
MICHAEL J. AMATO
VICE PRESIDENT, ENGINEERING SYSTEMS
E-Signed by Michael Thompson
VERIFY authenticity with eSign Desktop
FROM: Michael L. Thompson
Acting Deputy Assistant Inspector General
for Technology, Investment, and Cost
SUBJECT: Audit Report – Software Development Processes
(Report Number IT-AR-15-006)
This report presents the results of our review of U.S. Postal Service’s Software
Development Processes (Project Number 15TG004IT000).
We appreciate the cooperation and courtesies provided by your staff. If you have any
questions or need additional information, please contact Aron B. Alexander, director,
Information Technology, or me at 703-248-2100.
Attachment
cc: Corporate Audit and Response Management
Software Development Processes
Report Number IT-AR-15-006
3
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Table of Contents
Cover
Highlights ......................................................................................................1
Background ................................................................................................1
What The OIG Found .................................................................................1
What The OIG Recommended ..................................................................2
Transmittal Letter ..........................................................................................3
Findings ........................................................................................................5
Introduction ................................................................................................5
Conclusion .................................................................................................5
Incomplete Requirements and Test Phase ..............................................7
Non-National Applications Do Not Follow Development Policies ............8
Technology Solutions Life Cycle Governance and
Compliance Process ...............................................................................8
Technology Solutions Life Cycle Application Status Not Accurate ..........9
Final Development Documentation Is Not Maintained ............................9
Recommendations......................................................................................10
Management’s Comments .......................................................................10
Evaluation of Management’s Comments .................................................11
Appendices .................................................................................................12
Appendix A: Additional Information ..........................................................13
Background ..........................................................................................13
Objective, Scope, and Methodology ......................................................15
Prior Audit Coverage .............................................................................16
Appendix B: Documentation Review ........................................................17
Appendix C: Management’s Comments .................................................18
Contact Information ....................................................................................20
Software Development Processes
Report Number IT-AR-15-006
4
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Findings
Introduction
This report presents the results of our audit of the U.S. Postal Service’s Software Development Processes
(Project Number 15TG004IT000). Our objective was to determine whether the Postal Service’s software development processes
are adequate to manage development risk and reect best practices. See Appendix A for additional information about this audit.
The software development life cycle is used to develop or modify applications, rst by identifying a need for software and
extending through design and development, testing, acceptance and approval, and maintenance. An individual or group should be
assigned responsibility for each phase so that system design, development, and maintenance progress smoothly and accurately.
For the Postal Service to remain competitive, it is essential that its technology meets customers’ needs and business goals.
The Postal Service implemented two development methodologies to leverage technology and respond to customer needs.
The type of application or equipment being developed determines which methodology is used. All development, acquisition,
and maintenance projects must follow either the Software Development Life Cycle (SDLC)
1
or the Technology Solutions Life
Cycle (TSLC).
2
The SDLC and TSLC phases include activities that must be performed to maintain a secure environment and
comply with Postal Service policies.
The SDLC is based on a development process that requires the sequential design of software. Alpha and Beta tests are the
phases of this process where application functions and features are tested. Alpha testing validates all system requirements
and interface functionality and demonstrates the system is ready for testing in a live environment. Beta testing ensures that
all requirements and interfaces are validated and identies defects.
The TSLC is based on a development process, which allows for rapid development and collaboration between developers and
customers. System Integration Testing (SIT) and Customer Acceptance Testing (CAT) are similar to the SDLC’s Alpha and Beta
testing. SIT testing validates that an application conforms to design specications and requirements and identies and corrects
problems in the application before installation. CAT testing ensures the application satises the documented requirements and
is approved by the business owner.
Following each phase of the life cycle increases the likelihood that a new or modied application meets specic business and user
needs and that systems will have proper controls in place once deployed.
Conclusion
The Postal Service does not consistently manage software development risk or properly develop and maintain documentation for
applications in accordance with current Postal Service policies. For example, project teams are not always executing all required
phases of their software development process. In addition, non-national applications
3
do not always adhere to the approved
development processes and are not included in the governance and compliance process. Further, the current governance
and compliance review process does not ensure that all software development complies with Postal Service policies. Finally,
management is not consistently maintaining application status and nal development documentation in the required repositories.
4
1 The Postal Service Engineering Systems software development methodology for developing and deploying equipment and new systems.
2 The Postal Service Information Technology (IT) software development methodology used to develop and maintain technology solutions such as applications.
3 Local applications developed by eld personnel for their use only.
4 The TSLC Artifacts Library is a document repository that contains nalized project documentation for all applications.
Software Development Processes
Report Number IT-AR-15-006
5
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Software Development Processes
Report Number IT-AR-15-006
6
Technology Solutions Life Cycle (TSLC) Process
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Technology Solutions Life Cycle
TSLC
CLICK ON THE NUMBERS BELOW FOR MORE DETAILS
SHOW ALL
RESET
These issues exist because current policy does not clearly dene roles and responsibilities for documenting system requirements
and testing system functionality. In addition, software development processes do not address non-national application development.
Finally, management does not conduct quality reviews or follow-up to ensure that all phases of the process are complete. Without
an adequate software development process, the Postal Service is at risk of developing applications that do not meet customers’
needs or achieve business goals. Also, because the development process is inadequate, there is a higher risk of cost overruns
and project delays, which limit the organization’s ability to optimize infrastructure and leverage technology to drive business value.
We identied potential schedule delays and cost overruns valued at about $4.5 million.
Incomplete Requirements and Test Phase
Project teams using SDLC and TSLC are not properly developing and obtaining applications requirements and conducting system
and customer tests as required by the software development process.
5
We sampled 71
6
applications and found the following
issues related to the requirements and testing phases:
Requirements Phase
■
We found two instances
7
where requirements were developed during the test phase as opposed to the requirements phase,
and one instance where testing occurred before development was complete.
■
We found two instances where the customer
8
did not prepare the initial application requirements according to TSLC process
9
and, instead, the developer prepared the requirements.
■
We identied 37 instances where no application requirements were uploaded into the TSLC Artifacts Library and 13 instances
where incorrect documents were uploaded.
Testing Phase
■
We identied 136 instances where system and customer testing was not performed.
■
We found 47 instances where the customer did not conduct independent acceptance testing as required and; instead,
testers used the same test scripts to perform both system and user testing.
This occurred because current policies
10
do not clearly dene who is responsible for performing the TSLC requirements, design,
and testing phases. In addition, customers lack the knowledge to perform CAT testing. Finally, test environments were not always
prepared for testing, as required by policy,
11
and test exemptions
12
were not always obtained. As a result, the Postal Service is at
risk of developing applications that do not meet customers’ needs or achieve business goals. In addition, there is a higher risk of
excessive cost overruns and project delays due to increased development and operational costs for reworking issues and xing
system defects. Moreover, the development team could inadvertently introduce errors into the system or code that could expose
condential and sensitive information.
5 TSLC Agile Sprint 0/Requirements Process, Purpose Section, updated March 18, 2014; and Handbook AS-805, Information Security, Section 10-4,
Testing of Hardware and Software, May 2014.
6 Our sample consisted of ve SDLC-developed applications and 66 TSLC-developed applications.
7 An instance refers to a case or occurrence of anything.
8 The organization requesting the new application or modication to existing applications.
9 TSLC Agile Sprint 0/Requirements Process, Purpose Section, updated March 18, 2014.
10 TSLC Policy, Process Description Section, Requirements, Design, and Testing, March 18, 2014.
11 Handbook AS-805, Section 8.3.1, Distributed Postal Computing Environments; and Section 8.3.3, Testing Restrictions, May 2014.
12 Refers to the forms and approval required when testing cannot be performed in either the SIT or CAT environments.
Project teams are not properly
developing and obtaining
applications requirements
and conducting system
and customer tests.
Software Development Processes
Report Number IT-AR-15-006
7
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Non-National Applications Do Not Follow Development Policies
Some eld locations developed applications without following an approved methodology. These applications were connected
to network resources but were not controlled by the Chief Information Ofcer (CIO) organization. This occurred because eld
managers gave untrained personnel access to production and test environments to develop local applications. This type of
shadow development
13
was not visible to IT management due to a decentralized approval process. As a result, these applications
introduced vulnerabilities into the Postal Service network, some of which were exploited during the 2014 cyber intrusion.
On November 9, 2014, management took corrective action by revoking untrained eld personnel’s access to production and test
environments. Management also reviewed about 300 non-national applications and removed all but 18 from production.
14
In
addition, management updated policy to require IT management to approve all eld personnel’s access to production and test
environments. Finally, all non-national application development must follow current Postal Service procedures and IT management
must approve all existing applications before they are placed in production. Therefore, we are not making any further
recommendations at this time.
Technology Solutions Life Cycle Governance and Compliance Process
The TSLC scorecard
15
is designed to monitor governance of and compliance with the software development process. However,
management is not currently using the tool to ensure all software development complies with the Postal Service’s TSLC policies
and compliance standards. Specically, we determined that:
■
While the scorecard is capable of tracking all seven phases
16
of the TSLC process, management only uses it to track the
requirements, SIT, and CAT phases. This occurred because management decided to use the scorecard to monitor key controls
for Sarbanes-Oxley (SOX) applications
17
as opposed to monitoring the entire software development process as originally
designed.
■
The scorecard report lists all applications that do not comply with TSLC policies. However, the IT Compliance Management
Ofce (CMO) does not perform quality reviews or follow-up for non-SOX applications to verify that issues were corrected.
For example, we found 14 instances where Business Relationship Management program managers were notied that
documents were missing from the TSLC Artifacts Library but did not correct the issues. This issue exists because management
has not developed procedures for correcting discrepancies noted on the scorecard report for non-SOX applications.
If the compliance review process is not followed, the IT CMO may not be able to measure compliance with IT policies, identify
issues, and pursue corrective actions. In addition, if modications to applications are not adequately controlled, unauthorized
changes to programs, procedures, or data could compromise the integrity of the applications.
13 Shadow IT development is used to describe systems built and used inside organizations without explicit organizational approval. These systems are deployed
by departments other than IT.
14 Headquarters management reviewed and approved 18 non-national applications to stay on the network because they were needed for business purposes.
15 The TSLC scorecard is the tool used for the procedural review of key compliance areas. The monthly scorecard documents whether appropriate artifacts
have been uploaded for compliance.
16 See Table 1 for all TSLC phases.
17 Financial application supporting business processes and IT applications that have a pervasive impact on the IT control environment.
Management is not using
the TSLC scorecard to ensure
all software development
complies with the
Postal Service’s TSLC policies
and compliance standards.
Software Development Processes
Report Number IT-AR-15-006
8
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Technology Solutions Life Cycle Application Status Not Accurate
TSLC artifacts administrators are not updating the application status in the TSLC Artifacts Library. We found eight instances where
the application status in the TSLC Artifacts Library and Enterprise Information Repository (EIR)
18
did not match. For example, the
Payroll Retirement application status was active in the TSLC Artifacts Library, but did not exist in the EIR. This occurred because
current Postal Service policy
19
does not designate the responsibility for updating the TSLC Artifacts Library when systems are
retired. Application status needs to be accurate and up-to-date to ensure management makes correct decisions about system
needs. As a result of our audit, management took corrective action to update the status for four of the applications
20
we identied
and are currently reviewing and updating the status for the remaining four.
Final Development Documentation Is Not Maintained
Final development documentation for systems and applications is not maintained and stored in the TSLC Artifacts Library.
We reviewed 71 randomly selected non-SOX applications. Our sample consisted of ve SDLC-developed applications
21
and
66 TSLC-developed applications. In addition, we reviewed four judgmentally selected major investment
22
applications that follow
the TSLC process.
We found that all of the required artifacts for the ve applications developed under the SDLC process were present in the library;
however, we identied missing documentation for the 66 TSLC applications and the four major investment applications.
See Appendix B for the results of our review.
In addition to the TSLC process, management selected the BIDS application in our major investment sample to follow the tollgate
process.
23
This process requires the PM to upload required documents into the TSLC Artifacts Library within 10 days after
conducting the stakeholder meetings; however, we found that one of 30 required documents were not uploaded to the TSLC
Artifacts Library within 10 days of the meeting.
These issues occurred because current TSLC policy
24
does not require the PM to upload software development documentation
after each phase of the TSLC process is complete. In addition, nal tollgate documentation was not uploaded due to management
oversight. System documentation is a key element of governance and compliance. Without accurate and completed
documentation, programmers cannot accurately maintain the system and reviewers of the system cannot objectively evaluate
functions and controls.
18 The Postal Service database that maintains information for existing applications.
19 System Retirement Process, Section 7, Certify Retirement, updated EIR, November 3, 2014.
20 Accounts Receivables-Oracle, eClearance Direct Mail To Go Link, and Payroll Retirement.
21 We reviewed SDLC documentation for Advanced Facer Cancellar System (AFCS) Automated-ACP, Automated Parcel Bundle Sorter/Image Controller,
Combined Input Output Subsystem - C, Computer Forwarding System/Mechanized Terminal, and Software Storage and Transfer Processor - AFCS.
22 The Postal Service made major capital investments of $5 million or more for these applications and systems. We reviewed the Business Intelligence Data Store (BIDS),
CustomerFirst! Replacement, Package Remote Encoding System, and Retail Systems Software (RSS). Although it was added as a major investment application,
RSS is the only SOX application listed in our sample.
23 The tollgate process is used when the project involves critical, multi-stakeholder releases across the organization.
24 TSLC Policy, Initiate and Plan Process, March 18, 2014.
Management did not maintain
1,100 of the 3,451 required
documents for the
71 applications we sampled.
Software Development Processes
Report Number IT-AR-15-006
9
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
We recommend the acting vice president, Information Technology, direct the manager, Solutions Development, to:
1. Update Technology Solutions Life Cycle policies and processes to dene which groups are responsible for the requirements
gathering, design, and testing phases.
2. Implement guidance and training for business owners regarding the customer testing process and ensure that testing teams
follow Postal Service Handbook AS-805, Information Security.
We recommend the acting vice president, Information Technology, and the vice president, Engineering System, direct the
managers of Solutions Development and Engineering Software Management, respectively, to:
3. Ensure that all new system requirements and modications are gathered, analyzed, documented, and thoroughly tested prior
to migration to production.
We recommend the acting vice president, Information Technology, direct the manager, Business Relationship Management,
and the manager, Information Technology Quality Assurance, to:
4. Update development policies including the Technology Solutions Life Cycle (TSLC) governance and compliance policy to
include all software development phases in the monthly governance and compliance review process and update the system
retirement process to designate responsibility for updating the application status in the TSLC Artifacts Library.
We recommend the acting vice president, Information Technology, direct the manager, Business Relationship Management, to:
5. Revise policies to require program managers to upload required documentation into the Technology Solutions Life Cycle
(TSLC) Artifacts Library at the completion of each phase of the TSLC process.
Management’s Comments
Management agreed with the ndings and recommendations in the report and, in subsequent communications, also agreed
with the $4.5 million of monetary impact.
Regarding recommendation 1, management stated they will update TSLC policies and procedures to clarifying roles and
responsibilities. Management’s target implementation date is December 31, 2015.
Regarding recommendation 2, management stated they will provide TSLC training to development personnel, program
management personnel, and customer stakeholders. Management’s target implementation date is March 31, 2016.
Regarding recommendations 3, 4, and 5, management stated they will evaluate software development processes in the IT
organization and update the TSLC policies accordingly. In addition, management will implement processes and procedures
that reect current development practices and business needs using a risk-based approach. Management also stated they
will evaluate and update the retirement process accordingly. Management’s target implementation date is September 2016.
See Appendix C for management’s comments, in their entirety.
Recommendations
We recommend management
dene specic roles and
responsibilities for the
requirements and testing phases
and ensure that all system
requirements are documented
and tested prior to migration
to production; train personnel
to test correctly, and
revise policies to require
quality reviews, update
application status, and upload
documentation at the completion
of each development phase.
Software Development Processes
Report Number IT-AR-15-006
10
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Evaluation of Management’s Comments
The U.S. Postal Service Ofce of Inspector General (OIG) considers management’s comments responsive to the
recommendations and corrective actions should resolve the issues identied in the report.
The OIG considers recommendations 2, 4, and 5 signicant, and therefore requires OIG concurrence before closure.
Consequently, the OIG requests written conrmation when corrective actions are completed. These recommendations should not
be closed in the Postal Service’s follow-up tracking system until the OIG provides written conrmation that the recommendations
can be closed.
Software Development Processes
Report Number IT-AR-15-006
11
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Appendices
Click on the appendix title
to the right to navigate to
the section content.
Appendix A: Additional Information ..........................................................13
Background ..........................................................................................13
Objective, Scope, and Methodology ......................................................15
Prior Audit Coverage .............................................................................16
Appendix B: Documentation Review ........................................................17
Appendix C: Management’s Comments .................................................18
Software Development Processes
Report Number IT-AR-15-006
12
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Appendix A:
Additional Information
Background
Software development is the process of converting management or business needs into an application. Organizations use these
applications to drive innovation and competition, thus spending signicant resources developing, acquiring, and maintaining
applications that manage critical information.
When applications are properly designed, system development and documentation controls can prevent or disclose the following
types of errors:
■
Implementation of applications that do not have adequate application controls.
■
Development of applications that either do not meet management objectives or do not operate in accordance with original
specications.
■
Implementation of applications that have not been adequately tested.
■
Implementation of applications that are susceptible to unauthorized modication.
In March 2013, Postal Service IT management selected Agile Scrum as its primary TSLC methodology for all new system
development. Agile Scrum is an iterative and incremental development methodology in which requirements and solutions evolve
through collaboration between developers and business customers. Agile Scrum enables the Postal Service to manage its
operations, optimize the value of the infrastructure, and leverage technology to drive business value. In addition, Agile Scrum:
■
Increases customer satisfaction
■
Increases the number of projects completed on-time and within budget
■
Prioritizes customers’ requirements
■
Improves project communications
■
Improves code quality and decreases defects and re-work
■
Timely identies and escalates issues
Table 1 describes the TSLC phases and the required software development documentation.
Software Development Processes
Report Number IT-AR-15-006
13
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Table 1. TSLC Process
TSLC Phase Description Required Documentation
25
Initiate and Plan
Denes high-level business needs
and high-level project plan.
Business Needs Statement (BNS)
26
Tollgate, Change Control Board (CCB)
27
Requirements
Identies and documents business
requirements to use in developing
a technology solution.
Baseline Tollgate, Requirements with
Approval and Requirements Traceability
Matrix (RTM)
28
Analysis and Design
Develops the technology design
(application, infrastructure, security, etc.)
for developing the technology solution.
Finalize Release Tollgate
Build
Includes development of the technology
components specied in the design
document.
SIT Scripts
29
SIT
Conducted by the IT test team to validate
that the technology solution and its
features conform to the requirements
and design.
SIT Approvals,
30
SIT Results
CAT
Ensure the technology solution satises
the documented requirements and is
approved by the customer.
CAT Results, CAT Scripts, CAT Approval,
31
Implementation Tollgate
Release Management
Ensures that pre-implementation tasks are
dened, change management is followed
correctly, and post-implementation steps
are executed.
Closeout Tollgate
Source: TSLC Artifacts Library.
All functional areas under the CIO are responsible for development, acquisition, integration, deployment, and maintenance
efforts for applications and systems. In addition to this, IT Business Relationship Management is required to store nal baseline
documents in the TSLC Artifact Library; and the IT Strategy and Compliance group is responsible for developing, auditing, and
measuring compliance against TSLC policies, processes, standards, and controls. Finally, Engineering Software Management
(ESM) provides policy and procedures to all engineering design organizations for software products elded by Engineering.
ESM validates requirements and test procedures to ensure the quality of software products.
Table 2 describes the SDLC phases and required software development documentation.
25 Mandatory baseline artifacts that must be uploaded to the TSLC Artifacts Library prior to release into production.
26 Developed by the business owner (customer) to dene the business objectives and value of the application.
27 Identies all stakeholders of the technology solution and change control process for all releases.
28 Documents the origin of a requirement and ensures that all approved business and technical requirements are appropriately tested and implemented.
29 Step-by-step actions the tester takes during testing of the application.
30 SIT approvals are obtained from the group responsible for executing and documenting SIT scripts, and validating test results.
31 CAT scripts are the step-by-step actions taken during customer testing and CAT approvals are obtained from the group responsible for executing and documenting CAT
scripts and validating test results.
Software Development Processes
Report Number IT-AR-15-006
14
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Table 2. SDLC Process
SDLC Phases Description Required Documentation
32
Initiate and Plan
Denes the high-level business needs and
economics of the program
Decision Analysis Report
33
Dene Identies and documents software requirements RTM Checklist
Design Develops software design Software Master Test Plan
34
Develop Includes software preparation for testing
Software Modication Order
35
Project
Development Environment
36
Test
Ensures software testing is performed and
documented
Software Test Report
37
Deploy Initial version of software released to the eld Field Worthiness Evaluation Report
38
Close Out Finalizes the development process Software Integrated Support Specication
39
Source: Engineering Systems Process Assets Library and Engineering SDLC Process Training Manual.
Objective, Scope, and Methodology
Our audit objective was to determine whether the Postal Service’s software development processes are sufcient to manage
development risk and reect best practices. Our audit scope covered applications developed from January 2013 through
December 2014. We conducted work at Postal Service Headquarters, the Information Technology Service Center in Raleigh, NC,
and Engineering Headquarters in Merrield, VA.
To accomplish our objective, we reviewed policies and procedures for software development. We reviewed software development
documentation to determine whether it was current and reected best practices. We interviewed key ofcials to determine the
roles and responsibilities for the software development processes. In addition, we selected random and judgmental samples of
applications to determine if the mandatory documentation was maintained and stored in the artifacts repositories. Finally, we
reviewed documentation to determine whether business needs, requirements, and risks were identied and approved and testing
was conducted and documented.
We conducted this performance audit from October 2014 through July 2015, in accordance with generally accepted government
auditing standards and included such tests of internal controls as we considered necessary under the circumstances. Those
standards require that we plan and perform the audit to obtain sufcient, appropriate evidence to provide a reasonable basis for
our ndings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis
for our ndings and conclusions based on our audit objective. We discussed our observations and conclusions with management
on June 12, 2015 and included their comments where appropriate.
32 There are 39 required documents for the SDLC process. We only listed a few for each phase.
33 Explains the background, purpose, and cost/benet estimates of major operating expense investments.
34 Identies tests to perform, test environments, and test roles and responsibilities.
35 Describes and authorizes the software installation and maintenance release.
36 Denes programming, testing, and target environments. It also contains change, build, testing, and deployment procedures.
37 Provides detailed test results and a log of testing.
38 Used to measure risk of customer dissatisfaction with software release.
39 Identies the contractor’s plans for transitioning support of the software to the Postal Service.
Software Development Processes
Report Number IT-AR-15-006
15
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
We assessed the reliability of software development data by reviewing information stored in the TSLC Artifacts Library, EIR, and
Serena Dimensions Change Management. In addition, we interviewed agency ofcials knowledgeable about the data and process
and tested required documents. We determined that the data were sufciently reliable for the purposes of this report.
Prior Audit Coverage
The U.S. Postal Service Ofce of Inspector General (OIG) issued the Retail Systems Software Application Requirements report
(Report Number IT-MA-15-002, dated March 30, 2015), which found the RSS testing team did not follow the TSLC Agile Scrum
process, or adhere to policy requirements for planning and testing the application. Specically, the team did not develop adequate
test scripts and validate test results to ensure all documented requirements were met. Management agreed with our ndings and
recommendations. We did not claim any monetary impact in this report.
Software Development Processes
Report Number IT-AR-15-006
16
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Table 3. TSLC Development Documentation Review
Required
Documentation
Number of Expected
TSLC Artifacts
Number of
Documents Missing
Number of
Existing Documents
Percentage of
Documents Missing
Program-Level Artifacts
40
CCB 70 62 8 89%
EIR Registration
and Retirement
70 24 46 34%
Host Diagram
41
70 44 26 63%
Run Book
42
70 63 7 90%
Security C
and A Documents
43
70 37 33 53%
System Documentation 70 43 27 61%
Retirement 15 8 7 53%
Project-Level Artifacts
44
CCB 377 99 278 26%
Requirements with
Approval and RTM
377 90 287 24%
SIT Scripts 377 102 275 27%
SIT Approvals 377 104 273 28%
SIT Test Results 377 102 275 27%
CAT Scripts 377 107 269 28%
CAT Approvals 377 107 270 28%
CAT Test Results 377 108 269 29%
Source: TSLC Artifacts Library and OIG.
40 The initial development documents for an application.
41 High-level architectural diagram that documents connectivity, data and business ow, and support functions for all information resources.
42 A written set of procedures for the routine operation of the system.
43 The required certication and accreditation documentation.
44 Changes to an application after initial migration to production.
Appendix B:
Documentation Review
Software Development Processes
Report Number IT-AR-15-006
17
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Appendix C:
Management’s Comments
Software Development Processes
Report Number IT-AR-15-006
18
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Software Development Processes
Report Number IT-AR-15-006
19
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
Contact Information
Software Development Processes
Report Number IT-AR-15-006
20
Contact us via our Hotline and FOIA forms, follow us on social
networks, or call our Hotline at 1-888-877-7644 to report fraud, waste
or abuse. Stay informed.
1735 North Lynn Street
Arlington, VA 22209-2020
(703) 248-2100
Print
Appendices
Recommendations
Findings
Table of Contents
Highlights
