The Quality Imperative:
SAS’ Commitment to Quality
A corporate statement of SAS’ commitment to product quality,
service quality and customer satisfaction
ii
^
CONTENTS
QUALITY IMPERATIVE
Contents
Introduction ......................................................................................................... 1
Our Employees ...................................................................................................3
Work Environment.............................................................................................7
Protecting Privacy ...........................................................................................10
Cloud and Information Services ...................................................................11
The Foundation of Quality Development .................................................13
SAS
®
Architecture: Quality by Design ....................................................... 22
Research and Development at SAS .......................................................... 26
Planning and Requirements ......................................................................... 28
Code and Build .................................................................................................29
Testing ................................................................................................................31
Release .............................................................................................................. 37
Deployment ......................................................................................................39
Maintenance and Support ............................................................................40
Quality in Customer Service ........................................................................45
Appendix 1: Regulated Industry Issues ..................................................... 47
Appendix 2: Validating an Analytical Component ................................ 52
Appendix 3: Installation and Operational Qualication
for SAS
®
9.4 and SAS Viya ............................................................................ 62
Appendix 4: Employee Training ..................................................................66
Appendix 5: Quality in SAS Education ......................................................69
Appendix 6: Quality in SAS
®
Documentation .......................................... 72
Appendix 7: Quality in Consulting ............................................................. 75
Appendix 8: SAS
®
Offerings and Products ............................................... 85
Glossary .............................................................................................................89
What’s New .......................................................................................................91
^
CONTENTS
QUALITY IMPERATIVE
1
Introduction
Introducing SAS
For SAS, the pursuit of excellence lies at the heart of the products that we deliver. SAS
CEO Jim Goodnight says that “SAS’ commitment to quality permeates every division
and employee throughout our global company. From the software we produce to the
customers we serve, and through our collaboration with each other, quality is at our
core. It’s who we are and is the foundation on which our company was built.”
SAS provides an integrated suite of articial intelligence, analytics, business intelligence,
customer intelligence, data management, fraud and risk solutions. Our products
transform data into the information that organizations use to make good decisions. SAS
enables customers to apply analytics wherever they demand – whether in the cloud,
on-site, at the device, or with machines as full partners in human decision making. “We
aim to help every customer turn analytic insights into value,” Goodnight says. “We do so
by adapting to changing markets, working with disruptive technologies and remaining
relentlessly committed to innovation. This has allowed us to remain a leader across core
markets while providing innovative solutions to our customers’ most challenging
business problems.”
As a company, SAS runs our own operational systems on SAS technology. Since SAS
software is licensed, we understand that customers have a regular opportunity to
evaluate their SAS investment. As one of the largest customers of SAS software,
we live the importance of high-quality, reliable software.
To learn more about our company, customers and award-winning culture where quality
is integrated into all that we do, visit our website:
• SAS company information page (sas.com/en_us/company-information/prole.html)
• Customer success page (sas.com/customers)
• Annual Report (sas.com/annual-report)
• Diversity and inclusion (sas.com/diversity)
• SAS Corporate Social Responsibility (sas.com/csr)
• Security Assurance from SAS (sas.com/en_us/trust-center.html)
• Recognition from independent industry experts (sas.com/awards)
2
^
CONTENTS
QUALITY IMPERATIVE
The Quality Imperative
The SAS Quality Imperative (sas.com/qualitypaper) describes the essential role of
quality in the creation and delivery of SAS offerings, products and services. The paper
is organized so that topics ow from broad, corporatewide processes that provide the
foundation of quality at SAS to the detailed and technical processes that are used in
software development. Overall, the paper is intended to provide the reader with a
comprehensive picture of SAS’ quality culture and processes that are used to develop
the products and offerings listed in Appendix 8: SAS Offerings and Products.
The following are papers and sites related to the Quality Imperative that provide in-depth
content on targeted areas:
• JMP: A Commitment to Quality (http://www.jmp.com/qualitystatement)
• SAS Trust Center: Trust in Compliance
(https://www.sas.com/en_us/trust-center/sas-trust-compliance.html)
• SAS Managed Application Services (https://www.sas.com/en_us/solutions/cloud/
sas-cloud/managed-application-services.html)
JMP Statistical Discovery LLC (“JMP”) is a wholly owned subsidiary of SAS Institute Inc.
(“SAS”) and adheres to the commitments outlined in this document, to the extent
applicable.
3
^
CONTENTS
QUALITY IMPERATIVE
Our Employees
Quality Culture
Our culture is based on three simple things: trust, exibility and values.
SAS is a company built on relationships, and our relationships with
customers are only going to be as good as our employees’ experience
at SAS. That experience is based on meaningful work, empowering
leadership, and a world-class work environment.
– Jim Goodnight, CEO of SAS
Focusing on people and relationships – making employees a top priority – leads to more
productive, satised and dedicated employees. To achieve that ideal, employees must
be trusted and valued, and they must believe that they can make a difference. To support
the creative process and balance work and family, SAS provides a exible work
environment that enables them to be the most productive.
SAS’ strength comes from its culture, which is rich in diverse people,
talent and ideas. Our collective strength and passion for what
we do drive innovative solutions that solve the most complex
customer problems.
– Jennifer Mann, Vice President, SAS Human Resources
The company's work-life programs and unique corporate culture continue to receive
accolades – at global, regional and local levels – for being a great workplace. Many of
our country ofces have also been recognized for their workplace culture by various
prestigious organizations. A full list of corporate awards can be found at sas.com/
awards.
The recognition speaks to the employee-focused philosophy behind SAS' corporate
culture since the company’s founding in 1976: if you treat employees as if they make
a difference to the company, then they will make a difference to the company. At the
heart of this unique business model is a simple idea: Satised employees lead to
satised customers.
SAS’ work teams thrive on a diverse interplay of experience, backgrounds and
perspectives. Employees’ collective strength and passion for what they do ignites big
ideas and powerful bonds. SAS continues to provide equal employment opportunities
for all employees regardless of age, race, color, gender, gender identity, religion, creed,
ancestry, national origin, citizenship, marital status, sex, sexual orientation, disability,
medical condition, veteran status, pregnancy or any other protected class as dened
by federal, state, or local law.
4
^
CONTENTS
QUALITY IMPERATIVE
Quality Workforce
Designing and implementing SAS software requires a development staff with highly
developed programming skills and signicant subject-matter expertise. As part of SAS’
effort to attract and retain the best available employees, the Human Resources Division
has implemented several recruitment guidelines, as shown in the following examples:
• To be considered for an open position, all applicants must meet the specic
education, training and experience qualications for the open position.
• For each position at the company, there is a written job description that species
the necessary education or experience and job functions.
• Competency-based interviewing is used to identify the ability, experience and
knowledge that are required for a particular position. In addition, numerous interviews
are conducted with an applicant. This practice enables SAS to be as inclusive as
possible, while also enabling the interviewee to experience the culture and the people
who create the culture.
• SAS’ approach to performance management is aligned to business needs, employee
skills and career development. We promote ongoing conversations between managers
and employees around expectations, skills and development. Performance
management occurs continually.
• SAS performs post-offer and pre-employment drug, alcohol and criminal background
screening on every nal applicant for employment in accordance with SAS’ Human
Resources policy. In addition, we conduct background checks on contractors and
other third parties per the SAS Human Resources policy for contingent workers.
• All SAS staff nominated and assigned to any agreed project role must have the skills,
experience and knowledge required to meet assigned duties or deliver expected work
products. Recognition as a best place to work enables SAS to hire and retain the best
employees in the industry.
At SAS, analytical and statistical software is designed, written and tested by highly
specialized and educated engineers. For example, more than two-thirds of the engineers
working on analytics products have advanced degrees, and more than two-thirds of
those have PhDs in elds such as data science, statistics, mathematics, computer
science, econometrics and operations research. Product developers, including product
management and engineers, also have domain expertise in operations research, time
series analysis, nance, pharmaceuticals and other elds through previous work
experience and education.
Our organization is built on the high quality of our employees and the executives who
lead them. See the executive biographical information (https://www.sas.com/en_us/
company-information/leadership.html) to learn more about SAS executives. We
consider our organizational chart to be condential and do not disclose it.
5
^
CONTENTS
QUALITY IMPERATIVE
Diversity in the Workplace
We believe a diverse workforce
brings unique talents and
inspires teams to create software
that can change the world. Great
minds don’t always think alike,
so we make it a priority to
promote
an environment where varied
perspectives are encouraged.
Big ideas ignite when everyone is
treated with fairness and
respect.
— Jim Goodnight, CEO of SAS
Quality benets from all forms of diversity. SAS has a multidimensional culture that
blends different backgrounds, experiences and perspectives from employees around the
world. SAS’ culture encourages everyone to feel condent in expressing their ideas, and
to know that they will be respected for their unique contributions and abilities.
Read more about SAS’ commitment to diversity and inclusion by visiting sas.com/
diversity.
Employee Training
Employee technical training and professional development is critical to the SAS quality
process. It is an ongoing endeavor that begins during onboarding and continues
throughout employment. Employees continuously set and rene development goals
with managers that align with the skills needed to excel at their current or future roles.
SAS provides a variety of resources to help employees succeed with their career training
and development. Many training courses are accessible through the SAS Learning
Management System and selected external online learning platforms. Training content
can be delivered through a variety of channels that provide staff with options most
appropriate for their learning style, including self-paced learning and live instructor-led
courses. Furthermore, an extensive corporate library contains bound volumes, periodical
subscriptions, complete SAS documentation and audio and video training materials.
SAS offers several formal and informal mentoring programs to grow and nurture internal
talent. Additionally, we provide tuition reimbursement for degree-seeking employees to
support their ongoing higher education development.
For more information, see Appendix 4: Employee Training.
Employee Certications
SAS employees take great pride in achieving professional certications that enhance their
ability to deliver a quality product to customers. These certications include the following:
• Security certications, such as Certied Information Systems Security Professional
(CISSP
®
), GSEC – GIAC Security Essentials, CompTIA Security+ and Certied Secure
Software Lifecycle Professional (CSSLP).
• Project management certications, such as Project Management Professional (PMP).
• Quality and compliance certications, such as Certied Software Quality Engineer
(CSQE), Certied Quality Improvement Associate (CQIA), Certied Information
Systems Auditor (CISA) and Certied Quality Auditor (CQA).
• Process improvement certications, such as Six Sigma.
• Information technology certications, such as Information Technology Infrastructure
Library (ITIL), Microsoft Certied Professional (MCP) and various Microsoft Azure
certications.
• SAS software certications and see the value here: SAS software certications.
6
^
CONTENTS
QUALITY IMPERATIVE
Quality Starts With Communication
Employees are inuential at SAS because the company’s leadership understands that
employees contribute valuable feedback and serve as the main drivers of change,
momentum and innovation.
As a leading data analytics company with a global presence, connecting with employees
is essential. The SAS Internal Communications team embraces the challenge by using a
broad range of communication tools.
In addition to sharing information through the SAS intranet – the primary internal
communication vehicle – SAS also hosts webcasts, podcasts and town hall events.
Digital signage with global and local information is now available in most SAS ofces
as well. Lunch-and-learn sessions provide opportunities for peer-to-peer learning and
networking. Other internal forums enable subject-matter experts to instruct, share
knowledge and spark creativity. Instant messaging and real-time le sharing tools
enable SAS employees to communicate, collaborate and coordinate with each other
across the globe.
Social media use within SAS continues to grow — from blogs to the internal networking
platforms — to create a tight-knit virtual community. In addition, most major divisions
sponsor regular internal webcasts that enable employees to obtain updates on divisional
priorities and to ask questions of upper-level management. Several divisions also deliver
periodic newsletters, support dedicated divisional websites, or host collaborative
forums. These media are updated regularly with information about divisional priorities,
goals, news and changes. A weekly internal news recap email is available, all employees
receive a monthly newsletter from the executive leadership team highlighting the most
important updates over that period.
Whether executives are hosting a webcast for a global audience, holding a town hall
event at a regional ofce, or meeting informally over coffee with a handful of employees,
SAS nurtures an atmosphere of sharing and openness. Employees in the audience or
those watching online are encouraged to ask questions and share their feedback and
ideas. Besides seeking employee comments in face-to-face and virtual meetings, most
major events are followed by a survey, which enables employees to offer feedback
anonymously.
SAS also receives feedback from employees via the annual employee experience survey.
Working task forces address the biggest issues identied through this survey, making
recommendations to enhance existing efforts or create new ones. Many divisions
conduct their own feedback surveys to follow up on areas of specic concern to them.
7
^
CONTENTS
QUALITY IMPERATIVE
Work Environment
Corporate Services
The groups in Corporate Services are responsible for the safe and secure work
environment of all employees on the SAS Cary campus and at regional ofces
throughout the world. A Corporate Services Global Management team aligns
operational processes around best practices and related business strategies
globally. The Corporate Services Division works with ofces globally to adhere
to SAS’ statement on sustainability. For more information, see sas.com/csr.
Corporate Real Estate
The Corporate Real Estate Department is responsible for global workplace solutions.
This includes real estate strategy, space management, lease transactions, design and
construction, ofce branding, art, furniture and ergonomic support.
Facilities
The 300 developed acres that comprise the SAS world headquarters campus are
maintained by SAS’ Facilities Department. This group is comprised of employees who
work in Facilities Management, Facilities Services, Housekeeping and Interior and
Exterior Landscaping.
Security and Safety
SAS’ Security and Safety Department provides a safe and secure work environment
at SAS’ world headquarters and supports worldwide operations. The exact combination
of safety and security measures is based on the needs of the location. One example of
different needs is based on whether SAS leases a space within a secured building or owns
a building. SAS deploys physical, personnel, electronic and procedural measures, such as
the following:
• Security and safety awareness and education activities.
• Access control through staffed gates, card access readers and security
reception desks.
• Fire detection, alarm and suppression (protection) systems.
• Closed circuit television (CCTV) system.
• Uniformed security personnel for both proactive deterrent patrols and various
emergency and non-emergency (customer service) responses.
• Every leased US regional ofce has CCTVs and card access readers.
• Most global spaces have property management companies that we rely upon
who know the local constraints on securing buildings. For example, SAS ensures
that access controls are in place in all global locations, but some countries will not
allow CCTV.
8
^
CONTENTS
QUALITY IMPERATIVE
SAS strives to continually provide all employees with the safety and health knowledge,
tools and environment needed to have a safe, healthy and productive work life,
minimizing the risk of accidents, injury and exposure to health hazards. The Security
and Safety Department fullls these responsibilities by primarily focusing on these
organizational goals:
• Prediction and assessment by evaluating the probability, criticality and business
impact of potential security and safety risks.
• Prevention and protection by implementing the necessary controls to minimize the
negative business impact of identied security, safety and re risks while providing
a safe and secure environment.
• Detection and investigation by identifying security-related exposures to emergencies
and critical incidents to limit injury to people, as well as damage to property and
the environment.
• Compliance with all local, state and federal environmental regulations.
• In the US, the Safety Department is dedicated to ensuring compliance in accordance
with the Occupational Safety and Health Administration (OSHA).
Security and Safety is also the point of contact for the following:
• Coordination of many safety programs, such as CPR, rst aid, defensive driving, child
safety, life safety, hearing conservation, respiratory protection, bloodborne pathogens
and similar initiatives.
• Administering a comprehensive program to prevent, identify and correct indoor air
quality concerns and to reduce our impact on the environment.
• Loss control services.
Access to SAS Data Centers is restricted to authorized employees and contractors who
are tasked with maintaining the hardware or software in those environments and business
partners who support specic business operations.
• SAS Data Center management is responsible for authorizing and reviewing physical
access monthly.
• Entry controls including loading areas.
• Badge readers are located at each entry point to hosting rooms and badges must
always be worn and visible within the SAS Data Center.
• The SAS Global Hosting and US Professional Services environment for hosted
customers requires additional badge readers and PIN codes.
• Discreet signage compliant with security requirements.
Business Continuity Management
SAS’ Business Continuity Management (BCM) program refers to the company’s plans and
procedures aimed at protecting key assets and continuing critical business functions in
the event of anticipated and unanticipated threats. SAS’ global BCM Program, initiated
in 2004, helps inform organizational resilience and focuses on business resumption and
crisis management planning. Associated BCM activities include annual plan maintenance
and testing, staff training and management program review. For additional information
regarding SAS’ BCM Program, please refer to https://www.sas.com/content/dam/SAS/
en_us/doc/other1/csr-continuity-of-business-107776.pdf.
9
^
CONTENTS
QUALITY IMPERATIVE
Supplier Diversity
SAS is committed to diversity among its suppliers. SAS customers represent a wide
range of industries, people and locations – and SAS wants this same level of diversity
reected in the supplier community.
SAS develops and maintains collaborative relationships with suppliers that meet SAS’
global business needs. Any supplier that provides the goods and services relevant to
SAS must demonstrate a “value-added” benet. These benets can include cost savings,
competitive pricing, customer focusing, innovative business solutions and a commitment
to SAS’ values.
Suppliers interested in doing business through our SAS Supplier Diversity Program must
be a socio-economically disadvantaged business, dened as 51% owned and operated by
women, minorities, veterans, LGBTQ+ or persons with disabilities. Businesses must be
certied as “diverse” by a third-party agency and located within the United States.
As part of the Corporate Services Division, the supplier diversity team advances the
goals and objectives of the company’s supplier diversity initiatives. SAS is a member
of the National Minority Supplier Development Council, Disability:IN and the North
Carolina LGBT+ Allied Chamber of Commerce.
SAS works directly with organizations that actively support the education and
advancement of women, minority, veteran, LGBTQ+ and disabled-owned businesses.
Outreach to these organizations includes sponsorships, volunteering and serving on
committees and boards. The supplier diversity team and sourcing team collaborate
closely with these organizations to create training and opportunities for the diverse
businesses they serve. Economic opportunity and sourcing inclusion for diverse
businesses is important to our mission and vision.
10
^
CONTENTS
QUALITY IMPERATIVE
Protecting Privacy
SAS cares about individual privacy rights and is committed to complying with applicable
international privacy and data protection laws regarding the collection, use, maintenance,
sharing and disposal of personally identiable information (PII), including “personal data”
as dened by the General Data Protection Regulation (GDPR) and other regulated data.
These privacy laws range from sector-specic regulations in the United States, such as
the Health Insurance Portability and Accountability Act (HIPAA), to comprehensive data
protection regulations such as GDPR in the European Union and other similar laws
throughout the world.
The SAS Privacy Ofce, led by the SAS Chief Privacy Ofcer, utilizes a risk-based
approach to drive compliance along with the needs of the company through regular
assessments against privacy frameworks and dened metrics. Common privacy
frameworks are utilized to guide privacy management and prompt privacy-relevant
decisions for the company. These privacy frameworks include principles and standards
such as the International Organization for Standardization (ISO); laws, regulations and
programs such as Personal Information Protection and Electronic Documents Act, the
GDPR and HIPAA; and privacy program management solutions such as Privacy by Design
and National Institute of Standards and Technology (NIST).
The design of the program is a hybrid model, in which the privacy function is partly
centralized (in SAS’ Privacy Ofce) and partly decentralized with distributed
responsibility within each business unit via SAS Privacy Champions Network. As such,
SAS’ Privacy Ofce works with stakeholders across the company to develop, implement
and maintain an organizationwide governance and privacy program intended to ensure
compliance with applicable privacy laws and regulations.
To prevent unauthorized access or disclosure, to maintain data accuracy and to ensure
appropriate and lawful use, SAS has put in place reasonable physical, electronic and
managerial procedures to safeguard and secure such information. We respect individual
privacy rights and only use data in compliance with our Privacy Statement. For more
information, use the following resources:
• SAS Code of Ethics
• SAS Trust in Privacy site
• SAS Privacy Statement
◦ Persons in the European Union should access the EEA version of this Privacy
Statement applicable to processing of personal information subject to the
General Data Protection Regulation.
◦ For enterprise hosting managed by SAS, see the SAS Hosted Managed Services
Privacy Policy.
11
^
CONTENTS
QUALITY IMPERATIVE
Cloud and Information Services
SAS’ IT organization, Cloud and Information Service (CIS) partners with SAS business
units and customers to deliver global technologies and services that increase business
value through trust and partnership. CIS is committed to service excellence and has
established practices, including continuous service improvement, that provide a
framework for measuring and improving performance. The following quality processes
ensure adherence to the quality standards established by SAS, CIS and as required by
security and compliance standards.
Software Engineering
• Enterprise architecture to drive an integrated environment that is responsive to
change and supportive of the delivery of the business strategy.
• Adherence to software engineering guidelines and industry best practices to ensure
that quality is built into the design and implementation. This includes a strong focus
on the total experience.
• Comprehensive enterprise solution testing (unit, functional, usability, load and
performance, exploratory, accessibility and others) to ensure that solutions meet
functional and nonfunctional requirements.
• End user, customer-zero testing and deployment of SAS offerings to provide feedback
to SAS product development.
Managed Application Services (MAS)
• Technical reviews that evaluate customer requirements to ensure that hardware
sizing, costing and resourcing align with customer project requirements.
• Dedicated and trained build and operations resources that follow quality best
practices when deploying and maintaining customer environments.
• Deployment testing to validate that a newly deployed or updated environment
is properly installed and congured according to customer requirements.
• System monitoring to track server health and capture metrics of server and
solution availability.
• Dedicated project owners to ensure optimal cloud delivery and overall customer
experience throughout the life of the project.
• Required change management training and established change management
processes to identify, measure and control changes to MAS customer environments.
• Required security awareness training for all MAS employees and partners to ensure
that customer, legal and regulatory requirements are met.
For more information about the quality processes and controls in the SAS Cloud, see
https://www.sas.com/en_us/solutions/cloud/sas-cloud/managed-application-
services.html.
12
^
CONTENTS
QUALITY IMPERATIVE
Security and Compliance
• Layered industry standard security controls and defenses to protect the business.
• Penetration testing to identify and resolve systemic weaknesses within the overall
information security program.
• Security audit and compliance to ensure adherence to security controls and defenses.
Service Excellence
• Rich automation with a focus on self-service to ensure repeatable processes and to
drive efciencies.
• DevOps to ensure collaboration and communication of both software developers
and operations professionals while automating the process of software delivery and
infrastructure changes.
• Operational processes based on the Information Technology Infrastructure Library
framework (request, incident, problem, change, conguration management and
knowledge management) to ensure a quality service management approach.
• Supplier qualication and audits against set criteria to ensure that quality
requirements are met.
• Project management with a strong application of Agile methodologies to plan, track
and control global projects.
• Strong business relationship management with CIS’s internal business partners to
ensure business alignment of priorities and initiatives.
• Training and development for SAS employees to ensure that skill sets are strong
and relevant.
Environmentally Responsible IT
In today's business landscape, sustainability is a crucial aspect of SAS’ corporate
responsibility and part of our overall quality focus. Information technology (IT)
has become an integral part of business operations, but it also has a signicant
environmental footprint. The increased demand for computing resources, coupled
with the rise of electronic waste, energy consumption and carbon emissions, are
causing negative consequences for the planet.
SAS Cloud and Information Services’ Green IT Strategy is a key component of SAS'
corporate sustainability strategy. We actively manage IT assets to extend asset life and
reduce material impacts. We optimize and monitor energy use over the entire life cycle
to more efciently consume and produce energy. We procure energy-efcient and/or
eco-certied IT assets from suppliers with strong sustainability goals. Lastly, we enable
these activities through operational excellence and IT Operations Management to ensure
the reliability, performance and availability of IT systems and services.
Continuous Improvement
• Root cause analysis to prevent recurring incidents.
• Metrics and analytics to measure, optimize and forecast.
SAS CIS is committed to providing quality technology services. We have documented
IT policies and procedures that outline our approach. All SAS employees with access to
the SAS environment are required to be trained on these policies and procedures. These
policies and procedures are updated and approved annually by CIS leadership, including
the CIO and CISO.
13
^
CONTENTS
QUALITY IMPERATIVE
The Foundation of Quality Development
This section outlines key underpinnings of quality software development at SAS,
including external and internal standards, reliability and accuracy in algorithms and
articial intelligence and project team organization and oversight.
Technical Industry Standards
SAS continuously monitors external technical industry standards and engages with
regulatory and governing organizations to inform the evolution of SAS’ internal quality
processes, including the following:
• Standards and process organizations such as International Standards Organization
(ISO), National Institute of Standards and Technology (NIST) and System and
Organization Controls (SOC).
• Government regulations and programs such as the US FDA Code of Federal
Regulations, the Federal Risk and Authorization Management Program (FedRAMP),
Cybersecurity Executive Order 14028 and all applicable trade and export controls.
• Security organizations and frameworks such as Open Web Application Security
Project (OWASP), National Vulnerability Database (NVD), Common Vulnerability
Scoring System (CVSS) and Forum of Incident Response and Security Teams (FIRST).
• Global privacy laws such as General Data Protection Regulation and California
Consumer Privacy Act.
SAS continuously updates internal policies and standards as technology evolves and
customers’ requirements change, through:
• Continuous monitoring of industry standards, such as ISO, NIST and FedRAMP, and
incorporating relevant processes and controls across SAS where applicable.
• Maintaining strategic and cooperative relationships with leading hardware and
software manufacturers to establish effective integration and performance.
• Contributing to legislation and requirements through active participation in
committees and organizations.
• Encouraging and supporting employee membership in professional organizations
and committees.
• Requiring and enabling advanced cybersecurity training and certications for staff,
such as software developers and technical support engineers.
14
^
CONTENTS
QUALITY IMPERATIVE
R&D Policies, Standards and Processes
R&D uses internal policies, standards and processes to ensure that consistent
development methods, architectural components, software engineering processes and
tools produce quality deliverables for customers. Figure 1 illustrates the relationship
between policies, standards and processes and the software development life cycle.
Figure 1: R&D Policies, Standards and Processes
R&D policies, standards and processes are required aspects of software development,
and provide foundational consistency for coding practices, software security,
accessibility, user interfaces and other factors. They are formally documented and
continually reviewed, rened, expanded and enhanced. Guidelines, plans and templates
are less formal documents, as indicated by the dotted line in Figure 1. They are specic
to the software that is under development, and they help ensure conformance to
required policies, standards and processes.
Accessibility
SAS is committed to enabling people of all abilities to access the power of analytics. To
that end, SAS has established standards, provided centralized accessibility leadership
within R&D and created development and testing support structures for teams. The
internal standard for the accessibility of products, documentation, training and support
materials uses the Web Content Accessibility Guidelines (WCAG) version 2.1 at the A
and AA levels of conformance.
There is a central accessibility team within R&D. This team denes product requirements,
coordinates training, establishes guidelines and checklists, assists in testing applications
for compliance and works with customers to address accessibility questions.
R&D staff receive training in understanding how people use assistive technologies,
testing for accessibility conformance in Web content and how to develop accessible
Web applications. Product teams incorporate automated and manual accessibility
testing methodologies into their development processes. The central accessibility team
works with the product teams to dene accessibility priorities and develop plans for
improving the accessibility of the products. Additionally, testing and user experiences
are designed alongside people with disabilities, so that their lived experience helps
move us beyond conformance to enable people with disabilities to make better decisions
using data.
15
^
CONTENTS
QUALITY IMPERATIVE
The SAS Disability Support Center (https://support.sas.com/accessibility/) provides
direct access to the latest information about the accessibility of SAS products. It
includes links to user documentation for accessibility features, webinars, training and
much more. Users with disabilities can also directly email the central accessibility team
at accessibility@sas.com.
Application Programming Interfaces (API) Standard
The SAS API Standard was developed to guide developers in creating usable, consistent,
and secure APIs.
APIs are protocols that allow for communication and data transfer between separate
applications or between components of the same application. Private APIs are accessible
only between software components of the same family of software, while public APIs are
accessible to external third parties for use in their applications.
All new network interfaces must be Representational State Transfer (REST) APIs or
a binary protocol that supports tunneling over HTTP approved by the API Developer
Experience Team. REST APIs are preferred because they enable ease of use, reuse,
consistency, security and interoperability between all SAS REST APIs and access from
heterogeneous clients. In addition to the REST architecture requirement, the API
Standard also denes how underlying domain resources are accessed, what media
types are accessible, which status codes may be used and other guidelines.
Software Globalization
Internationalization is the process of designing and developing software so that
it functions properly anywhere in the world without requiring re-engineering.
Internationalization ensures that the software can be adapted to meet the language
and cultural needs of SAS customers.
Localization is driven by market demand. It is the process of adapting software for a
particular country or region. Localization includes the translation of text for the user
interface, system messages and documentation.
Software globalization is the combination of internationalization and localization
to ensure that products are global-friendly and can be rolled out worldwide at a
moment’s notice.
All SAS software products must be internationalized. The globalization team produces
standards and guidelines that enable product teams to develop software for users
around the world.
Internationalization:
• Develop software components that handle character data correctly, regardless
of the character encoding.
• Support collation of data according to expected linguistic or cultural guidelines.
• Present dates, times and numbers in a format that is culturally correct.
• Test English and other locales simultaneously during the software development
life cycle to verify that products follow the internationalization standard.
• Support the ability for customers to translate a single report into many languages.
16
^
CONTENTS
QUALITY IMPERATIVE
Localization:
• Localize software on a schedule that allows for simultaneous delivery in English.
• Maximize reuse of translations to ensure consistency and a higher-quality user
experience.
• Be the voice for the non-English-speaking customer.
Software Security
To design, deliver and maintain products that meet customers’ security requirements,
all R&D product teams are required to comply with internal software security policies,
standards and processes. SAS uses the following publications to help product teams
evaluate and remediate security weaknesses and vulnerabilities:
• Common Weakness Enumeration (CWE) list
• Common Architectural Weakness Enumeration (CAWE) catalog
• Common Attack Pattern Enumeration and Classication (CAPEC) catalog
• Open Web Application Security Project (OWASP) Top Ten lists
• CWE/SANS Top 25 list
• Known Exploited Vulnerabilities Catalog from U.S. Cybersecurity and Infrastructure
Security Agency (CISA)
• Mitre ATT&CK framework
For more information, see the papers and links highlighted on the SAS Security
Assurance website (sas.com/en_us/trust-center/sas-trust-security.html). Details on
testing for compliance with internal software security policies, standards and processes
are found in this paper’s Software Security Testing section.
Third-Party Software and Open Source Contributions
The Third-Party and Open Source Software standard mandates a process for
evaluating and managing third-party and open source software that is used internally
for development activities and/or incorporated into production offerings. This process
applies to any software (whether open source or proprietary) that is developed by an
entity other than SAS. It plays an important part in protecting SAS and SAS customers
by evaluating third-party software against a predened set of legal, business and
technical guidelines. Additionally, the Open Source Contribution process provides
guidance for releasing SAS-owned software to the public under an open source license.
Terminology
The SAS Terminology Standard species that all SAS software and supporting
materials must be written in a way that does not contribute to social biases with the
use of insensitive language. SAS uses bias-free language guidelines, incorporating style
and grammar that reect the diversity of the wider SAS community, both internally
and externally.
17
^
CONTENTS
QUALITY IMPERATIVE
User Interfaces
SAS follows a user interface standard that is based on research, usability testing and other
disciplines. This standard is compiled with input and approval from the user experience
design, visual design, accessibility, documentation, internationalization and legal teams.
The SAS user interface standard affects the following areas:
• User interaction and visual design
• Accessibility
• Embedded user assistance and terminology
• Internationalization
• Legal notices
Following this standard promotes these results:
• Ensures that SAS products are usable
• Establishes a consistent look-and-feel for SAS products
• Provides a high-quality user experience to all users
Algorithm Choices
SAS staff reviews the relevant literature and evaluates established algorithms for
numerical stability, time requirements and space requirements. Algorithms are chosen
that provide the best combinations of these sometimes conicting requirements. If
satisfactory algorithms are not found in the prevailing literature, then SAS staff may
perform research to develop better algorithms in-house. All algorithms in SAS software
are tested extensively. Furthermore, the analytical and statistical software
documentation provides sections that cover computational details and references
to source literature.
Numerical Accuracy
SAS’ uncompromising pursuit of accuracy has rmly established SAS software as one of
the most reliable products in the market today. Extensive use of SAS software in medical
and pharmaceutical research, government statistical reporting and government and
academic epidemiological studies attests to customers’ condence in the accuracy
of SAS software.
Two of the most critical issues in software development, especially for analytical
software, are the accuracy and reliability of results. In this context, accuracy describes
the degree of agreement between the reported result and the unique true value, if such
exists. Sometimes, rather than a unique solution, any solution from a set of solutions is
also acceptable. Reliability is a more subjective measure, considering the degree of
condence in the accuracy of the result.
There are two factors that affect the accuracy of a computed result. The rst factor is
the hardware’s ability to represent real numbers in nite precision. Not all real numbers
can be represented in binary nite precision and that means that representation of real
numbers might introduce errors because of binary rounding. Arithmetic operations might
also introduce rounding errors.
18
^
CONTENTS
QUALITY IMPERATIVE
The second factor is the software itself. Internally, for analytical computations in SAS
software, all numeric representations, functions and operations are calculated by using
double-precision, oating-point arithmetic that offers the maximum level of precision
provided by the underlying architecture. Although single-precision arithmetic allows
only 6 to 7 signicant digits, double-precision arithmetic allows 15 to 16 signicant
digits. Accuracy might be further limited by the algorithms that are selected and by the
implementation strategies. Algorithms must be chosen carefully and coded to achieve
optimal performance, as measured by speed, efciency and precision.
At SAS, developers carefully select and code efcient algorithms for numeric operations
to guarantee a reasonable number of correct digits and the maximum domain of
evaluation on most machines. When the software cannot guarantee this predetermined
level of accuracy, it is designed to return a missing result rather than a potentially
inaccurate result.
In response to industrial concerns about the numerical accuracy of computations from
statistical software, NIST’s Information Technology Laboratory provided data sets
with certied values for a variety of statistical methods (NIST 2007). As one of many
approaches to ensure accuracy, SAS integrates NIST data into automated tests and
compares SAS results to the results that are supplied by NIST.
Developers take steps to verify that SAS works correctly with operating system
datetime functions. In general, developers rely upon operating system datetime values
and perform checks to ensure correct functioning for special situations, such as daylight
savings time.
SAS procedures have numerous options that alter the nature and extent of output.
However, the same output is always produced with the same options, even across
hardware and operating systems within standard machine precision limits (typically
1E-12 or smaller). This assumes that the same random number seed is specied for
algorithms requiring pseudo-random number generation.
For more information about validating a statistical procedure, see Appendix 2: Validating
an Analytical Component.
For more information about numerical precision, see the technical paper Assessing the
Numerical Accuracy of SAS Software. (http://support.sas.com/rnd/app/stat/papers/
statisticalaccuracy.pdf)
Trustworthy Articial Intelligence
SAS continues our rich tradition of leading the growth of responsible innovation in the
eld of analytics. Trustworthy articial intelligence (AI) and responsible innovation are
an integral foundation of SAS and have been for decades. Customers in every industry
capitalize on SAS’ innovations to gain insight and help responsibly operationalize their
business strategy. At SAS, we are guided by our core principles of human-centricity,
inclusivity, accountability, robustness, transparency and privacy and security in creating
our analytic capabilities, products and platform. We believe that trustworthy design,
development, deployment and use of analytics and AI-driven solutions help ensure
sustainable improvements for our customers, the economy and society.
19
^
CONTENTS
QUALITY IMPERATIVE
SAS’ commitment to trustworthy AI builds on our already strong foundation in AI,
which includes:
• Machine learning
• Advanced analytics
• Deep learning
• Natural language processing
• Optimization
• Forecasting
• Fraud detection
• Computer vision
• Data lineage
• Model governance
• Decisioning
SAS’ innovation and capabilities aim to address the imperative toward the responsible
development and implementation of AI. SAS engages with policymakers, regulators,
stakeholders and like-minded organizations to help shape the future of AI regulations
and ensure that SAS and our customers remain at the forefront of these developments.
Organizations should operationalize AI in fair, transparent, accountable and carefully
managed ways. SAS has the resources, expertise and products to help our customers
meet their business objectives while considering applicable legal requirements. SAS
supports customers with comprehensive training on the responsible use of AI, model
interpretation and model management.
Furthermore, diverse teams are more likely to create solutions that anticipate unfair
bias and take steps to avoid or mitigate it. SAS’ diversity and inclusion efforts encourage
multidimensionality both within SAS and beyond, for example, through SAS’ investments
in fostering diversity in STEM talent and traditionally socially disadvantaged businesses.
Shared Sublibraries and Code Reuse
We have a rich tradition of reuse, and we regularly use our prior work as the building
blocks of innovations in applications. SAS software products share the same sublibraries
or components. As a result, there is a high incidence of code reuse. Developers are
encouraged to reuse routines when possible. Each routine is tested in the development
environment and then across supported operating environments. We have several reuse
categories of shared components:
• Golang libraries
• MultiVendor architecture (MVA) and threaded kernel libraries
• SAS Component Language code
• .NET
• Java libraries
• JavaScript libraries
Low-level, reusable modules are unit-tested and then used in developing more complex
modules. Shared components are tested on machines with multiple SAS releases and are
also tested after installing or uninstalling releases. The benet of shared sublibraries is
20
^
CONTENTS
QUALITY IMPERATIVE
that a signicant percentage of code has been tested collectively across a wide variety
of operating and computing environments. SAS is built with source code and components
that are reused from release to release. This adds stability to the software because each
successive release of SAS software inherits code that has been tested and used in prior
releases. Features such as security and compliance are centralized, ensuring they’re
implemented and tested by domain experts, while providing consistent functionality
across SAS software.
R&D Engineering Standards
In addition to the other policies, standards and processes in this chapter, there are
additional mandatory R&D engineering requirements, primarily covering internal CI/CD
processes. These various requirements are included as part of the R&D Engineering
Standards.
Supporting R&D Quality Development
At SAS, every executive, manager and product team member are responsible for quality.
By intelligently structuring teams and promoting a culture of quality throughout the
company, SAS can support and encourage the innovation and creativity that our
customers have come to expect.
Product Teams
Product teams are the heart of software development at SAS. Product teams are cross-
functional, typically including developers and testers, product and project managers, and
other roles determined by product functionality and development phase. A loose matrix
organization among diverse product teams enables each team to customize processes in
an agile manner, while guided by internal policies, standards and processes. This exible
relationship encourages high degrees of innovation and facilitates knowledge sharing,
while ensuring product consistency and interoperability.
Product teams use retrospectives to improve processes and software quality. During
the retrospective process, teams discuss project details and identify opportunities for
improvement. Teams then develop a concrete plan of action for implementing the changes
and following up on the results. Over the course of the software development cycle, teams
may perform retrospectives at any or all of these times:
• Periodically within a release cycle.
• On a release boundary.
• On an ad hoc basis to immediately address a specic issue.
In addition to retrospectives, development teams use other continuous improvement
methods to improve software quality and the customer experience. These include a
focus on personal and professional development, sharing best practices, conducting both
internal and external usability reviews, and acting on customer feedback.
21
^
CONTENTS
QUALITY IMPERATIVE
Project Management
SAS R&D project managers play a key role, supporting the product team’s agile
processes and enabling teams to deliver projects on time, within budget, and according
to functional and quality specications. Each offering is assigned a project manager who
works with a product team to scope the work and establish a schedule. The scope of
work and planned schedule become the foundation for the ongoing tracking and
oversight of the project. On an ongoing basis, they work with product teams to identify
project risks and develop mitigation plans to address them. When actual status deviates
from the plan, project managers work with team management to determine and
implement actions to get the project back on course. Examples of actions can include
changes in scope, timelines, resource allocation and so on. Throughout the project,
project managers ensure adequate visibility into the overall health of the project via
status reporting, project reviews and surfacing project data to a companywide scorecard
to ensure that the project is meeting the stated goals and objectives. Status reporting
and project reviews occur at both the individual team level, as well as at the executive
and enterprise level.
Quality and Compliance Team
The R&D Quality and Compliance team drives continuous improvement of R&D quality
and security practices to minimize risk to SAS and our customers by:
• Expanding and supporting quality directives and initiatives throughout SAS through
education, facilitation and publication.
• Ensuring that R&D policies, standards and best practices are aligned with corporate
vision and are dened, communicated, adopted and measured.
• Achieving, expanding and maintaining information security and related industry-
supported compliance certications.
• Providing expertise on quality practices, standards, internal controls and external
certications.
Quality Review and Oversight
The R&D Executive Team consists of executive representatives from across the R&D
organization. This team provides cross-divisional management and oversight of all
R&D product releases. The team meets regularly to evaluate the progress, quality and
readiness of upcoming releases.
Dashboards visible to all R&D employees surface key quality and progress metrics from
each development project. The R&D Executive Team reviews these dashboards regularly
to anticipate areas of concern and develop targeted mitigation strategies, such as
resource-balancing across teams, to better help teams deliver on their schedule,
feature and quality commitments. The R&D Executive Team encourages openness
and transparency in progress reporting so that teams can comfortably surface project
concerns to the R&D Executive Team, condent that the information will be received
and dealt with in a fair and positive manner. The Chief Technology Ofcer is accountable
for the overall quality of SAS software.
22
^
CONTENTS
QUALITY IMPERATIVE
SAS
®
Architecture: Quality by Design
SAS
®
Viya
®
SAS Viya was created to deliver an elastic and scalable cloud-ready analytics engine that
embraces open analytics coding environments. SAS Viya provides a unifying environment
or the entire analytics life cycle, with powerful analytic techniques that are accessible from
a variety of interfaces, including programming, scripting and visualization. These include:
• A multicloud architecture with no infrastructure lock-in. SAS Viya can scale to
accommodate growing data volumes, more users, or more complex analytics. SAS
Viya supports both public and private cloud deployments.
• Supporting a single, consistent platform for management of the entire analytics life
cycle, which is open to both SAS and other programming languages such as Python,
R, Java and Lua calling into a single, underlying analytics code base.
• Providing access to analytic techniques (machine learning, descriptive statistics,
forecasting methods, optimization algorithms and so on) from a variety of interfaces
– programming, scripting and visualization.
• Automatically distributing data and analytical workloads across the cores of a single
server or the nodes of a massive computing cluster, taking advantage of parallel
processing regardless of data size.
Consolidated Analytic Environment
SAS Viya can be accessed via modern visualization clients, REST APIs and interfaces
from other programming languages. The SAS Viya analytic procedures are consolidated
in SAS Cloud Analytic Services (CAS server) with a single point of administration and
management. All interfaces to SAS Viya access this layer for analytic processing so that
no matter how users interact with SAS Viya, they receive consistent results.
Cloud-Ready Technology Stack
SAS Viya is built on a cloud-ready technology stack. From the SAS Cloud Analytic
Services that power SAS Viya analytics at the core to the microservices that supply the
REST APIs and functional interfaces, SAS Viya is built to be cloud native. SAS Viya uses
open-source languages and technologies such as Go, Java and Spring Boot to deliver a
set of microservices. These microservices support common functionality, such as login
and authorization, identity management, preferences, auditing, data management, data
access and more. SAS Viya uses the OAuth open standard for authorization, allowing
SAS Viya to integrate with third- party clients and services. SAS Viya also provides
public REST APIs and uses TLS to secure communications. SAS Viya can deploy on
IaaS providers like AWS and Microsoft Azure, as well as on premises in private cloud,
virtualized and physical machine environments.
23
^
CONTENTS
QUALITY IMPERATIVE
SAS
®
Viya
®
4
SAS Viya 4 is a continuation of SAS’ journey to a fully cloud-native and optimized
architecture. A system capable of managing the full analytics life cycle requires many
different components.
Figure 2: SAS Viya 4 Architecture
Cost-Efciency
Running enterprise software in a cost-efcient manner is not achieved through a single
decision or goal. It requires a coordinated set of tasks that include limiting the baseline
cost, allowing scale where necessary, and allowing an “off” switch when the software is not
in use. The SAS Viya 4 architecture addresses each of these concerns:
• Many components have been moved from Java to a more memory-efcient
platform, Golang.
• Components can be scaled individually, enabling customers to allocate cost where
necessary to meet their business goals.
• SAS provides mechanisms within its deployment to turn off (and back on) as much
of the system as practical when it is not being used.
Containers and Kubernetes
SAS delivers its Viya 4 software via container images. Container images are a
deployment currency of the cloud due to their isolation properties. Using containers
enables us to ensure that required libraries are installed correctly and that compute-
related resources are utilized and shared appropriately. Containers delivered by SAS
use the Open Container Initiative specication and are compliant with it.
Managing many different containers can be a challenge without a framework and tooling
in place designed for it. Kubernetes is a standard answer provided for today’s cloud. By
deploying in Kubernetes, SAS takes advantage of its many features to deliver a solid
administrative experience such as automatic handling of restart, managed updates
without disruption and the scheduling of diverse workloads on appropriate hardware.
24
^
CONTENTS
QUALITY IMPERATIVE
Continuous Delivery
Another characteristic of cloud-native software is increased frequency of updates.
Features and xes are made available to consumers when they are ready rather than
waiting for a coordinated roll-out at some predened interval. To achieve this, SAS has
incorporated common continuous delivery practices into its software factory.
As code is being proposed for integration to the mainline branch, it goes through an initial
set of validations that include peer review, unit tests, basic integration tests and linting.
Once merged, code goes through a set of increasingly narrow gates dealing with complex
integration scenarios, performance validation and more in-depth functional tests. Only
after passing all of these gates does it make it to a point of being customer-visible.
Frequent releases do not completely remove the need for cross-team collaboration.
In these instances, SAS utilizes feature ags to avoid enablement of code prior to the
coordinated release.
SAS
®
9.4
At SAS, quality by design is evidenced in several areas such as:
• The use of intelligent components (intelligent clients, intelligent storage,
intelligent servers)
• SAS software’s MultiVendor Architecture
• Shared sublibraries and code reuse
• The use of maximum numerical precision
Third-party components are often integrated into SAS offerings, making intelligent
architecture a necessity. Our intelligent architecture is used for SAS Foundation, the
SAS intelligence platform and SAS solutions.
SAS
®
Foundation
SAS Foundation is based on MultiVendor Architecture, which facilitates developing,
managing and maintaining the source code of the system and enabling quality to be built
in at the structural level. Because the SAS Foundation development process is built
around SAS MultiVendor Architecture, the amount of code that is rewritten for each
operating environment on which our software runs is minimized. Thus, the chance for
errors decreases because about 85% of the code is reused on all operating environments
that are supported by SAS software.
The fundamental goal of SAS MultiVendor Architecture is to provide the highest degree
of portability across a broad range of operating environments while exploiting the
advantages of each.
25
^
CONTENTS
QUALITY IMPERATIVE
SAS
®
Intelligence Platform
SAS 9.4 uses an n-tier architecture that enables distributed functionality across
computer resources so that each type of work is performed by the resources that
are most appropriate to the job. For a large company, the tiers can be installed across
multiple machines with different operating systems. For prototyping, demonstrations
or small enterprises, all the tiers can be installed on a single machine. The architecture
consists of the following four tiers: data sources, SAS servers, middle tier and clients.
SAS 9.4 middle-tier components include a service-oriented architecture that is built
around its Web Infrastructure Platform. Java J2SE and J2EE technologies, which are
portable and reusable, are used for desktop client and web application components
of SAS
®
9.
The adoption of J2SE and J2EE technologies enables SAS to use the development,
testing and customer acceptance baselines of the broad Java vendor and customer
community. These technologies take advantage of Java’s portability to operating system
and application server deployment environments. This further enables customers to use
the IT standards, acceptance testing, support stafng and operating practices that are
established within their organizations for this class of application.
For more information, see “Architecture of the SAS Intelligence Platform,” in SAS 9.4
Intelligence Platform: Overview, Second Edition (https://go.documentation.sas.com/doc/
en/bicdc/9.4/biov/titlepage.htm).
SAS
®
Solutions
SAS solutions provide industry-specic functionality in these key focus areas: analytics
platform, articial intelligence and machine learning, customer intelligence, data
management, fraud and security intelligence, risk and retail. Most solutions extend SAS
architecture by using a component based on Java for the business logic and surface the
functionality through web-based thin-client and rich-client presentation layers. Using
SAS architecture gives SAS solutions several advantages:
• Solutions can scale from one machine to multimachine implementations to meet
the performance needs of the customer.
• The common metadata repository enables common data sharing and management
across systems.
• Most important, the SAS architecture enables the solutions to draw upon the
analytical power of SAS to differentiate SAS solutions from those of competitors.
26
^
CONTENTS
QUALITY IMPERATIVE
Research and Development at SAS
The Research and Development Division of SAS drives software research, development
and production. This section provides an overview of information sources used for SAS
R&D research and the steps in the development process. Subsequent sections provide
details on each step in the software development life cycle. The SAS software
development life cycle continuously evolves as R&D embraces proven industry best
practices and improvements. This section reects the software development life cycle
currently in operation at SAS.
To meet customer needs and industry requirements, JMP uses specic renements,
processes and tools, as described in JMP: A Commitment to Quality
(jmp.com/qualitystatement).
Research
The SAS software development process begins with gathering ideas for a potential new
product, function or enhancement. Ideas for new or enhanced functionality and
architecture are collected from information sources such as:
• Customer Advisory Board, councils and focus groups.
• Feedback from consultants, development partners, early adopters and customers.
• Analyst research.
• Market research.
• Professional conferences and communities.
• The SASware Ballot
®
(https://communities.sas.com/t5/SASware-Ballot-Ideas/
idb-p/sas_ideas).
• Technical Support (sas.com/support).
• Feedback from SAS Education Division courses.
• Usability and accessibility studies.
• SAS Innovate, SAS Explore, and regional, international and special interest user
group meetings.
This information is collected in various input documents and made accessible to team
members. Teams work together to evaluate emerging technologies and architectures,
exploring and experimenting to determine optimal solutions.
27
^
CONTENTS
QUALITY IMPERATIVE
Software Development Life Cycle
SAS’ software development life cycle involves the phases shown below in Figure 3.
As part of continuous improvement, SAS is actively and rapidly rening software
development life cycle methods to align with DevOps principles and continuous
integration/continuous delivery approaches more closely.
Figure 3: The Software Development Life Cycle
Subsequent sections describe each phase of the diagram in detail:
• Plan (see the Planning and Requirements section)
• Code (see the Code and Build section)
• Build (see the Code and Build section)
• Test (see the Testing section)
• Deploy (see the Deployment section)
• Release (see the Release section)
• Maintain (see the Maintenance and Support section)
• Support (see the Maintenance and Support section)
28
^
CONTENTS
QUALITY IMPERATIVE
Planning and Requirements
In the planning phase of SAS’ software development cycle, product teams estimate
tasks, determine resource needs, identify risks and verify that the project team and
management are committed to the plan. Product teams create road maps to capture
longer-term plans, including the main themes of upcoming releases. Prioritized features,
requirements and updates are captured in a product backlog. This prioritized product
backlog is continuously rened throughout the software life cycle.
Project managers monitor, track and review progress with the team. Planning and
managing software development projects enable product teams to achieve the intended
project outcomes. Teams continuously revise release plans, prioritize enhancements and
allocate resources based on progress. See R&D Project Management for more details.
The product development team works closely with product management to create
detailed requirements from the prioritized backlog. Requirements identify a capability,
technical characteristic, or quality factor that bounds a product or process problem
for which a solution can be pursued. Requirements analysis and validation include
determining whether they are necessary, understandable, achievable, complete,
unambiguous and veriable. Requirements are documented and continuously reviewed
and prioritized as described above.
The product team translates requirements into user-interface concepts and interaction
designs. They may design APIs, create user ows and build interactive prototypes that
anticipate end user behavior. Product teams create a high-level architecture for the
software based on the requirements and designs.
29
^
CONTENTS
QUALITY IMPERATIVE
Code and Build
Product development teams write code and tests based on R&D policies, standards,
processes and coding guidelines. Product development teams also write, update and
implement automatic and manual testing based on test plans and testing guidelines,
as explained in more detail in the Testing section. Working code, automated tests and
documentation are managed in a centralized source management system. Code is tested
and evaluated for stability on an ongoing basis. After passing all automated tests and
audit procedures, code is then integrated into the build image on the platforms that are
scheduled for release. This process iterates continuously throughout the code and build
phases of the software development life cycle.
Problem Reporting and Resolution
Coding includes testing, both automated and manual. If tests fail, the problem is
reviewed to determine whether it is a problem with the test, the documentation or
the code. If the test has an error, the test is updated. If the documentation has an error
or needs additional clarity, documentation resources are notied and make updates. If
the problem is in the code, testers and developers determine the root cause, identify
potential solutions and implement xes as needed. All problem reporting and resolution
activities are tracked in a ticketing system.
High-priority problems include those that cause system failure or that produce incorrect,
unreliable or misleading results. Problems that result in the loss or corruption of data,
performance degradation and potential security vulnerabilities are also considered high-
priority. The same is true of problems that depart signicantly from intended product
function. Low-priority problems include nonfunctional cosmetic features or problems for
which there are convenient workarounds. These problems might be deferred for xes in
later releases of the software. Product teams consult with Product Management and
Technical Support to prioritize the urgency of a problem given their understanding of
current customer use and user feedback. In all cases, problems are tracked through
problem tracking systems until the problem is resolved and veried.
30
^
CONTENTS
QUALITY IMPERATIVE
Change Control
Throughout the software development life cycle, strict control is maintained over all
source code, which the company protects as a principal asset and trade secret. The
toolset, which is routinely updated to take advantage of the latest technologies,
controls development access to source code and the ability to make changes and xes.
Through the source management system, developers can check out source code into
their private work area for changes and xes. During this time, other developers can
simultaneously check out the same les. The source management system automatically
evaluates changes at check-in and ags differences. Code is merged automatically
unless the differences require manual intervention. Teams may also rely on capabilities
such as role-based access control, linking changesets to ticketing systems and
mandatory code reviews before code can be merged. The source management system
logs all source code changes.
Revision history is kept for all modules in source management, thereby maintaining
earlier versions in addition to a history of who made changes and why those changes
were made.
Version control methods, such as semantic versioning, assign each software build a
unique identier.
31
^
CONTENTS
QUALITY IMPERATIVE
Testing
Overview
SAS R&D staff embrace both the benets and challenges of measuring software quality
throughout its life cycle. Through the use of continuous integration techniques and
continuous automated testing, the effort to ensure that our software meets or exceeds
customer expectations never ends.
Testing software requires skilled software professionals, effective development and
testing processes, and robust test automation tools. Given our CEO's core philosophy
that staff are the company's greatest asset, SAS actively recruits, trains and retains
highly qualied Software Development Engineers in Test (SDETs). These SDETs are
skilled in incorporating domain and product expertise into the development of
automated testing. Early in the development life cycle, SDETs engage with product
management to understand and rene both functional and non-functional software
requirements and to increase their testability. Team members jointly contribute to
enhancing test strategies, including how to best automate and implement a cohesive
testing program for their product.
Software development and testing team engineers meet regularly to discuss process
improvement opportunities and test automation innovations. A central quality portal
and internal testing site provides guidance and best practices. Key management roles
participate in formal quality reviews and must attest to the product’s quality before the
software may ship.
Testing professionals work closely with software engineers and product management
to verify the successful implementation of new features and to validate the continued
baseline of existing features. Teams choose from an evolving set of testing
methodologies, including requirements-based testing, use case and misuse case testing,
exploratory testing, consumer-driven contract testing and systems testing. Product
teams also collaborate with specialized teams within SAS to ensure their testing
incorporates key areas such as software security, performance, accessibility, localization
and other dimensions of quality.
When dening a test strategy for their products, product teams perform these tasks:
• Provide feedback on software requirements and design.
• Validate acceptance criteria for product requirements.
• Write and support test procedures, automation tools and reusable test libraries.
• Perform early exploratory testing.
• Execute automated tests and manual test scripts.
• Report test results, including discovered defects.
• Verify xes and perform regression testing.
• Report test results.
• Monitor quality metrics.
• Analyze and improve test coverage.
• Review customer documentation.
• Test nonfunctional requirements, such as performance and security.
• Test for adherence to R&D policies and standards.
• Write and support qualication tests and samples.
32
^
CONTENTS
QUALITY IMPERATIVE
Directors and managers are accountable for ensuring that product development teams
follow SAS’ quality procedures.
Test Documentation
Each product team is responsible for planning the overall testing strategy for their
projects. Teams also use agile approaches for maintaining project-specic test plans and
documentation. The entire R&D community collaborates on a due diligence checklist to
ensure consistency for testing sign-off. This checklist serves as a source document when
planning the test strategy for a software release.
Teams may use a variety of approved test tools and processes—both internal and third
party—to document and manage test artifacts and test results. Group test plans
document the testing strategy for an entire group or department. Teams may develop
individual test plans for specic projects, products, solutions or features.
Where appropriate, teams may perform early exploratory testing during the planning
and documentation process. This technique enables teams to get acquainted with the
software, understand changes in the current release and begin designing automated
tests. Little or no formal documentation is created for this type of testing, although
defects can be entered if anomalies are uncovered. As the test automation matures,
teams generate more robust documentation of expected results and permutations of
use cases under test. Test documentation covers not only product functionality, but
also input validation, error-handling capabilities and stress or performance testing.
Test Cases
Teams maintain baseline suites of legacy tests to verify product functionality delivered
in previous releases. Engineers continuously improve test suites by designing and writing
tests to validate new functionality and store the versioned tests in a source code
repository.
Engineers design test cases to determine if specic components of the software
perform correctly. Correctness of a test result is evaluated against existing benchmarks,
knowledge of past performance, expected behavior as identied in documented
requirements, known results as published and an understanding of the software’s design.
Product teams employ a variety of testing methodologies to verify and validate software,
based on the specic needs of the product under test, such as:
• Analytical tests, including statistical and numerical validation. See Appendix 2:
Validating an Analytical Component for details on how SAS handles this critical
testing step.
• API and unit tests to verify the correct behavior of components before system
integration.
• Compatibility testing to assess the ability of software or web applications to function
across different browsers, congurations or cloud platforms.
• Error testing examines how syntax and run-time error conditions are handled.
• Functional testing determines whether the software functions as expected.
• Internationalization testing checks the software readiness for any required
localization development. Localization testing is done by native language users.
• Migration testing checks that customers can move to current versions of the software
without problems.
33
^
CONTENTS
QUALITY IMPERATIVE
• Performance testing evaluates whether the software performs at levels that are
consistent with customer requirements in production.
• Regression testing identies whether software changes have introduced errors or
unintended behavior.
• Security testing can include verifying role-based access control and least privilege;
evaluating susceptibility to risks such as OWASP Top 10 weaknesses, CWEs, Common
Vulnerabilities and Exposures, encryption mechanisms; and assessing error handling,
input handling and application programming interface security.
• Stress testing creates an overload situation to determine how the software product,
procedure or module functions under the stressed condition. Stress testing also
evaluates the ability of the software to recover from overloaded conditions, to
measure how the project performs under peaks and ebbs in use.
• System testing validates the complete and fully integrated product in a production-
like environment.
• Usability testing and accessibility testing evaluate usability of a particular feature for
all customers, regardless of ability.
Testing Tools
Teams employ a broad range of testing tools:
• Multiple industry-standard third-party and open source test automation frameworks,
such as Selenium, Microsoft Playwright and others.
• Coverage analysis tools for C, Java, Go and JavaScript source code to highlight
potential areas for improved coverage and optimize regression testing.
• Test drivers that execute command-line-based tests on multiple platforms (cloud
platforms, operating systems and programming environments).
• Internal and third-party continuous testing tools to use unit, integration and
acceptance test suites for continuous quality.
• Test management and tracking tools to record test cases and their results history for
all platforms.
• Internal and third-party tools for continuous integration and validating software xes.
• Security testing tools that scan for common software security vulnerabilities under
both static and dynamic conditions. For more detail, see the Security Testing section.
• Performance monitoring tools.
• Problem tracking tools such as Jira for tracking defects, enhancements, issues and
suggestions found during and after software development and testing.
Various reports such as the following are available to teams for evaluating software quality
and testing progress:
• Queries of the number, age, type and severity of defects by product.
• Verication status of individual defects and responsible individuals or departments.
• Trending quality of code by tracking the rate of incoming defects versus the rate of
resolving defects.
• Stability of code by tracking the number of test and source les pushed within a
given period.
34
^
CONTENTS
QUALITY IMPERATIVE
Test Execution
Automated test suites are executed using both internal and commercially available tools.
Such testing is typically repeated on several test congurations. Engineers may also
execute manual test scripts to verify more complex aspects of the software that do
not lend themselves easily to automated testing. Test results are available, typically
in graphical or dashboard format, and are saved to a repository for traceability and
diagnostic purposes.
Performance Testing
Performance testing consists of applying specic loads and then examining various
aspects of the system under test. As part of this effort, exploratory testing and
regression testing is executed across releases and relative to other versions of SAS.
For new releases and redeveloped solutions, performance requirements are provided
by product management. One primary cloud platform is tested for performance
characteristics by many of the product groups. Other platforms are tested based on
customer usage and technological differences in architecture and resources. Performance
testing results are retained for future release comparison.
The testing groups use internal and third-party tools to test code and I/O performance,
big data scalability, algorithm effectiveness against third-party databases and SAS
internal data sets. Much of the work is automated, and parameters are set so that
performance bottlenecks are agged for analysis. The utilization of computer resources
is also assessed, including memory, I/O and CPU. For deep analysis, monitoring and
proling tools are used.
Performance of baseline, peak, concurrent and endurance load-levels are conducted
on desktop, mobile and web applications. This testing is based on multiuser scenarios
driven under load conditions using application load testing tools. This ensures software
quality by identifying performance bottlenecks, memory leaks and scalability problems.
Performance engineers provide advice to product development teams and product
management on code changes, data architecture changes, application architecture
changes and technical architecture or hardware. Performance engineers also provide
recommendations for cloud provider infrastructures that are most appropriate for SAS’
software offerings.
Software Security Testing
Overview
Product teams are required to perform security tests in accordance with internal
software security policies, standards and processes. Security testing includes security
function testing, application vulnerability testing, dynamic scanning of applications,
static source code scanning and container image scanning. All software components,
including third-party components, are required to be scanned against Common
Vulnerabilities and Exposures published in the National Vulnerability Database,
maintained by the US government’s National Institute of Standards and Technology.
35
^
CONTENTS
QUALITY IMPERATIVE
In addition to scanning web applications and the web application server environment, SAS
uses a suite of tests that are specic to SAS technology. Depending on the software type,
these tests can include:
• Industry-recognized security scanning approaches to ag common security issues,
such as those identied by OWASP, CWE, CAWE and CAPEC.
• Testing with users of different role-based security access to make sure that each
user has the appropriate access levels.
• Data access, based on row-level permissions, to conrm that data authorization
is applied appropriately for each user.
• Password and encryption security.
• Correct behavior with Transport Layer Security enabled protocol (HTTPS).
• Validated credential protection when using SAS/ACCESS
®
engines to connect to
data sources (for example, user ID and password).
• Product-specic security tests for appropriate user authorization and error testing.
• Integration testing of security features and controls.
• Penetration tests for some congured deployments.
SAS software security testing tools are focused on eliminating known application
weaknesses such as those described in OWASP, CWE, CAWE and CAPEC. Issues that
are detected during security testing are entered into the problem reporting system and
evaluated promptly for appropriate xes and resolutions.
SAS takes the following steps to deliver secure applications:
• Education and training: SAS provides ongoing training in techniques to mitigate
development errors and vulnerabilities. SAS licenses tools that are designed to
generate test cases for security vulnerabilities, including those described on
OWASP and CWE lists as well as other lists.
• Deliver shared security components across SAS products: SAS develops shared
components and coding guidelines for common issues and robust input sanitization
to provide strong security protection across SAS products.
• Monitor and analyze industry issues: SAS monitors and analyzes industry issues
regularly, and draws on the evolving information from OWASP, CWE, CAWE and
CAPEC to evaluate and remediate identied security weakness and vulnerabilities.
• Frequently update security analysis tools and techniques: SAS performs vulnerability
testing using the most current tools and techniques for feature and maintenance
releases.
Results of vulnerability tests and scans conducted by SAS are company condential.
By policy, SAS does not share the tests or the individual results.
36
^
CONTENTS
QUALITY IMPERATIVE
Customer Notication
SAS provides several forums where customers can get information about updates to SAS
products, including security xes. The SAS security bulletins page (http://support.sas.com/
security/alerts.html) provides updates about security issues. Security xes for released
products are highlighted through the standard technical support process for hot xes,
including the SAS support community. Customers can subscribe to the community or sign
up for support newsletters (or both) to receive regular updates about hot xes and other
important news from SAS.
• To subscribe to SAS Technical Support News, go to http://support.sas.com/techsup/.
• To subscribe to the SAS support community, see https://communities.sas.com/t5/
Getting-Started/How-to-learn-about-hot-xes-to-SAS-software/ta-p/283553.
Note that results can be ltered using the keyword SECURITY.
37
^
CONTENTS
QUALITY IMPERATIVE
Release
Sign-Off
Product sign-off occurs when the following conditions have been satised:
• Planned new functionality has been implemented and tested.
• Requested xes have been implemented and tested.
• Quality metrics meet release criteria.
• Due diligence completed.
The R&D director, development manager, test manager and other key roles conduct
reviews according to due diligence guidelines. The due diligence guidelines include
enterprise attributes and other quality expectations that all software must meet before
the software can be released. Once everyone agrees, the product team completes the
required sign-off and the DevOps Division provides external access to the software.
Production Media
Production software is available by software download for most releases.
Software Production for Target Audiences
SAS meets the challenges of delivering high-quality software by following a clearly
dened rollout process for our new releases. The phases of the rollout process are linked
to internal milestones and are dened by the target audience for each software
development release: development partners, early adopters and general availability.
• Development partners phase. A preproduction software development phase in which
software is provided to customers who have contractually agreed to use the software,
and to provide feedback to SAS about its features and functionality. The goal of
this phase is to validate that the software is being developed according to the
requirements that have been identied by marketing specialists. This phase is
optional, but is most frequently applied to newly developed offerings or major
enhancements to an existing offering.
• Early adopters phase. A preproduction software development phase that occurs after
much of the development has been completed. In this phase, a copy of a software
offering is provided to a customer who has contractually agreed to install and use the
software, and to provide feedback to SAS. Problems that are reported from customers
might be addressed during later phases of this same release, and features or
enhancements are collected for consideration in a future release.
• General availability phase. A software development phase in which the nal
production release of an offering is made available to all customers.
38
^
CONTENTS
QUALITY IMPERATIVE
Virus Protection
To mitigate the risk to our customers, SAS has dened an architecture that mitigates the
risk of viruses being introduced during the release process. Compilation and linking are
performed on Windows nodes protected with the Cisco Secure Endpoint management
software and with minimal network access. After les are compiled and linked, the
remainder of processing is completed on FreeBSD build bubble nodes.
Once products are ready for customer delivery, all components for all platforms are
stored on protected UNIX le servers, which do not allow uploads from Windows hosts.
After a product goes into production, additional restrictions are placed on the
components so that only a limited number of UNIX hosts can write to them.
Digital Signatures
Digital signatures ensure the integrity of SAS software. All SAS components that interact
with the operating system (or that otherwise require digital signatures to work properly)
are signed using a trusted SAS certicate. Windows executables, installation les,
Outlook plug-ins and extensions, various Java les, and other pieces are signed
as required.
What’s New
A list of changes and enhancements for each release is available by selecting What’s
New in SAS at http://support.sas.com/documentation/whatsnew/index.html or
accessing the Help menu provided with the software.
39
^
CONTENTS
QUALITY IMPERATIVE
Deployment
SAS
®
Viya
®
SAS Viya applications are deployed as containerized applications, improving deployment
efciency and conguration exibility. Deploying containers allows SAS to ensure that
required libraries are installed correctly and that compute-related resources are utilized
and shared appropriately. The high-level steps in a SAS Viya deployment are:
• From my.sas.com, customers download deployment assets for their order.
• Customers create a manifest based on a le that is customized to their environment.
• To deploy the software, customers run a command against the manifest that
they created.
For all details on how to deploy SAS Viya, see the SAS Viya Deployment guide at https://
go.documentation.sas.com/doc/en/itopscdc/v_012/dplyml0phy0dkr/titlepage.htm.
SAS
®
9
SAS
®
9 can either be deployed on-premises or hosted as a managed application. For more
information about SAS’ managed application services, see https://www.sas.com/en_us/
solutions/cloud/sas-cloud/managed-application-services.html.
Whether installed and congured on a single machine or in a distributed environment,
the SAS deployment process involves the following high-level steps:
• Planning the deployment.
• Creating users and groups and designating ports.
• Creating a SAS Software Depot.
• Installing required third-party software.
• Installing and conguring SAS.
For all details on how to deploy SAS
®
9, see the SAS
®
9 Administration Guide at https://
go.documentation.sas.com/?cdcId=bicdc&cdcVersion=9.4&docsetId=biwlcm&docsetT
arget=home.htm&locale=en.
40
^
CONTENTS
QUALITY IMPERATIVE
Maintenance and Support
Quality in SAS Technical Support
SAS provides customers with 24-hour, follow-the-sun technical support via telephone,
email, chat and online support as part of the annual licensing fee. Customers with
questions and problems that are related to SAS software can contact Technical Support
as follows:
• Telephone (9 a.m. – 8 p.m. ET)
• Chat (9 a.m. – 6 p.m. ET)
• SAS Customer Support site at https://support.sas.com/en/support-home.html
(available 24/7/365)
• Email (available 24/7/365)
Problems that cannot be resolved immediately are routed to subject matter experts, who
prioritize problems based on severity. SAS Technical Support strives to return initial calls
within a two-hour period for severe problems, and up to a maximum of 24 hours for less
severe problems.
Customers experiencing critical software problems should contact SAS Technical
Support by phone. The average telephone hold time is fewer than 30 seconds, and
approximately 60% of questions are resolved within 24 hours. For critical problems that
occur outside of the business hours listed above, customers are directed to one of SAS’
worldwide support centers in North America, Europe or Asia/Pacic. Using global
resources, SAS can provide customers with 24-hour, follow-the-sun support.
SAS Technical Support takes pride in providing fast and accurate responses to customer-
reported questions and problems. However, a key goal of SAS Technical Support is to
empower customers with the tools they need to nd answers and resolve problems on
their own. Therefore, a full range of electronic support services and a variety of self-help
resources are available on the SAS Customer Support site (http://support.sas.com/).
From this website, customers can do the following:
• Contact SAS Technical Support for help at https://support.sas.com/en/technical-
support/contact-sas.html.
• Search the knowledge base at https://support.sas.com/en/knowledge-base.html
which contains SAS Notes, sample programs and user documentation.
• View SAS Viya administrator resources at https://support.sas.com/en/
documentation/install-center/viya/administration.html and SAS 9.4 and earlier
administrator resources at https://support.sas.com/en/documentation/install-
center/94/intelligence-platform.html. Administrator resources include downloads,
hot xes, maintenance updates, security bulletins and system requirements.
• Suggest ideas for software enhancements at https://communities.sas.com/t5/
SASware-Ballot-Ideas/idb-p/sas_ideas.
• Obtain documentation about technical support services at https://support.sas.com/
en/technical-support.html, including information about support levels (Standard and
Premium) and support services and policies.
41
^
CONTENTS
QUALITY IMPERATIVE
• Interact with other SAS customers through our SAS Support Communities at
https://communities.sas.com/.
• Find SAS Training information at http://support.sas.com/training/, including
classroom training (on-site, web and mentor training), free tutorials, certication
programs, e-learning and the SAS Learning Subscription.
All these details and more, including the most up-to-date services and content, can
be accessed on the SAS Customer Support website at https://support.sas.com/en/
support-home.html. For the most current, detailed information about SAS Technical
Support, see SAS Technical Support Policies, available at https://support.sas.com/en/
technical-support/services-policies.html.
Software Fixes in the Field
Hot Fixes and Maintenance Releases
Overview
SAS releases regular updates to software products in the form of hot xes, maintenance
releases and product releases. Hot xes are SAS’ timely response to customer-reported
problems. They are also a way to deliver occasional security-related updates that can
affect any software product.
SAS Technical Support acts as the central point of contact for customer-reported
problems regarding production products. When a customer reports a problem that
requires a hot x, Technical Support enters the information about the issue into the
problem reporting system, noting that a hot x is requested. R&D and Technical Support
work together to review the requested hot xes and determine which xes will be made.
Technical Support then authorizes the x, and it enters the hot x process. Once R&D
provides the x, it is tested to ensure that the issues are resolved. In addition, regression
testing that is appropriate to the scope of the x is performed. If the problem solution is
surfaced on the web and the x includes UI changes, the R&D groups also evaluate the
need to rerun vulnerability testing.
Fixes are cumulative. If a new x requires a change to content that is included in an
existing x, then the existing x either is replaced or is updated with a newer x that
contains the original x plus any new xes.
While SAS strives to provide xes for all serious problems, there might be cases where
it is impractical or impossible to generate a x. For example, a x might not be possible
because of compatibility issues or because of the potential for the introduction of
unwanted side effects.
SAS
®
Viya
®
Platform
Releases on the SAS Viya platform are delivered in short, continuous-delivery cycles
called cadences. New Stable cadence releases are delivered once a month. Long-Term
Support (LTS) cadence releases are delivered once every six months.
SAS recommends that customers update to the current cadence releases as they
become available in order to keep software up to date with the latest functionality
and software xes.
42
^
CONTENTS
QUALITY IMPERATIVE
The list of available versions for the deployment assets from your software order are
available at My SAS at my.sas.com.
Patch updates can be released as soon as they are ready rather than waiting for
the next Stable or LTS scheduled release and can include critical xes and
security updates.
To learn more about cadence releases, visit the SAS Technical Support Policies page
at https://support.sas.com/en/technical-support/services-policies/sas-viya-
platform.html.
For more information about updating SAS Viya software, see SAS Viya Platform
Operations: Updating Software at https://go.documentation.sas.com/doc/en/
itopscdc/default/k8sag/titlepage.htm?fromDefault=.
SAS Viya Programming 3x
For SAS Viya, software updates replace some or all the existing deployed software
with the latest releases of that software. Updates are performed with the same
commands that are used to install SAS Viya, using the same software order and the
same playbook. More details about updating SAS Viya software are available in the
"Managing Your Software" section of the SAS Viya deployment guide for specic SAS
Viya releases and operating systems at https://support.sas.com/en/documentation/
install-center/viya/deployment-guides.html. Updates are posted on the Technical
Support Hot Fixes web page at https://tshf.sas.com/techsup/download/hotx/
hotx.html.
SAS
®
9.4
For SAS 9.4, available hot xes are automatically included when downloading an
order from the SAS Software Depot. If there is a signicant time lapse between
order download and deployment, additional hot xes may have become available. All
customers have access to the download pages and can download individual xes from
the SAS Technical Support Hot Fixes web page, available at https://tshf.sas.com/
techsup/download/hotx/hotx.html. For more hot x information, see the
"Applying Hot Fixes" section of the SAS 9.4 Guide to Software Updates and Product
Changes at https://go.documentation.sas.com/?cdcId=pgmsascdc&cdcVersion=9.4_
3.5&docsetId=whatsdiff&docsetTarget=p1vm9m5b9znanwn1vsr3bi4f1iog.
htm&locale=en.
In addition, SAS uses maintenance releases to update and support SAS 9.4.
Maintenance releases include xes and enhancements, documentation updates and
localizations. Maintenance details are posted to the SAS Customer Support site at
https://support.sas.com when they are available. Information about the current
maintenance release is available on the Maintenance Release Announcement web
page at https://support.sas.com/software/maintenance/.
43
^
CONTENTS
QUALITY IMPERATIVE
Alerting Customers
To stay informed about new hot xes and to receive notications when they are available,
visit the Communities: SAS Hot Fix Announcements page at http://communities.sas.com/
t5/SAS-Hot-Fix-Announcements/bg-p/hf.
To sign up for SAS Technical Support News and to request that operational
announcements be delivered via email, visit SAS Technical Support at https://support.
sas.com/en/technical-support.html and look for the following sign-up area:
SAS Technical Support News and operational announcements can be obtained by:
• Logging on to a SAS prole.
• Clicking Edit Prole: https://www.sas.com/prole/ui/auth.html#/edit.
• In the Subscriptions section, clicking Technical Support Updates – News and
Operational Announcements.
SAS documents Alert Priority issues, as well as problems that are not alert status, in the
form of SAS Notes. Customers can search for Alert Priority issues in the Samples & SAS
Notes database at https://support.sas.com/en/search.html?q=*%3A*&fq=siteArea%3A%
22Samples%20%26%20SAS%20Notes%22.
SAS lists product change notications on the Product Advisory Notices from the SAS
website at http://support.sas.com/techsup/pcn/index.html.
Migrating to New Releases
For information about moving from one release of SAS to another or when changing
operating systems, see the guidance provided on the following websites:
• Migration: http://support.sas.com/rnd/migration/
• Migration validation: http://support.sas.com/rnd/migration/planning/validation/
index.html
The MIGRATE procedure, which has validation macros associated with it, can assist with
migrating data and catalogs. Migrating code is more involved. Therefore, SAS recommends
that customers develop a migration plan that encompasses a sampling strategy based on
the total number of programs that need to be migrated and on the acceptance level. The
ANSI/ASQC Z1.4, ISO 2859 standard (Military Standard 105E) can help customers
determine the appropriate sample size.
44
^
CONTENTS
QUALITY IMPERATIVE
Operations
SAS provides technical support in accordance with the Technical Support policies.
However, if a customer chooses not to install the most current release of the software,
then the level of support that is available diminishes over time.
For more information about support, hot xes and source code archiving, see the following
resources:
• For documentation on support services and policies for current and prior releases
of software, see the SAS Technical Support Policies web page at https://support.sas.
com/en/technical-support/services-policies.html.
• For more on source code archiving, see the Business Continuity Management white
paper at http://www.sas.com/content/dam/SAS/en_us/doc/other1/continuity-of-
business.pdf.
Product Retirement
The R&D division directors, Marketing and Technical Support management work together
to determine when a product or host should be retired. Any remaining customers are
notied of product status. To learn more about product support levels, see the following
resources:
• The Support Levels by Product page: https://support.sas.com/en/technical-support/
support-levels.html.
• For the SAS Viya platform, complete the steps in the “View the Update Checker
Report” section in the SAS Viya Operations: Software guide at https://
go.documentation.sas.com/?cdcId=itopscdc&cdcVersion=v_001LTS&docsetId=k8s
ag&docsetTarget=p1it185kd37v25n1aoybu799tpk4.htm&locale=en.
• The table in the section “Support Levels for SAS 9.4 & Earlier Releases by Release
Number“ at https://support.sas.com/en/technical-support/services-policies/
sas-94-earlier.html.
45
^
CONTENTS
QUALITY IMPERATIVE
Quality in Customer Service
Quality in SAS
®
Communities
Helping SAS customers connect with each other facilitates knowledge and information
sharing, so SAS provides the following communication avenues for connecting with the
broader user community:
• SAS Communities: Collaborate with SAS and other SAS users about programming,
data analysis, and deployment issues, tips and successes at https://communities.sas.
com/t5/community/communitypage?nobounce.
• SAS Users Groups: Network, teach and collaborate with other SAS users. SAS users’
groups are independent, volunteer organizations run by SAS users. SAS partners with
these groups and provides a wide range of services. See https://www.sas.com/en_us/
connect/user-groups.html.
• SAS Social Media Portal: Stay connected with SAS and other SAS users through our
social channels, including Knowledge Exchanges, communities.sas.com, Facebook,
YouTube and X (formerly Twitter): http://www.sas.com/social/
• SAS Explore: An annual conference planned and hosted by SAS for developers,
data scientists and analysts, and IT professionals featuring demos, hands-on
workshops, tutorials, speakers and networking opportunities. Learn more at
https://explore.sas.com.
Quality in SAS Education
For SAS Education, delivering high-quality training support for SAS software technology
and solutions is not limited to the classroom. SAS Education consists of several teams,
all dedicated to providing the best customer service possible. From expert instructors
who help design award-winning courses to a customer service group who makes sure
that all calls are answered by a real person, we are condent that each customer is
getting the quality training that will help them make better, fact-based decisions specic
to their business – small or large.
For more information about SAS Education, see Appendix 5: Quality in SAS Education.
Quality in Customer Documentation
Documenting SAS software is much like developing the software itself. SAS
Documentation Division staff researches new features, plans the library that is needed
to document these features, develops and continuously updates the documentation,
converts it to the necessary formats, performs extensive testing, and distributes the
nal documents.
Currently, SAS Documentation produces the following types of documentation:
• Reference and usage documentation, administration guides and migration guides
on the web.
• Online help that is built into the software.
For more information about SAS’ documentation processes, see Appendix 6: Quality in
SAS Documentation.
46
^
CONTENTS
QUALITY IMPERATIVE
Quality in Consulting
SAS provides consulting services that enable organizations to reap the maximum
benets from their investments in technology as rapidly as possible. The SAS
Professional Services and Delivery Division offers the experience of domain and industry
thought leaders in the world of business intelligence and predictive analytics, armed with
SAS’ commitment and heritage of solving the most complex business challenges facing
the industry today.
SAS has been partnering with our customers to solve their business problems for
more than three decades. Our consultants take the time to listen and learn about our
customers’ business challenges and enterprise goals to establish a foundation for
strategic advancement toward those goals. This mutual collaboration enables us to
deliver the right SAS technology and customized services to solve our customers’
unique business requirements. We have amassed in-depth industry knowledge and
domain expertise while drawing upon industry and technology best practices and
proven methodologies.
47
^
CONTENTS
QUALITY IMPERATIVE
Appendix 1: Regulated Industry Issues
Introduction
Many large customers rely on SAS to enable their compliance with regulatory
requirements. SAS Legal Services, in conjunction with teams providing solutions for
regulated industries, continuously monitor regulatory affairs worldwide. This appendix
answers commonly asked questions from regulated customers, including those from the
life sciences and other industries.
Note that the United States Food and Drug Administration (FDA) does not certify
software tool vendors. We consider SAS a tool: Our customers must validate systems
that they build with SAS, but they do not need to validate SAS software. SAS is
developed using a controlled process that consists of distinct development phases.
Quality control activities are performed during various phases to make sure that quality
is built into the software. SAS understands FDA requirements for computerized system
validation and can identify existing practices and procedures that conform to FDA
expectations. SAS also understands the FDA-regulated industry’s motivation to assess
technology providers like SAS. Some validation methods for SAS procedures (PROCs)
that are used extensively by life sciences companies, as well as other SAS components
such as actions or other routines, are covered in the section Validating an Analytical
Component. The methods described might be useful in designing test cases to validate
programs or applications that are built using SAS components. Companies must develop
their own validation process for any tools that they use. For further details, see the SAS
Software Development Life Cycle section in this paper, as well as stories from
customers in the life sciences industry on the Customer Success (sas.com/customers).
Customers have inquired whether source code is available for an FDA audit if required
for regulatory compliance needs. SAS would allow the FDA to examine relevant portions
of the source code on a secure machine at SAS headquarters pursuant to appropriate
condentiality agreements.
ISO 9001 Quality Management Systems Certication
The SAS entities with ISO 9001 Quality Management Systems certication as of
this document’s publication date are listed below. The list below has the potential
to be incomplete, as new entities may have achieved certication since this paper’s
publication. To obtain a copy of a specic certicate received by SAS, send email to
[email protected].
• SAS UK (SAS Software Ltd.)
• SAS Institute Australia Pty Ltd.
• SAS Italy (SAS Institute SRL)
• SAS Poland (SAS Institute Sp. zo.o)
• SAS Spain (SAS Institute, S.A.U.)
48
^
CONTENTS
QUALITY IMPERATIVE
ISO 27001 Information Security Management Systems
Certication
The SAS entities with ISO 27001 Information Security Management Systems certication
as of this document’s publication date are listed below. The list below has the potential
to be incomplete, as new entities may have achieved certication since this paper’s
publication. To obtain a copy of a specic certicate received by SAS, send email to
[email protected].
• SAS UK and Ireland (SAS Software Ltd.)
• SAS Event Stream Processing R&D
• SAS Italy (SAS Institute SRL)
• SAS Spain and Portugal (SAS Institute, S.A.U.)
• SAS Finland (SAS Institute Oy)
ISO 14001 Environment Management Systems Certication
The SAS entities with ISO 14001 Environment Management Systems certication as
of this document’s publication date are listed below. The list below has the potential
to be incomplete, as new entities may have achieved certication since this paper’s
publication. To obtain a copy of a specic certicate received by SAS, send email to
[email protected].
• SAS Spain (SAS Institute, S.A.U.)
Complying with Title 21 CFR Part 11
The United States regulation known as Title 21 Code of Federal Regulations (CFR)
Part 11 (https://www.ecfr.gov/current/title-21/chapter-I/subchapter-A/part-11),
or the “Electronic Records; Electronic Signatures” rule, provides information about what
constitutes trustworthy and reliable electronic records and electronic signatures. Many
of our customers who are regulated by the FDA are required to comply with this rule,
which sets forth the criteria under which the FDA considers electronic records and
electronic signatures to be trustworthy, reliable and generally equivalent to paper
records. CFR Title 21 serves as the predicate rule and has been in force since its inception
in 1997. Although the requirements of CFR Title 21 were originally written for the paper
record, CFR Title 21 now explicitly applies to electronic records and signatures as well.
Part 11 does not outline details such as whether a record or signature is required, who
signs it and so on, because this is determined by the underlying predicate rules. Predicate
rules are the rules that are set forth by the Federal Food, Drug and Cosmetic Act, Public
Health Service Act, and FDA regulations. Part 11 governs the treatment of these records
and signatures that fall under predicate rules when they are created and maintained
electronically.
The FDA has issued industry guidance for the use of electronic health record data in
clinical investigations (https://www.fda.gov/media/97567/download). In issuing such
guidance, the FDA sought to assist sponsors, clinical investigators, and other interested
parties in using electronic health records (EHRs) in clinical trials. This guidance claries
recommendations on applying Part 11 electronic records regulations to electronic data
capture (EDC) systems. Among other things, the FDA provides guidance on the use of
interoperable or fully integrated EHR and EDC systems, appropriate validation methods,
recordkeeping requirements and the use of certied and noncertied EHR technology.
49
^
CONTENTS
QUALITY IMPERATIVE
However, we recognize that this guidance provides non-binding recommendations and
that certain specications may change. Therefore, we continue to monitor FDA
regulations and guidelines that pertain to SAS or to customers using SAS software.
SAS technologies provide the capability to use SAS and implement SAS solutions in
a way that is compliant with 21 CFR Part 11. We provide tools to help customers build
a Part 11 compliant application. Compliance with this regulation ultimately depends on
how your application or the SAS solution is installed and used, how users are trained,
and other factors. Customers need to use SAS according to the system requirements,
install it according to the installation instructions, and use the DATA step and each
procedure or solution according to the user documentation.
Although SAS includes features that enable users to comply with 21 CFR Part 11,
simply using SAS software or any of SAS’ solutions will not automatically make a user
compliant. All elements must be present in a proper environment to be 21 CFR Part 11
compliant, including adherence to compliant standard operating procedures. Users
should refer to the predicate rule or consult the FDA or its guidance documents to
determine whether their system is in compliance with regulatory expectations.
SAS customers can use SAS products to build data collection, analysis and other
systems that can be used in compliance with Part 11. They can also use programming
languages such as the Java Programming Language, C# and Visual Basic. We enable
these clients to access SAS using the Integration Technologies API. Developers of such
systems would need to determine which features are needed for the system that they
are designing and then build the appropriate checks into the system. Such features could
include audit trails, security checks and electronic signatures.
Regarding audit trails and integrity constraints, the audit trail feature of Base SAS has
the essential elements to address and enable the controls and procedures for a 21 CFR
Part 11 audit trail.
The FDA accepts a SAS transport format as a method for accepting and archiving data
sets. The SAS transport format is an open format, has a free viewer, is used extensively
in the life sciences industry and has long-term support. Other software vendors can write
transport format using the specications described on the FDA and SAS Technology
web page.
The FDA now requires all new CDER and CBER study submissions to use industry
standard data structures (http://www.fda.gov/ForIndustry/DataStandards/default.
htm). The FDA requires the CDISC Study Data Tabulation Model (STDM), Standard for
Exchange of Nonclinical Data (SEND) and Analysis Data Model (ADaM) for exchanging
electronic data and report-ready tables. See the CDISC section for more information
about how SAS supports CDISC.
SAS addresses revision control with SAS tools, applications, procedures and custom
application interfaces (APIs or engines). SAS also interfaces well with other revision
control software or ling systems such as Documentum. Custom engines for interfacing
with clinical data management and electronic data capture systems (Medidata Rave is
one example) have been developed. SAS/ACCESS can also be used to obtain repetitive
versions of data from a laboratory information management system (LIMS) or clinical
data management system (CDMS). The COMPARE and CONTENTS procedures can be
50
^
CONTENTS
QUALITY IMPERATIVE
used to monitor changes or revisions regarding content in data. Functionalities such
as data integrity constraints and audit trails can be enabled to assist in this process.
All this functionality is supported by the SAS Life Science Analytics Framework, which
provides a real-time assessment of metadata structure and revisions, or through data
management solutions such as SAS Data Preparation.
The SAS Life Science Analytics Framework is a 21 CFR Part 11 enabling technology. (See
sas.com/en_us/software/life-science-analytics-framework.html.) SAS Life Science
Analytics Framework software was designed and introduced to specically address the
issues associated with 21 CFR Part 11 and the FDA’s Guidance for Industry. The software
provides these capabilities while offering an enhanced operating environment for
managing clinical data, programs, logs, documents and reports. Careful consideration
was given to the intended use with respect to data warehousing, analysis and reporting,
electronic submissions, and related e-signature requirements. Application of both
process and quality management has enabled the software to meet the intended
requirements of the system’s 21 CFR Part 11 functionality.
The Health and Life Sciences (HLS) R&D organization follows the SAS software
development process. To meet the needs of their FDA-regulated customers, they have
implemented an additional quality management system (QMS) to govern their software
development. HLS R&D might use additional tools that are not generally used by SAS
R&D but that are validated for use through the HLS R&D QMS.
CDISC
SAS has been an active supporter and platinum member of the Clinical Data Interchange
Standards Consortium (CDISC) since 2000 with both resource and administrative
support. For details, see cdisc.org. SAS views the FDA’s adoption and requirement
of the Study Data Tabulation Model (SDTM), Standard for Exchange of Nonclinical Data
(SEND), Analysis Data Model (ADaM), Dene-XML and other CDISC data standards for
the electronic Common Technical Document as very signicant events. We recognize the
value that data standards give the industry in providing the key elements for improving
global public health. Implementing and applying the CDISC standard in commonly used
pharmaceutical industry software makes it possible for both product sponsors and
regulatory authorities to benet from the value of standard data structure and elements.
SAS provides standard processes within its production software to facilitate using
SDTM, SEND and ADaM data models, Dene-XML, Dataset-XML, Operational Data
Modeling and laboratory data. See the SAS statement on CDISC support at sas.com/en_
us/industry/life-sciences/sas-cdisc.html for more information.
HIPAA and HITECH
The health care reforms made by Title II of the Health Insurance Portability and
Accountability Act of 1996 (HIPAA) and the Health Information Technology for Economic
and Clinical Health (HITECH) Act of 2009 provide federal protections for the privacy and
security of individually identiable health information. The United States Department of
Health and Human Services has issued regulations governing HIPAA/HITECH that
require health care organizations and other covered entities, as well as their business
associates, to meet certain minimum standards of privacy and security with respect to
health care data and databases. These regulations also direct how such data and
databases are to be stored, viewed, accessed and shared. SAS software includes security
51
^
CONTENTS
QUALITY IMPERATIVE
and other built-in features that customers can use to implement HIPAA/HITECH-
compliant applications, though each customer must assess its specic needs in the
context of its own computing environment. See SAS Software Security Framework:
Engineering Secure Products (sas.com/content/dam/SAS/en_us/doc/whitepaper1/
sas-software-security-framework-107607.pdf) for an overview. SAS is available to
assist with HIPAA/HITECH compliance issues related to the use of SAS technologies
and solutions.
Sarbanes-Oxley Compliance
Satisfying the various requirements of the Sarbanes-Oxley Act (SOX) generally requires
management of data, processes and technologies to ensure appropriate internal controls
associated with nancial risk. Compliance with SOX often involves a review of multiple
systems and application of software tools and technologies that, among other things,
must address conguration and change management, business process management,
and documents and records management. SAS software can help customers achieve
SOX compliance, though each customer must assess its specic needs in the context
of its own computing environment.
US Government Conguration Baseline
As a vendor of desktop software products to the United States federal government, SAS
validates, through our release management process, that our desktop software products
on the Microsoft Windows platform comply with the US Government Conguration
Baseline, formerly known as the Federal Desktop Core Conguration (FDCC). R&D
validates the software and archives the validation reports as a part of due diligence
before releasing the software.
Service Organization Controls Certication
SAS engages an independent third party to perform an annual Service Organization
Controls (SOC) 2 Type II audit certication for SAS Cloud hosting management, which
includes a SOC 3 general-use report. The SAS SOC 2 Type II and SOC 3 reports pertain
to the security, availability and condentiality trust principles and include controls
related to the in-scope data centers’ physical security and environmental safeguards,
logical access, change management, monitoring, risk assessment and management,
communication and information, systems operations, and availability over network
devices as described in the report for the dened effective period. Learn more by
visiting the SAS Trust Center (sas.com/en_us/trust-center/sas-trust-compliance.html).
52
^
CONTENTS
QUALITY IMPERATIVE
Appendix 2:
Validating an Analytical Component
Numerical Accuracy
We use a variety of methods to verify the accuracy and precision of the results generated
by our software. Libraries of regression tests using automated tools are run periodically
throughout the testing cycle to test functionality and data integrity. These reusable
libraries of tests cover syntax, options, functionality, valid and invalid data, errors, stress,
and results for the procedure/function/solution. These tests are run, and differences are
resolved, before the release is declared ready for production.
Validating an Analytical Component
Our development staff has the education, training and experience to perform their
assigned tasks. They use a variety of methods to verify, to the extent possible, that the
software produces accurate, reliable and numerically precise results. A combination of
methods is used to validate an analytical product and the algorithms in it. These
methods are listed below.
Writing Independent Validation Code
Except when similar results are available within existing and previously tested SAS
analytical software, independent verication of numerical results via SAS/IML
®
, DATA
step code or hand calculations is performed whenever possible and practical. SAS/IML
is a powerful matrix programming language that is used to recreate the numerical output,
or pieces of the output, produced by the SAS software being validated. Sometimes the
same algorithm used by the procedure, action or other component being checked is
coded into SAS/IML, but occasionally a different algorithm is used if the two algorithms
are expected to produce the same results. Replication of results via SAS/IML or DATA
step code is the most reliable method for validation, because it is an independent and
veried conrmation of the numerical output.
Comparing to Similar Results in Other Algorithms in SAS
®
Many SAS components produce equivalent output, such as parameter estimates,
covariances, or solutions to modeling problems. Similar output produced by new
components is validated by comparing it to these previously validated results.
Similarly, when a macro program exists that produces some of the same results
as the new component, it can be used for validation.
Running Simulation Studies
Simulation studies may be performed in instances where closed-form solutions do not
exist or used as checks when computations are extremely time- and memory-intensive.
Verifying Against Published Results
Comparison against other software vendors’ applications is sometimes made, especially
in those cases where the vendor’s application is highly regarded as producing quality
results. Comparison is also sometimes made versus results published in books or journal
articles. Note that matching another vendor’s results or just verifying a match versus a
published result is not usually considered sufcient validation.
53
^
CONTENTS
QUALITY IMPERATIVE
Comparing Against Open-Source Software
Numerical validation against open source software is sometimes performed to ensure
that the results are as accurate or more accurate than comparable open source software
and that the performance is at least comparable, if not better. Our numerical validation
test suites include a representative set of test scenarios running comparable algorithms
for accuracy comparisons. Numerical results are compared for accuracy when the true or
optimal model or solution is known.
Validation of SAS
®
Components That Consume Open-Source Technology 
SAS software in some cases uses open source technology in its components. To make
sure that the components that use these open source routines work seamlessly, SAS
aims to recreate the same results across a variety of scenarios as those derived from
directly applying an equivalent wrapper directly to the open source technology.
Validating by simultaneously using replication and comparison ensures that SAS
components are using open source technology effectively.
Completing Consistency Checks Within the Component
Certain consistency checks are performed to help validate results. Here are some
examples:
• Checking that results with a WEIGHT variable that has all weight values equal
to 1 are identical to results obtained without using the WEIGHT variable.
• Verifying that results with a FREQ variable match results when not using the FREQ
variable but instead repeating each observation by the value of the FREQ variable.
• Verifying that results with a BY statement match those obtained for each value of
the BY variable analyzed individually.
An example of our validation techniques for the REG procedure is included below.
Example of Validation Techniques
While many analyses and results require complex code to validate, the following simple
example is used to provide a basic illustration of some of the validation techniques that
are performed at SAS. The test case veries the results from the REG procedure by
comparing them to a classic textbook analysis. The data comes from Neter, Wasserman
and Kutner (1990), and the test case veries the ANOVA table, the Fit Statistics and the
Parameter Estimates table that PROC REG produces.
The data consists of sales information from 15 marketing districts, and PROC REG ts
a multiple regression model. Besides comparing the results to the textbook results,
this example also illustrates validation using the IML procedure and cross-validation
using the GLM procedure. Finally, the example concludes by illustrating some basic
consistency checks.
54
^
CONTENTS
QUALITY IMPERATIVE
data Zarthan_Company;
input sales target_population discretionary_income @@;
datalines;
162 274 2450 120 180 3254 223 375 3802 131 205 2838
67 86 2347 169 265 3782 81 98 3008 192 330 2450
116 195 2137 55 53 2560 252 430 4020 232 372 4427
144 236 2660 103 157 2088 212 370 2605
;
ods listing close;
ods rtf le=’Zarthan.rtf’;
ods select ANOVA FitStatistics ParameterEstimates;
proc reg data=Zarthan_Company;
ods output ANOVA=reg_ANOVA
FitStatistics=reg_FitStatistics
ParameterEstimates=reg_ParameterEstimates;
model sales = target_population discretionary_income;
run;
ods rtf close;
PROC REG Results
Analysis of Variance
Source DF
Sum of
Squares
Mean Square F Value Pr > F
Model 2 53845 26922 5679.47 <.0001
Error 12 56.88357 4.74030
Corrected
Total
14 53902
Root MSE 2.17722 R-Square 0.9989
Dependent Mean 150.60000 Adj R-Sq 0.9988
Coeff Var 1.44570
Parameter Estimates
Variable DF
Parameter
Estimate
Standard Error t Value Pr > |t|
Intercept 1 3.45261 2.43065 1.42 0.1809
target_population 1 0.49600 0.00605 81.92 <.0001
discretionary_
income
1 0.00920 0.00096811 9.50 <.0001
55
^
CONTENTS
QUALITY IMPERATIVE
Comparison to Textbook
The results from PROC REG can be compared to those given in the textbook to verify
that there are no discrepancies. For this example, the following quantities are reported
in the textbook:
ANOVA Results
Statistic Result Page
SS Model 53,844.716 256
SS Error 56.884 256
SS Total 53,901.600 256
df Model 2 256
df Error 12 256
df Total 14 256
MS Model 26,922.358 256
MS Error 4.740 256
F* 5,680 257
p-value < .001 257
* Note that PROC REG reports as 5,679.47. The discrepancy is due to the textbook
example rounding the quantities involved in the ratio before the ratio is computed.
Fit Statistics
Statistic Result Page
R-Squared 0.9989 257
Parameter Estimates
Statistic Result Page
Beta: Intercept 3.4526127900 252
Beta: target population 0.4960049761 252
Beta: disc. Income 0.009199080867 252
Std. Error: target pop. 0.006054 258
Std. Error: disc. Income 0.0009681 257
56
^
CONTENTS
QUALITY IMPERATIVE
Direct Validation
SAS/IML can be used to compute the corresponding quantities from the PROC REG output:
proc iml;
use Zarthan_Company;
read all var _all_ into data;
y=data[,1]; * dependent variable;
n=nrow(y); * sample size;
x=j(n,1,1)||data[,2:3]; * x matrix, augmented for intercept;
p=ncol(x); * number of parameters;
beta=inv(x`*x)*x`*y; * parameter estimates;
yhat=x*beta; * predicted values;
resid=y-yhat; * residuals;
sse=ssq(resid); * Sum of Squares for Error;
dfe=nrow(x)-ncol(x); * error degrees of freedom;
mse=sse/dfe; * Mean Square Error;
cssy=ssq(y-y[+]/n); * Corrected Total Sum of Squares;
rsquare=(cssy-sse)/cssy; * R-Square;
stdbeta=sqrt(vecdiag(inv(x`*x))*mse);* Std error of estimates;
t=beta/stdbeta; * parameter t-tests;
df=j(nrow(t),1,1); * parameter degrees of freedom;
t_prob=1-cdf(‘F’,t##2,df,dfe); * p-values for t-tests;
dft=n-1; * corrected total df;
dfm=dft-dfe; * Model degrees of freedom;
ssm=cssy-sse; * Sum of Squares for Model;
msm=ssm/dfm; * Mean Square Model;
F=msm/mse; * F statistic;
F_prob=1-cdf(‘F’,F,dfm,dfe); * p-value for F statistic;
root_mse=sqrt(mse); * Root MSE;
mean_y=y[+]/n; * Dependent Mean;
coe_var=(root_mse/mean_y)*100; * Coeicient of Variation;
adj_r=1-((n-1)#(1-rsquare))/(n-p); * Adjusted R-Square;
* create matrices of the corresponding REG tables;
anova_table=(dfm//dfe//dft)||(ssm//sse//cssy)||(msm//mse//{._})
||(F//{._ ._}`)||(F_prob//{._ ._}`);
t_statistics=(root_mse//mean_y//coe_var)||(rsquare//adj_r//{0});
parameter_estimates=df||beta||stdbeta||t||t_prob;
* create data sets of these matrices to be used with the COMPARE procedure;
create iml_anova(label=’Analysis of Variance’ )
from anova_table[colname={df ss ms fvalue probf}];
append from anova_table;
create iml_tstatistics(label=’Fit Statistics’ )
from t_statistics[colname={nvalue1 nvalue2}];
append from t_statistics;
create iml_parameterestimates(label=’Parameter Estimates’ )
from parameter_estimates[colname={df estimate stderr tvalue
probt}];
append from parameter_estimates;
quit;
57
^
CONTENTS
QUALITY IMPERATIVE
* print the SAS/IML validation results for a visual scan;
proc print data=iml_anova noobs; run;
proc print data=iml_tstatistics noobs; run;
proc print data=iml_parameterestimates noobs; run;
SAS/IML Validation Results
ANOVA Table
DF SS MS FVALUE PROBF
2 53844.72 26922.36 5679.47 0
12 56.88 4.74 _ _
14 53901.60 _ _ _
Fit Statistics Table
NVALUE1 NVALUE2
2.177 0.99894
150.600 0.99877
1.446 0.00000
Parameter Estimates Table
DF ESTIMATE STDERR TVALUE PROBT
1 3.45261 2.43065 1.4204 0.18094
1 0.49600 0.00605 81.9242 0.00000
1 0.00920 0.00097 9.5021 0.00000
 
* compare via PROC COMPARE the PROC REG results to the SAS/IML validation results
proc compare data=reg_anova compare=iml_anova
error briefsummary note method=relative(1) criterion=1e-6;
attrib _all_ format = label = ‘’;
var df ss ms fvalue probf;
run;
proc compare data=reg_tstatistics compare=iml_tstatistics
error briefsummary note method=relative(1) criterion=1e-6;
attrib _all_ format = label = ‘’;
var nvalue1 nvalue2;
run;
proc compare data=reg_parameterestimates compare=iml_parameterestimates
error briefsummary note method=relative(1) criterion=1e-6;
attrib _all_ format = label = ‘’;
var df estimate stderr tvalue probt;
run;
58
^
CONTENTS
QUALITY IMPERATIVE
PROC COMPARE Results
The COMPARE Procedure
Comparison of WORK.REG_ANOVA with WORK.IML_ANOVA
(Method=RELATIVE(1), Criterion=0.000001)
NOTE: All values compared are within the equality criterion used.
NOTE: One or both of the data sets WORK.REG_ANOVA and WORK.IML_ANOVA contain
variables not in the other. However, all comparisons are equal for the
variables specied.
The COMPARE Procedure
Comparison of WORK.REG_FITSTATISTICS with WORK.IML_FITSTATISTICS
(Method=RELATIVE(1), Criterion=0.000001)
NOTE: All values compared are within the equality criterion used.
NOTE: One or both of the data sets WORK.REG_FITSTATISTICS and
WORK.IML_FITSTATISTICS contain variables not in the other. However, all
comparisons are equal for the variables specied.
The COMPARE Procedure
Comparison of WORK.REG_PARAMETERESTIMATES with WORK.IML_PARAMETERESTIMATES
(Method=RELATIVE(1), Criterion=0.000001)
NOTE: All values compared are within the equality criterion used.
NOTE: One or both of the data sets WORK.REG_PARAMETERESTIMATES and
WORK.IML_PARAMETERESTIMATES contain variables not in the other.
However, all comparisons are equal for the variables specied.
 
Comparison to Other SAS
®
Procedures
The PROC REG results can be compared to PROC GLM output:
* run the corresponding model with PROC GLM;
proc glm data=Zarthan_Company;
ods output OverallANOVA=glm_ANOVA(label=’Analysis of Variance’)
FitStatistics=glm_FitStatistics
ParameterEstimates=
glm_ParameterEstimates(label=’Parameter Estimates’);
model sales = target_population discretionary_income/solution;
run;
59
^
CONTENTS
QUALITY IMPERATIVE
* compare the REG results to the GLM results;
proc compare data=reg_anova compare=glm_anova
error briefsummary note method=relative(1) criterion=1e-6;
attrib _all_ format = label = ‘’;
var df ss ms fvalue probf;
run;
proc compare data=reg_parameterestimates
compare=glm_parameterestimates
error briefsummary note method=relative(1) criterion=1e-6;
attrib _all_ format = label = ‘’;
var estimate stderr tvalue probt;
run;
* visually compare the statistics that correspond in Fit Statistics because the two tables have a different structure:
proc print data=reg_FitStatistics; run;
proc print data=glm_FitStatistics; run;
PROC REG Results
Obs Model Dependent Label1 cValue1 nValue1 Label2 cValue2 nValue2
1 MODEL1 sales Root MSE 2.17722 2.177222 R-Square 0.9989 0.998945
2 MODEL1 sales Dependent Mean 150.60000 150.600000 Adj R-Sq 0.9988 0.998769
3 MODEL1 sales Coeff Var 1.44570 1.445699 0
PROC GLM Results
Obs Dependent RSquare CV RootMSE DepMean
1 sales 0.998945 1.445699 2.177222 150.6000
 
Consistency Checking
A simple check of the WEIGHT statement can be performed. Note that complete testing for WEIGHT would include
tests where the weight values are not all equal to 1, with SAS/IML validation performed.
* add a weight variable to the data set, with values all equal to 1;
data check_weight; set Zarthan_Company;
weight=1;
run;
* run PROC REG with the weight variable;
proc reg data=check_weight;
ods output ANOVA=reg_ANOVA_weight
FitStatistics=reg_FitStatistics_weight
ParameterEstimates=reg_ParameterEstimates_weight;
model sales = target_population discretionary_income;
weight;
run;
60
^
CONTENTS
QUALITY IMPERATIVE
* compare the results from the REG run without a weight to the REG run with weights all equal to 1;
proc compare data=reg_anova compare=reg_anova_weight
error briefsummary note method=relative(1) criterion=1e-6;
attrib _all_ format = label = '';
run;
proc compare data=reg_tstatistics
compare=reg_tstatistics_weight
error briefsummary note method=relative(1) criterion=1e-6;
attrib _all_ format = label = '';
run;
proc compare data=reg_parameterestimates
compare=reg_parameterestimates_weight
error briefsummary note method=relative(1) criterion=1e-6;
attrib _all_ format = label = '';
run;
PROC COMPARE Results:
The COMPARE Procedure
Comparison of WORK.REG_ANOVA with WORK.REG_ANOVA_WEIGHT
(Method=RELATIVE(1), Criterion=0.000001)
NOTE: All values compared are within the equality criterion used.
NOTE: The data sets WORK.REG_ANOVA and WORK.REG_ANOVA_WEIGHT compare equal.
The COMPARE Procedure
Comparison of WORK.REG_FITSTATISTICS with WORK.REG_FITSTATISTICS_WEIGHT
(Method=RELATIVE(1), Criterion=0.000001)
NOTE: All values compared are within the equality criterion used.
NOTE: The data sets WORK.REG_FITSTATISTICS and WORK.REG_FITSTATISTICS_WEIGHT
compare equal.
The COMPARE Procedure
Comparison of WORK.REG_PARAMETERESTIMATES with
WORK.REG_PARAMETERESTIMATES_WEIGHT
(Method=RELATIVE(1), Criterion=0.000001)
NOTE: All values compared are within the equality criterion used.
NOTE: The data sets WORK.REG_PARAMETERESTIMATES and
WORK.REG_PARAMETERESTIMATES_WEIGHT compare equal.
61
^
CONTENTS
QUALITY IMPERATIVE
References
National Institute of Standards and Technology 2007. “NIST STRD Background
Information.“ Accessed November 6, 2015. See http://www.itl.nist.gov/div898/strd/
general/bkground.html.
Neter J., W. Wasserman, and M. H. Kutner. 1990. Applied Linear Statistical Models,
Third Edition. Boston, MA: Irwin.
62
^
CONTENTS
QUALITY IMPERATIVE
Appendix 3: Installation and Operational
Qualication for SAS
®
9.4 and SAS Viya
SAS 9.4 includes three qualication tools – the SAS Installation Qualication Tool (SAS
IQ), the SAS Operational Qualication Tool (SAS OQ) and the SAS Deployment Tester
– to help customers verify installation and test the operation of SAS at their sites. These
tools provide a consistent, repeatable process for verifying the initial installation and
future updates to SAS while also providing a framework for running customer-written
tests. Output of SAS IQ and SAS OQ includes a data le that contains the test results
and output formatting options. SAS Deployment Tester generates an online report of
the success of each test run.
Background
Originally, these tools were conceived as testing tools to assist with the internal testing
of the installation processes. However, the feature set has been broadened to allow
distribution to customers. The SAS Qualication Tools (SAS IQ and SAS OQ) that are
delivered to customers are the same tools that are used in the testing and qualication
of the SAS
®
9 installation software. SAS Deployment Tester was designed to test the
complex conguration of the SAS 9.4 Intelligence Platform, although any of the SAS OQ
tests or customer-written tests can be run using this tool.
Customer Considerations
SAS IQ assists regulated customers in demonstrating compliance and qualifying the
installation. SAS IQ can be used as part of the installation process, as an interim check
on the state of the SAS system, and as an automated tool to maintain an audit history.
SAS OQ also assists in demonstrating compliance and includes tests that are designed
to be included in operational qualication. SAS OQ can be used as part of the initial
qualication process and as an automated tool to maintain a history when changes are
made to the SAS installation. SAS Deployment Tester runs SAS OQ tests and tests for
the SAS 9.4 intelligence platform.
SAS customers need to install SAS on appropriate hardware and software according to
the installation instructions. When running SAS
®
9 or later, customers should run the SAS
IQ and SAS OQ tests after installation. Customers should then write procedures that
document how SAS is used at their site. Any systems they have built using SAS might
need to be validated with each new SAS release. Many companies do this by running
the programs at the earlier release, and then running the tests at the new release and
comparing results.
SAS Professional Services and Delivery can provide support to customers, including drug
sponsors and contract research organizations, on validation efforts. Consultants take steps
such as these:
• Gather user and functional requirements and prepare validation documentation
inclusive of validation plans, test protocols and test scripts.
• Install and congure SAS software according to the instructions and alert notes that
are delivered by SAS as part of the software shipment.
• Execute IQ and OQ according to approved plans and test scripts. Our services
personnel can also assist customers with Performance Qualication or User
Acceptance Testing if requested.
63
^
CONTENTS
QUALITY IMPERATIVE
• Provide knowledge transfer to IT staff and end users on the above, recommending
formal SAS training where needed.
• Provide project management for all the above activities.
Operation
The testing process for the installation of SAS 9.4 has three phases. The rst phase is the
initial installation process on the target platforms. The second phase is the execution of
SAS IQ. The third phase is the execution of SAS OQ or SAS Deployment Tester.
Phase 1
During this phase, SAS is installed at the customer site. Customers should follow
the installation instructions provided in their packages or see the online instructions
provided on the SAS Install Center (http://support.sas.com/documentation/
installcenter).
Phase 2
SAS IQ assists you in demonstrating that the SAS system has been installed and
maintained to the manufacturer’s specications. SAS IQ veries the integrity of each le
in SAS
®
9 and provides the customer a set of reports detailing the results. To execute SAS
IQ, follow the instructions in support.sas.com/documentation/installcenter/en/
ikinstqualtoolug/66614/PDF/default/qualication_tools_guide.pdf.
Phase 3
SAS OQ assists you in demonstrating that the SAS system is operational. SAS OQ
uses SAS programs provided by the component development groups and will execute,
process, and report the program results. To execute SAS OQ, follow the instructions in
support.sas.com/documentation/installcenter/en/ikinstqualtoolug/66614/PDF/
default/qualication_tools_guide.pdf.
SAS Deployment Tester is a diagnostic tool used for assessing a SAS 9.4 Intelligence
Platform deployment. After an installation or upgrade, you can use SAS Deployment
Tester to ensure that your SAS software and critical server components have been
installed and congured correctly. To learn more about the SAS Deployment Tester,
including prerequisites for use, how to add tests, and how to use the Deployment
Tester, see https://go.documentation.sas.com/doc/en/bicdc/9.4/bisag/
n1c1m0fm7gxs54n104kqx33wqgo7.htm.
Content
The content of the testing tools is generated within the R&D community. During the
weekly build process, a data le is created that contains the md5sum values for each
le that is used by SAS IQ for each target platform. These values are validated over the
course of weekly testing by several testing groups in R&D. The test cases and test tables
used by SAS OQ and SAS Deployment Tester are developed by the testing groups that
are responsible for the SAS
®
9 component. The tables and tests are executed and
validated on all target platforms during weekly testing by several testing groups in R&D.
64
^
CONTENTS
QUALITY IMPERATIVE
Verication of Test Results
Customers can create new tests that t their unique needs. A test can be constructed
in several ways. There are general approaches to determining the result of a test:
• Human verication
• Programmatic verication
• Self-verication
Human verication of a test is the least efcient of the approaches. This method requires
that a person visually inspect the results of a test including the SAS log, the SAS output,
and the return code from the SAS application. It is time-consuming and very repetitive
work to go over the same text les again and again. This can lead to incorrect results
interpretation after just a few iterations.
Programmatic verication means that a test program, such as a SAS program, cannot
reliably determine its own result status, so it needs additional programs that run after
it is completed to help determine the result.
Filtering and comparing with a benchmark is a classic and common example of
programmatic verication. When this method is used, the output le is ltered to remove
non-deterministic data, of which the current date and time are good examples. Then the
ltered version of the output le is compared against a benchmark le in a byte-by-byte
manner. If the ltered output le matches the benchmark, then the test is deemed to
have passed. Otherwise, the test has failed. Benchmarks are problematic in that they
require frequent maintenance, host-specic versions and ltering, which could result
in test failures that are not real. These failures require that resources be allocated to
analyze the differences and decide whether there is a problem with the program being
tested, whether a new benchmark needs to be created, or whether some addition or
change to the lter needs to be made. However, programmatic verication is very
reliable and requires no special programming skills other than ltering to create the
ltered output le.
The self-verication approach means writing a test program that can reliably determine
whether the feature being tested worked and then reporting that through a simple return
code. Self-verication avoids all the problems of benchmarking and produces very
reliable, durable, low-maintenance test ware, but it does require additional, up-front
investment and programming skills. Tests must be carefully written to make sure that
passing results are accurate.
SAS has put a lot of effort into writing as many as possible of the supplied SAS OQ tests
in a self-verifying format. We suggest that users follow our example and try to do the
same. Here are some suggestions, tips and best practices.
Every SAS DATA step, procedure, and global statement should set the value of at least
one of the automatic SAS macro variables from this list: SYSERR, SYSRC, SYSLIBRC,
SYSFILRC, SYSLCKRC and SYSINFO. Users should check these macro variables in their
test programs at every opportunity.
65
^
CONTENTS
QUALITY IMPERATIVE
PROC COMPARE can be used effectively to validate many procedures. Any procedure
that can produce a SAS data set as output can be reliably validated by directly
constructing a SAS data set with the variables and values that are expected to be
generated. Users can execute a simple DATA step with DATALINES input to accomplish
this or any number of straightforward DATA step techniques. Then they can use PROC
COMPARE to verify that the procedure-generated data set matches the one that
they’ve created by hand. The result can be quickly checked by looking at the SYSINFO
macro variable.
Other Notes
Customers often need to establish a baseline for their tests in a prior SAS release and
then verify the results in a new SAS release. The SAS OQ provides a clear migration path
for customers who are concerned about moving to a new release of SAS.
Both SAS IQ and SAS OQ are supported for the Windows and UNIX operating
environments in SAS
®
9. SAS IQ and SAS OQ can be used only with les that are provided
through the normal R&D delivery process for SAS
®
9 and later. For example, hot xes
applied by using the normal R&D install process are veried regardless of the delivery
mechanism. Files that use post-processing methods, such as ZIP or TAR archives, cannot
be veried.
66
^
CONTENTS
QUALITY IMPERATIVE
Appendix 4: Employee Training
Just as SAS actively cultivates continuous improvement in products and processes, SAS
employees cultivate continuous improvement in the skills and abilities that are critical to
developing a quality workforce. By helping employees meet their professional and
personal development goals, SAS empowers them to deliver the highest-quality
software and services to customers.
New Employees Development
New SAS employees attend an orientation session to learn about SAS history, culture,
compliance training, mandatory company policies, and employee programs and services.
Afterward, direct managers collaborate with their new employees to tailor a training plan
that balances job skill needs with the employee’s strengths. A customized program might
include live classroom courses, virtual learning, self-guided study or one-on-one training
sessions with a mentor. Managers may engage with subject matter experts, Human
Resources and SAS Education to identify the most effective learning options and
channels for each new employee.
Leadership and Management Development
The Leadership Development program increases organizational effectiveness at SAS
by providing all managers with leadership and management development opportunities
through the Manager Journey program. The program includes a core curriculum of six
to eight classes that individuals complete within their rst months managing at SAS.
The mission is to increase managers’ competence in leadership and interpersonal
communication skills, as well as their commitment to developing the overall
performance of their individual staff members and teams. Both the transfer of training
and the consistent use of new skills are fostered through action plans, skills coaching,
and varied follow-up opportunities to embed learning in day-to-day behaviors. In
addition, Leadership Development offers coaching and consultation in leadership,
conict management, team development, energy management, meeting planning
and facilitation.
Existing managers have the option to complete Manager Essentials Plus. The two-day
program refreshes managers on leadership essentials and provides information on their
roles, responsibilities and communication skills, especially for providing feedback and
coaching staff. Participants learn about time and energy management and the need to
care for their own well-being as they continue their management duties.
Additional programs are available for employees who are considered to have high
potential for a leadership role.
Career Development Planning
Although skills acquisition occurs primarily through experience, exposure to thought
leaders and relevant educational opportunities are essential to professional
development. SAS’ Global Career Mentoring Program fosters employee connections
within the SAS global workforce. The program’s mission is to accelerate talent
development and expand functional expertise and innovation across the company.
Additionally, the SAS Career Circle program enables employees to participate in group
discussion facilitated by peer leaders who are uniquely prepared to help employees
reect on and explore career options.
67
^
CONTENTS
QUALITY IMPERATIVE
Technical Training
Technical Skills + Role-Based Skills + Transferrable Skills + Certications =
Competence + Credibility + Condence
Since technology skills are crucial to the success of SAS, a wide range of internal and
external technical training is offered to employees. Though this training, employees
may become accredited in externally recognized certications relevant to their roles,
such as Amazon Web Services, Microsoft Azure, Google Cloud Platform and Kubernetes
as well as role-based certications such as Project Management Professional (PMP),
Information Technology Infrastructure Library (ITIL), The Open Group Architecture
Framework (TOGAF), Certied Information Systems Security Professional (CSSIP)
and many more.
Employees have full access to the catalog of SAS Education classes and workshops.
Additionally, SAS provides access to a variety of external learning platforms that offer
training on current technologies and practices. These learning platforms feature adaptive
skill assessments, custom training plans and interactive learning opportunities.
Lifelong Learning
Live Training
Employees may use several learning channels to enhance their knowledge, job
performance, and technical or managerial skills. Live training encompasses technical
skill development in areas such as analytics, articial intelligence and data management,
as well as topics in DevOps, security and cloud services.
Interpersonal development assists employees in enhancing their human skills. SAS
workshops help employees strengthen their ability to communicate and resolve conict,
as well as coach them on how to improve understanding and build more effective
relationships.
SAS employees may attend any SAS training classes offered by the SAS Education
division. These public courses are offered live online or in a traditional classroom setting
(when available).
On-Demand Training
SAS employees may take training on demand from our corporate Learning Management
System (LMS). The LMS enables employees to view their training records, see course
descriptions and schedules, register online, receive reminders before class, and sign up
for interest and waiting lists. This environment provides many on-demand offerings,
including self-study materials, live web training, and downloadable videos from both
external and internal sources. Dedicated training specialists work with internal subject-
matter experts to coordinate and record workshops on new technologies – often before
products are released. By providing this training early, the LMS empowers those in the
eld with the hands-on experience needed to successfully deploy SAS technology.
68
^
CONTENTS
QUALITY IMPERATIVE
Mandatory Training
Mandatory employee training is delivered and tracked through the LMS, fostering a
culture of continuous employee skill development. All employees, and applicable third
parties, are required to take appropriate courses to ensure a respectful and ethical
culture, and to protect critical customer and corporate assets. Mandatory training
encompasses subjects such as diversity in the workplace, corporate ethics, and SAS
corporate policies, such as policies on data protection and information security.
Mandatory training can also target specic roles, such as the technical skills program
for SAS R&D designed to elevate managers’ and engineers’ skills in next-generation
development methodologies.
The SAS Library
The mission of SAS Library and Information Services is to deliver quality research to
support initiatives and encourage data-driven technical and business decisions across
every major division of SAS. It offers more than 10,000 books, thousands of online
periodicals, and access to premier subscription databases and third-party online
learning platforms.
In addition to online journals, magazines and books, the SAS Library provides a research
service that assists employees across all divisions with requests of varying complexity.
It monitors access to external education providers for training in next-generation
development methodologies. The SAS Library maintains agreements with local
universities and document delivery vendors for a widened information base.
Collaborative Education and Knowledge Sharing
Within SAS, employees continually exchange subject-matter expertise to optimize the
development, delivery and support of SAS products and solutions. SAS employees are
driven to excel, and frequently share their knowledge by either plugging into one of the
existing channels for collaborative education or by innovating their own. Below is just a
small sampling of the many collaborative and knowledge-sharing channels at SAS:
• Big Ideas – formal employee presentations designed to enlighten and inspire.
• Blogs – SAS hosts a blog site, and employees frequently use blogging to
disseminate information.
• Lightning Talks – short talks focused on specic topics such as cybersecurity.
• Quality Week – a week of articles, blogs, presentations, seminars, activities and
formal sessions that are coordinated around the topic of whole company quality.
• Specialized forums – employee-driven forums on technical topics such as DevOps,
Product Security and software testing that are open to all. Employees with shared
interests and skill sets gather to share information and best practices based on their
working experience.
• Unconferences – gatherings of subject-matter experts around a particular topic,
with no set agenda.
69
^
CONTENTS
QUALITY IMPERATIVE
Appendix 5: Quality in SAS Education
SAS Education offers technical training and professional development in a variety of
training methods that allow all learning styles, budgets, and curriculum needs to be met.
Our web-based learning options continue to grow in order to serve all industries.
• SAS Live Web classrooms allow real-time interaction between instructors and
students while working together in a virtual lab, giving customers access to the latest
SAS software without leaving their work environment.
• SAS Education offers connected classroom environments in Arlington, VA and Cary,
NC on request, which bring full-day training content to in-person and Live Web
virtual students.
• SAS digital subscriptions are self-paced, on-demand educational products that offer
customers a way to learn at their own pace. They include multiple courses that cover
a specic career pathway or certication.
• Whether you’re looking to train your team or build your own data skills, we’ve got you
covered with three types of subscriptions designed to meet the needs of individuals
or small, medium and large enterprises, including SAS Premium Learning Subscription,
SAS Learning Subscription, and many topic-based digital learning subscriptions.
• We partner with third-party learning platforms Coursera and LinkedIn to offer
beginning and intermediate SAS training.
• In addition to fee-based training, we offer nearly 400 free SAS tutorials and several
free e-learning courses. Free course topics include programming, statistics,
administration, open source integration, and SAS Viya.
SAS Education supports the professional development needs of its users by offering a
concierge-style approach to learning.
• SAS Education Adoption Services provide expert guidance via a data-driven approach
to identifying skills gaps, creating custom learning plans and providing ongoing
learning support.
• Business Knowledge Series courses provide knowledge and experience from a global
network of industry experts through focused, in-depth seminars.
• Conferences and events provide group settings for knowledge transfer, training,
certication and networking.
• The SAS Certied Professional program enables users to earn globally recognized
credentials that conrm their expertise in using the software. These credentials, in
turn, provide companies with a very valuable resource: highly skilled personnel.
SAS Education is involved in rollout and enablement of major new software initiatives to
support sales and adoption, including supporting SAS Viya education through:
• Free how-to tutorial videos supporting SAS Viya and related products, found at
http://video.sas.com/#category/videos/sas-viya.
• A free SAS Viya Administration Enablement e-learning course, found at https://learn.
sas.com/course/view.php?id=508.
• Instructor-led courses for administration, data management, programming, advanced
analytics, SAS Visual Analytics and solutions, found at https://learn.sas.com/search/
index.php?q=viya. Additional courses will be available to support the most recent
release of Viya and the related products.
70
^
CONTENTS
QUALITY IMPERATIVE
• Tutorials, Hands-on Workshops, YouTube videos and courses for SAS Explore.
• Video tutorial libraries for Early Adopter releases and trials as needed.
Quality in SAS
®
Training Courses
To ensure that SAS training courses are useful for our customers and that they meet
the ever-changing needs of their business, SAS Education employees design and
develop training courses at both the individual course level and the curriculum level.
Our process is based on established instructional systems design theory and practice
and incorporates the process areas of the Capability Maturity Model (a software
development methodology). Further, it provides a framework for continuous quality
improvement. The analysis, design and development phases of this process are of
particular importance.
The analysis phase begins with signicant input from a variety of internal and
external sources specic to the industry and to the proposed training objectives. The
development team collects pertinent data from related courses, students, customers
and resources across SAS and works with the course Project Sponsorship Team to
develop training programs that will be helpful to our users.
In the design phase, SAS instructional designers use the results of the analysis phase
to plan the instructional sequence of individual courses. During this phase, the work plan
is written, course structure and ow are considered, and a detailed course outline is
developed. Most important to quality in this phase is course design and content review.
Subject matter experts review course design plans and provide feedback to the project
sponsorship team about topics such as instructional ow, course data, delivery methods
and technical issues. The feedback helps course developers nalize the training content
while exposing potential weaknesses in the instructional ow and examples. This
constructive feedback allows for additional quality improvement as the project
moves forward.
In the development phase, the input that is received during the course design and
content review sessions is used to create the training content. At numerous points during
the process, the project development team consults with the technical reviewers, who
look for accuracy and instructional ow while testing the demonstrations and programs
on appropriate platforms. Typically, technical reviewers include at least two instructors
and several subject matter experts from a variety of sources.
The test teach is an opportunity to validate the course content, ow and style in a real-
world setting delivering the new course to students. The audience of each test teach
consists of a combination of students whose main objective is to learn the material and
of subject matter experts who critique the training content.
A nal quality check is performed when a lead editor (or multiple editors) conducts a
comprehensive review of the course. Then, the production lead who is assigned to the
course development project performs quality checks on the nal materials.
The Education Project Ofce tracks each course development project to see that
documented processes are followed and offers project quality assurance, project
status reporting and project management support.
71
^
CONTENTS
QUALITY IMPERATIVE
Quality in Instructor Training and Certication
SAS instructors are not only recognized for their outstanding teaching skills, but are
often considered to be thought leaders in their areas of instruction. To make sure that our
instructors have the necessary expertise in the subjects that they teach, SAS holds each
instructor to exceptionally high standards. Every instructor is encouraged to become SAS
certied. Before teaching a new course, they present the teaching material to an internal
expert before being given the responsibility of presenting to customers. Then, they team
teach with experienced instructors gaining feedback on a chapter-by-chapter basis
before teaching on their own.
SAS Global Certication follows a rigorous, industry-standard development process
ensuring that all exams are valid and reliable in measuring important SAS skills. As a
global program, the knowledge measured in each exam is relevant to employers and
practitioners around the world. SAS is a leader in IT certication, with innovations in
performance-based testing and sharing of best practices with other organizations in
the industry.
Quality in Customer Service
Serving more than 30,000 customers a year, SAS Education has remained dedicated to
the users of SAS products and services. Our commitment to quality extends beyond the
technical aspects of the division’s work. Because SAS Education regards the relationship
that it develops with each training student as its greatest resource, the same high
standards for quality that are built into course development, instructor training and
certication are part of its customer care.
SAS Education’s customer service department is in communication with students
multiple times during the student life cycle. Before the start of a class, each student
could receive several communications based on when they register: a conrmation email
with information about what to expect during the course, reminders closer to the date of
the course, verication that the particular course will run and, in addition, a customer
service representative might contact a student to learn about any special requests or
needs. During a course, students enjoy an educational environment with the latest
technology staffed by experienced training center professionals who engage in open
communication with each student.
Perhaps the most important communication, however, is the post-class contact. SAS
Education instructors provide contact information to their students so that a student
can contact the instructor after class to ask questions or to receive consultation about
the material covered during a course. After attending a class, each student receives a
thank you email that provides access to extended learning where applicable and a link
to collect all feedback. By accessing the extended learning pages for the course, students
can download the course data; access extra practice examples, papers and FAQs; and
nd additional resources to help them develop the skills they learned in class.
At SAS Education, our commitment to quality, coupled with a desire to develop lasting
relationships with our students, has enabled us to become a model for other industry
training providers.
More information about SAS Education and its offerings is available on the SAS Training
website at sas.com/training.
72
^
CONTENTS
QUALITY IMPERATIVE
Appendix 6: Quality in SAS
®
Documentation
Researching New Features
Project managers and writers in the SAS Documentation Division are in constant contact
with product developers to keep abreast of new features as they are being developed.
Writers attend development meetings and subscribe to and participate in newsgroups
and blogs that are related to the products that they are documenting. They work with
new product features as the features are being developed and work with the appropriate
developers to ensure that the documentation is clear, complete and accurate.
Planning the Documentation Library
If a product is new, the writer analyzes the audience and its task workow to determine
what types of documentation are needed (for example, a deployment guide, an
administrator’s guide, a user’s guide or product help).
If an existing product is being updated, the writer reviews the documentation set and
determines whether new types of documentation are needed, or existing documentation
is outdated or obsolete. Writers consult with SAS Technical Support for input on how to
improve the documentation. SAS regularly surveys customers regarding their satisfaction
with the documentation and provides a feedback link from the documentation on
the web.
As products are updated, in addition to updates to individual documents, changes
are compiled into a single What's New topic that is delivered with the product
documentation. A summary document (called What's New in SAS) provides a high-
level overview of all the changes and new products in a SAS release. For example,
the most recent version for SAS 9.4 contains information from the initial release in
July 2013 and any subsequent SAS 9.4 releases, such as SAS 9.4M8. This summary
document is available (as HTML, PDF and EPUB) from the web. Also, documentation
that is available on the web is continually updated as needed and is labeled with the
date of the latest update.
Developing Content
Writers and software developers work together to produce new documentation.
The writer studies the software specications and works with the software that
is in development. The writer also develops and tests examples.
Most documentation is authored in an Extensible Markup Language (XML) authoring
environment, although some documentation is authored in LaTeX, Adobe FrameMaker,
Microsoft Word, or directly in HTML. When a rst draft of the documentation is ready,
project managers and writers send it out for technical review.
Employees in the SAS R&D, Technical Support, Worldwide Marketing, and Education
divisions are asked to review the documentation. These reviewers check the
documentation for technical accuracy, completeness and clarity, and send comments
back to the writers.
73
^
CONTENTS
QUALITY IMPERATIVE
Editing
All documentation is edited. There are three types of edits:
• Substantive edits occur early in the development of the documentation. These edits
address the overall structure, organization and writing style of the document.
• Copy edits concentrate on spelling, grammar, punctuation, consistency and style.
• Policy edits check for legal issues such as trademark violations, and glaring
grammatical errors that could affect comprehension or translation.
Most documentation receives a copy edit. All documentation receives at a minimum
a policy edit. As project schedules permit, substantive edits occur at the same time as
technical reviews.
Searching
For online documents, we provide search functionality. Search functionality is tested
for accuracy.
Testing and Publishing Documentation
After documentation is written and edited, it is transformed to several output types
(HTML, PDF and EPUB) and published to an internal documentation delivery site where
it can be accessed for testing.
For all output types, editors and testers use both manual processes and automated tools
to test the integrity of links within each document, and to test links to other documents.
If the testers nd errors, they send the errors to the writers for resolution. If the
conversion tools have generated the error, a problem report is submitted to the DevOps
Documentation Engineering Department. After all errors have been xed, the testers
verify that errors have been resolved correctly. This process is repeated as needed.
When documentation is complete and accurate, it is published to our external
documentation delivery site on the date that coincides with the release of the software.
For documentation that is part of the SAS help, various R&D product groups also test the
documentation for their specic products. If R&D testers nd an error, the tester records
the problem in the problem reporting system, and the problem is routed to the writer. The
writer xes the error, and the transformation, testing and publishing cycle is repeated
as needed.
Controlling Changes to the Documentation
Our source les are under a revision control system that is like the source management
system used for SAS source code. The revision control system maintains a revision
history for all les, and previous versions can be restored if needed.
Distributing Documentation to Customers
When we ship new or updated software, our documentation is updated to reect the
enhancements to the software. New and revised documentation is posted to the web
in HTML, PDF and EPUB formats. Selected titles are available for purchase in print and
e-book formats from bookstores and online booksellers.
74
^
CONTENTS
QUALITY IMPERATIVE
Tracking Problems After Software Updates
All substantive changes to documentation are tracked in a problem reporting system,
including changes to existing information and information about new features. When
we republish a document, technical errors are corrected, and revisions are reviewed and
tested as appropriate. The SAS Documentation Division encourages feedback from users
by email or through the SAS website.
Developing Software Used to Author and
Deliver Documentation
The DevOps Documentation Engineering Department develops and supports both
the SAS documentation delivery system and the software that is used by the SAS
Documentation Division to create help content for online delivery and printed books.
The R&D developers and testers use the same tools, processes, and protocols for
software development that are described in the main body of this document so that
our documentation delivery system software meets the same quality standards for
a worldwide audience.
Managing Terminology
Quality documentation depends on “quality at the source.” This means that we establish
and follow guidelines for correct, consistent, culturally sensitive, and clear words and
phrases as we describe and explain how to use SAS software. The SAS Documentation
Division leads a companywide initiative in terminology management to help us provide
quality communication and documentation for our customers. This initiative includes
managing a central repository for SAS terminology, as well as managing processes to
establish terminology quality checks throughout product development and delivery.
Terminologists in the SAS Documentation Division have the primary responsibility for
researching, creating and updating entries to the terminology database, which serves
as a resource for the entire company. Technical writers and technical editors use a
customized application that checks documents for clarity, consistency and correct
terminology. SAS Documentation also works closely with R&D to develop quality
terminology in software error messages and in user interface text.
In addition, SAS Documentation collaborates with our European and Asia Pacic
localization ofces. By focusing on quality at the source, SAS software and
documentation can be translated more accurately and efciently.
Terminology management is recognized as critical to quality offerings in a global market.
SAS Documentation is committed to continuing its leadership role in establishing quality
terminology across SAS products.
75
^
CONTENTS
QUALITY IMPERATIVE
Appendix 7: Quality in Consulting
The SAS Professional Services and Delivery division helps SAS users implement
their SAS products, solutions and offerings on-site and in the cloud. We are also well
prepared to help our clients with expert SAS services. Engaging SAS consultants can
help you tap the full power of SAS technology or services and reap maximum returns
on your investment.
To help you gain that return, we deliver implementation services of exceptional quality
– across the full range of SAS offerings – in parallel with our expert guidance. We are
committed to your satisfaction with our software and services and have a vested
interest in making sure that you get the most out of your SAS investment. To do that,
we use project methodologies that include quality management (quality assurance and
quality control), industry standard project governance practices, and highly qualied
consultants. In addition, we have experts in business transformation advisory services
and strategic consulting to help our customers navigate any organizational changes
needed to best use the power of analytics.
Our goal is to become your trusted technology and business partner.
Our Experience, Our Consultants
SAS Professional Services and Delivery offers experienced domain and industry thought
leaders in the world of business intelligence and predictive analytics. SAS consultants
have bachelor’s, master’s and doctoral degrees, as well as certications, in such areas
as computer science, statistics, operations research, project management and business
administration. SAS consultants are also experienced in performance management,
detailed consulting operations, applications development, and system analysis
and design.
When we utilize personnel from our alliance partners, we know that our clients will see
them as part of the SAS team. We work hard to make sure that partner personnel have
the same qualications and expertise that any other member of the SAS project team
would have based on the needs of the implementation. Our alliance partners represent
a select group of vendors who share the same commitment to implementation
excellence that we do.
What makes SAS Professional Services and Delivery exceptional?
• SAS Professional Services and Delivery has the experience and know-how to manage
the continual life cycle of SAS implementations.
• SAS Professional Services and Delivery knows one size does not t all. We bring
the experience of working with thousands of our clients, addressing each as a new
environment with unique needs.
76
^
CONTENTS
QUALITY IMPERATIVE
• SAS Professional Services and Delivery enables our customers to innovate and drive
value from tactical installations to strategic business transformation with a proven
methodology that adapts to each client’s capabilities, business conditions and
environment.
• SAS Professional Services and Delivery brings proven SAS implementation
methodologies and approaches that have been developed through our collective
experiences in thousands of successful projects. Available only from SAS, these
methodologies and road maps are the basis for the customization and implementation
for your company that brings proven success and increased business value.
• With our industry experience as users and business leaders, SAS Professional
Services and Delivery employees bring the contextual experience needed to drive
value and solve complex business challenges.
SAS consultants take the time to listen and learn about customers’ business
challenges and enterprise goals to establish a foundation for a strategic and successful
implementation. This is a requirement built into our methodologies. This enables us to
deliver the right SAS technology and customized services to solve customers’ unique
business requirements. By combining a staff of SAS experts, a proven implementation
methodology, quality management and project governance, we provide an excellent
consulting choice for our customers.
SAS Professional Services and Delivery information is available at sas.com/consulting.
SAS Project Methodologies
SAS project and delivery methodologies are the basis for all SAS Professional Services
and Delivery engagements; these methodologies ensure that business requirements are
aligned with SAS technology and support.
All SAS methodologies feature the following key components, with their respective
benets for project planning and execution:
• Detailed work breakdown structure enables the project team to create project
schedules faster and ensures that they have a common approach.
• Roles and responsibilities matrix enables determination of resources for each task,
and for the establishment and management of teams that work faster and better.
• Questionnaires and templates shorten time for project planning, assessment
and documentation.
• Estimation, communications and risk assessment tools help to increase mutual
understanding and satisfaction.
• Where appropriate, agile techniques, processes, and principles such as iterative
development and prototyping help optimize the work effort and communicate status.
All the methodologies’ key components enable SAS to quickly deliver superior projects.
Here are the key SAS project methodologies:
• SAS Project Management Methodology, focusing on project management processes.
• SAS Intelligence Platform Implementation Methodology, focusing on technical
implementation.
• SAS Agile Plug-In, focusing on the use of agile practices within implementation.
77
^
CONTENTS
QUALITY IMPERATIVE
SAS Project Management Methodology
The SAS Project Management Methodology is based on best industry standards
including the Project Management Institute’s Body of Knowledge, PRINCE2 and iterative
development practices.
The SAS Project Management Methodology supplies the basis on which all SAS projects
are executed. Based on industry standard project management principles, it takes into
consideration the specic requirements of a SAS project. In short, the SAS Project
Management Methodology accomplishes the following:
• Supports the delivery of the project within the agreed time frame, budget and required
features (project scope).
• Helps set and maintain the right expectations with all project stakeholders.
• Provides the necessary techniques and tools to monitor and control the project life
cycle and project risks.
SAS
®
Intelligence Platform Implementation Methodology
The SAS Intelligence Platform Implementation Methodology (IPI) is the most versatile
of SAS’ implementation methodologies. It is applicable to projects that contain any
combination or all the following:
• Data quality evaluation and resolution of issues.
• Data integration, or creating a data mart, data lake or warehouse.
• Data mining, forecasting, model development and other analytics.
• Business intelligence delivery such as query and reporting solutions.
Covering a complete implementation of the SAS Business Analytics Framework,
the methodology contains the quintessential knowledge and best practices of SAS’
more than 40 years of experience. The IPI is structured as a hybrid approach to
implementation activity and includes iterative development in all three of its branches.
Comprehensive by design, the IPI is customizable to be adapted to projects with a
narrow focus, such as data integration, data quality, data mining, or pure-play business
intelligence projects. In such projects, only a subset of phases, activities, and tasks
applies, thus avoiding unnecessary overhead.
SAS methodologies contain best practices and recommendations for areas such as:
• Project planning, estimation and execution.
• Project phases, activities, tasks and subtasks.
• Work breakdown structures.
• Assignment of roles and responsibilities.
• Questionnaires and templates.
• Project deliverables.
• Key objectives.
78
^
CONTENTS
QUALITY IMPERATIVE
SAS
®
Agile Plug-In
In today’s fast-paced business environment it is often necessary to use agile practices to
better address business challenges. These proven practices can address unique business
situations where requirements for technology may not be well dened or the approach
that the client wants to take is entirely new. In such cases, the nal deliverables or
implementation results may need to be “discovered” rather than prescribed in advance.
SAS has developed an approach that is based heavily on the Scrum framework that
enables SAS consultants to work with our clients using agile practices. SAS will work
with interested clients to make sure that there is alignment between the business and
technology needs of the implementation and the agile approach used on the project. Our
approach combines our deep experience with our existing methodologies by borrowing
tools, templates and practices where appropriate and using them in an agile context. SAS
has made a large investment in training our consultants and managers in the successful
use of Scrum and our agile practices.
This commitment, along with our many successful experiences using this approach,
can benet clients who have a need for innovation within our software or solution
implementations.
SAS Business Advisory
It is critical to the success of our clients that implementations deliver the value that
our clients expect when they engage us to do an installation and development of SAS
applications. SAS has a built-in incentive to make sure that these implementations succeed
in delivering the business value that motivates our clients to invest in SAS software and
services. As a demonstration of our commitment to our clients, SAS can provide business
advisory services to help make sure a client achieves their business objectives inherent in
the implementation. These services, which can be delivered as part of the implementation
during or after the implementation. These services can include:
• Business process transformation.
• System optimization.
• Analytical model development or optimization.
• Data governance or management.
• Strategic planning.
We recommend that our clients consider these and similar services and ask their
SAS representative about how such services can help them get the most out of
their SAS investment.
79
^
CONTENTS
QUALITY IMPERATIVE
Quality Management in SAS Project Management
Methodology
One important knowledge area within the SAS Project Management Methodology is
quality management. The purpose of quality management activities is to ensure that
the development process is carried out in accordance with written approved technical
standards and guidelines conforming to corporate policies and SAS methodologies.
Quality management supports the delivery of high-quality products and services by
providing the project staff, all levels of managers, and SAS with appropriate visibility
into, and feedback on, the processes and associated work products throughout
the development life cycle. One of the purposes of quality management is to
motivate action.
Quality management is a process made of two main components:
• Quality assurance.
• Quality control.
The rst ensures that planned processes are implemented, while the latter ensures that
the specied requirements are satised and that each of the components of the nal
product performs predictably.
Quality assurance and quality control might occasionally look at the same product
but from different perspectives. Product quality is, thus, a key measure of the
software process.
Quality Assurance and Procedures
Quality Assurance
Quality assurance (QA) focuses on the processes that are used to generate software
solutions, and its objective is to prevent defects by continually improving those processes.
It is a matter of establishing performance standards, measuring and evaluating
performance to those standards, and reporting performance. QA also requires taking
action when performance deviates from standards, such as:
• Ensuring that all projects follow current policies, standards and guidelines.
• Monitoring the results of those projects.
• Reporting the results to the management team for evaluation.
A quality assurance plan is developed at the beginning of the project depending
on the project needs. It ensures that appropriate quality activities are built into the
development and support process. The QA plan also gives the project team a guideline
to use to better meet the quality objectives of the project. The QA plan document
denes which activities should be included to meet the quality objectives of the project.
This information might be incorporated into other project documents or created as a
separate document.
80
^
CONTENTS
QUALITY IMPERATIVE
The document should meet these requirements:
• Accessible to all stakeholders.
• Rened on an ongoing basis.
• Specic to each project.
• An approved and controlled document.
Quality audits are conducted at specic points in the project to ensure that the
appropriate standards, policies and methodologies are being followed. In addition, these
audits also inspect the work products produced to determine whether required internal
and external work products have been produced. Quality audits do not test the work
products for accuracy; they determine only whether the work products have been
produced and whether they contain the appropriate authorization signatures.
Quality Assurance Procedures
Create a quality assurance plan for each project
A quality assurance plan might include items such as the following:
• Purpose.
• Denition and acronyms.
• Policies, standards, practices and guidelines, including identication
of the specic SAS development methodology to be used.
• Reviews and audits.
• Testing.
• Tools, techniques and methods.
• System and user manuals.
• Conguration management.
• Supplier control (if necessary).
• Education.
• Security.
• Existing systems.
• Operating procedures.
• Performance and revalidation.
• Specic components covered.
Conduct quality reviews
Quality reviews can include steps such as these:
• Requirements specication review conducted with the customer.
• Design specication review conducted with the customer for the user
interface portion of the design.
• Code inspections performed at peer-review sessions.
• Conguration audits performed periodically throughout the project.
• User documentation review conducted with the customer.
• Test plan review performed at peer-review sessions unless they involve
acceptance testing. In that case, they are conducted with the customer.
81
^
CONTENTS
QUALITY IMPERATIVE
SAS projects’ overall performance is reviewed on a regular basis to provide condence
that the project satises the established quality objectives. SAS uses a project review
process that combines the in-depth knowledge of the proposal and project delivery
teams with the experience of senior project managers to provide an objective appraisal
of the project’s viability and performance throughout its life cycle.
The project reviews support the following objectives:
• Improve customer satisfaction.
• Maintain SAS standards for quality.
• Use resources effectively.
• Manage and monitor delivery performance.
• Reduce project loss.
• Enhance project team satisfaction and capabilities.
• Support reuse of intellectual capital.
• Ensure compliance with SAS Project Management Methodology
and business best practices.
Quality Control
Quality control (QC), on the other hand, is focused on the product that is being created
by the implementation project, on testing that product and attempting to nd and
correct defects before the product is delivered to the customer. It includes aspects of
QA related to monitoring, inspecting, and especially testing. QC focuses on ensuring
that stakeholder needs are satised and on providing a high degree of assurance that
the components and system operate according to preapproved requirements and
specications.
The challenge of QC is to ensure that all business requirements have been addressed
and that the product functions up to dened success criteria before it is delivered to the
customer. A QC test plan is created well before coding is begun. Like all plans, test plans
are strategic directions for the testing process. The test plan includes items such as:
• Which types of testing will be performed?
• Which items will be tested and when?
• Which resources will be needed?
• What prerequisites are needed to prepare for testing?
• How will responsibilities be assigned?
• What are the expected results?
• What mitigation action will be taken when tests fail?
82
^
CONTENTS
QUALITY IMPERATIVE
Quality Control Procedures
The testing process has three parts:
• Test planning.
• Test case development.
• Testing.
Therefore, before testing can begin, we meet these prerequisites:
• An approved test plan.
• Complete test cases for use in the testing.
• A signed-off and managed set of requirements to test for.
• A documented set of the customer’s success criteria.
Quality Control Tests
• Unit testing: Testing at the lowest level sufcient to ensure that every source
statement has been executed at least once under test.
• Integration testing: Testing the interfaces between otherwise correct components
to ensure that they are compatible.
• System testing: Testing an entire software system end to end to discover common
system bugs, such as resource loss, synchronization and timing problems, and shared
le conicts.
• Testing to requirements: Testing from the users’ perspective, typically end to end,
to verify the operability of every feature.
• Stress testing: Subjecting a software system to an unreasonable load while denying
it the resources needed to process that load.
• Regression testing: More specically, this is equivalency testing – that is, re-running
a suite of tests to ensure that the current version behaves identically to the previous
version except in those areas known to have been changed.
• Beta testing or acceptance testing: Testing that is usually done by representative
users typically in the nal stage of testing before ofcial release.
Quality Management
To sum up, SAS quality management is a quality assurance approach that involves
the following:
• Objectively evaluating performed process, and work products, against the applicable
project management methodology and the applicable development methodology
process descriptions, standards and procedures.
• Identifying and documenting noncompliance issues.
• Providing feedback to project staff and managers, as well as to SAS Professional
Services Management on the results of the quality assurance activities.
• A description of the quality assurance reporting chain and how it ensures objectivity
of the process and product quality assurance function needs to be dened to ensure
objectivity.
• Ensuring that noncompliance issues are addressed.
83
^
CONTENTS
QUALITY IMPERATIVE
When local resolution of noncompliance issues cannot be obtained, SAS uses
established escalation mechanisms to ensure that the appropriate level of management
can resolve the issue.
When noncompliance issues are identied, they are rst addressed within the project
and resolved there, if possible, with a clear set of action plans. Any noncompliance issues
that cannot be resolved within the project are escalated to the appropriate level of
management at SAS Professional Services for resolution.
Project Governance
In order to facilitate effective communication and a quality implementation, we use
an agreed-upon project governance process throughout the full project life cycle. The
recommended approach to governance is outlined below and is integral to the SAS
Project Management Methodology.
Outstanding services governance accomplishes these goals:
• Provides a framework to dene, rene and guarantee project success.
• Actively engages the project sponsor on an executive steering committee.
• Drives the accuracy of schedule estimation.
• Increases the likelihood of services engagements on budget.
• Improves project execution.
• Proactively mitigates or reduces project risks.
• Facilitates continuous communication with all project stakeholders.
Effective project governance ensures predictability and avoids any unpleasant surprises.
Key to this is to secure clarity of roles through a formal project organization and shared
project expectations. Formal commitment to the project charter among all stakeholders
facilitates effective project governance.
Clarity of Roles: Project Organization
A formal project organization that claries each role should be established for the
project. The gure below shows an example:
Figure 4: Example of Formal Project Organization
84
^
CONTENTS
QUALITY IMPERATIVE
Project Governance: Roles
Steering Committee
The steering committee represents the interests of the business (from both a user and
a supplier perspective) and is responsible for setting the overall direction of the project.
The steering committee signs off on a key project governance document or a project
charter at the end of the project planning phase.
With its sign-off to the project charter, the steering committee sets the shared
expectations for the scope and timelines that the project team will be working to
meet. After setting the expectation, the steering committee can control the project by
exception – requiring further action to be taken only when events occur, or changes are
requested that deviate from the agreed project charter.
Project Management
The project managers are responsible for planning the project and presenting a draft
project charter to the steering committee for its review and sign-off. SAS recommends
that the project charter be developed by both SAS and client project managers in
partnership, which is done in close liaison with the various experts on the project team.
This ensures that the estimated timelines are realistic and consider the complexity
of tasks.
After the project charter has been signed off by the steering committee, the project
managers run the project on a day-to-day basis, according to agreed reporting routines.
Typically, steering committee meetings are organized at the end of each project phase
to facilitate status reporting and to verify the continued validity of the plan for the
next phase.
Project Team
The project team plays a crucial role during the planning cycle. It provides expert advice
regarding the complexity and duration of tasks. During the project execution phase, the
project team is responsible for delivering the various expected work products according
to the project’s agreed specications.
Communications are always customized to meet the jointly agreed upon information
needs of the project and of the stakeholders.
85
^
CONTENTS
QUALITY IMPERATIVE
Appendix 8: SAS
®
Offerings and Products
The SAS platform is an integrated system of software products that provide complete
control over data access, management, analysis and presentation. SAS solutions are
compatible with the SAS platform. There are other products distributed by SAS but not
integrated into the product. For the most recent product list, see sas.com/en_us/software/
all-products.html#all-products-a-z.
SAS products and solutions, and other products that are distributed by SAS and subject to
this document’s publish date, include the following (this list is subject to change over time):
Base SAS
®
JMP
®
JMP
®
Clinical
JMP
®
Live
JMP
®
Pro
JMP
®
Student
SAS/ACCESS
®
SAS/AF
®
SAS/CONNECT
®
SAS/ETS
®
SAS/GIS
®
SAS/GRAPH
®
SAS/IML
®
SAS/IML
®
Studio
SAS/OR
®
SAS/QC
®
SAS/SHARE
®
SAS/STAT
®
SAS
®
360 Discover
SAS
®
360 Engage
SAS
®
360 Plan
SAS
®
Adaptive Learning and Intelligent Agent System
SAS
®
Add-In for Microsoft Ofce
SAS
®
Analytics for IoT
SAS
®
Anti-Money Laundering
SAS
®
Asset and Liability Management for Banking
SAS
®
Asset Performance Analytics
SAS
®
Assortment Planning
SAS
®
Banking Analytics Architecture
SAS
®
Business Intelligence Dashboard
SAS
®
Business Intelligence Report Services
SAS
®
Business Orchestration Services
SAS
®
Business Rules Manager
SAS
®
Capital Requirements for Market Risk
SAS
®
Clinical Enrollment Simulation
SAS
®
Clinical Trial Data Transparency
SAS
®
Commodity Risk Analytics
SAS
®
Continuous Monitoring for Procurement Integrity
SAS
®
Continuous Monitoring Framework
SAS
®
Cost and Protability Management
SAS
®
Credit Assessment Manager
SAS
®
Credit Scoring
86
^
CONTENTS
QUALITY IMPERATIVE
SAS
®
Currency Transaction Reporting
SAS
®
Customer Due Diligence
SAS
®
Customer Intelligence 360
SAS
®
Data Governance
SAS
®
Data Integration Server
SAS
®
Data Loader for Hadoop
SAS
®
Data Management
SAS
®
Data Preparation
SAS
®
Data Quality
SAS
®
Data Surveyor for SAP
SAS
®
Decision Manager
SAS
®
Demand Planning
SAS
®
Demand Signal Repository
SAS
®
Deployment Tester
SAS
®
Detection and Investigation for Banking
SAS
®
Detection and Investigation for Government
SAS
®
Detection and Investigation for Health Care
SAS
®
Detection and Investigation for Insurance
SAS
®
Dynamic Actuarial Modeling
SAS
®
Econometrics
SAS
®
Energy Forecasting
SAS
®
Enterprise BI Server
SAS
®
Enterprise Guide
®
SAS
®
Enterprise Miner
™
SAS
®
Enterprise Miner
™
for Desktop
SAS
®
Environment Manager
SAS
®
Event Stream Manager
SAS
®
Event Stream Processing
SAS
®
Event Stream Processing Studio
SAS
®
Expected Credit Loss
SAS
®
Factory Miner
SAS
®
Federation Server
SAS
®
Field Quality Analytics
SAS
®
Financial Crimes Monitor
SAS
®
Financial Management
SAS
®
Financial Planning
SAS
®
Forecast Analyst Workbench
SAS
®
Forecast Server
SAS
®
Forecasting for Desktop
SAS
®
Foundation Services
SAS
®
Fraud Management
SAS
®
Fraud Network Analysis
SAS
®
Governance and Compliance Manager
SAS
®
Grid Manager for Hadoop
SAS
®
Grid Manager for Platform Suite
SAS
®
Health
SAS
®
High-Performance Entity and Network Generation
SAS
®
In-Database Technologies for Azure Synapse Analytics
SAS
®
In-Database Technologies for Cloudera
SAS
®
In-Database Technologies for Databricks
SAS
®
In-Database Technologies for Hadoop Cloud Services
SAS
®
In-Database Technologies for Teradata
87
^
CONTENTS
QUALITY IMPERATIVE
SAS
®
Information Delivery Portal
SAS
®
Information Map Studio
SAS
®
Information Catalog
SAS
®
Information Governance
SAS
®
In-Memory Statistics for Hadoop
SAS
®
Insurance Analytics Architecture
SAS
®
Integration Technologies
SAS
®
Intelligence and Investigation Management
SAS
®
Intelligent Decisioning
SAS
®
Inventory Optimization Workbench
SAS
®
IT Resource Management
SAS
®
IT Resource Management Adapter for SAP Server
SAS
®
LASR Analytic Server
SAS
®
Law Enforcement Intelligence
SAS
®
Life Science Analytics Framework
SAS
®
Machine Learning
SAS
®
Markdown Optimization
SAS
®
Merchandise Allocation
SAS
®
Merchandise Planning
SAS
®
Metadata Bridges
SAS
®
Metadata Server
SAS
®
Mobile Investigator
SAS
®
Model Implementation Platform
SAS
®
Model Manager
SAS
®
Model Risk Management
SAS
®
Network Algorithms
SAS
®
Ofce Analytics
SAS
®
OLAP Cube Studio
SAS
®
OLAP Monitor
SAS
®
OLAP Server
SAS
®
Optimization
SAS
®
Pack Optimization
SAS
®
Personal Login Manager
SAS
®
Production Quality Analytics
SAS
®
Promotion Optimization
SAS
®
Qualitative Assessment Manager
SAS
®
Quality Analytic Suite Foundation
SAS
®
Quality Knowledge Base for Customer Information
SAS
®
Quality Knowledge Base for Product Data
SAS
®
Real-Time Decision Manager
SAS
®
Regular Price Optimization
SAS
®
Risk Analytics Builder
SAS
®
Risk and Finance Workbench
SAS
®
Risk Dimensions
®
SAS
®
Risk Engine
SAS
®
Risk Modeling Workbench
SAS
®
Risk Reporting Repository
SAS
®
Scalable Performance Data Engine
SAS
®
Scalable Performance Data Server
SAS
®
Scoring Accelerators
SAS
®
Simulation Studio
SAS
®
Size Optimization
88
^
CONTENTS
QUALITY IMPERATIVE
SAS
®
Size Proling
SAS
®
Social Network Analysis
SAS
®
Solution for CECL
SAS
®
Solution for IFRS 17
SAS
®
Solution for IFRS 9
SAS
®
Solution for LDTI
SAS
®
Solution for Regulatory Capital
SAS
®
Solution for Solvency II
SAS
®
Solution for Stress Testing
SAS
®
Studio
SAS
®
Studio Analyst
SAS
®
Studio Engineer
SAS
®
Text Analytics
SAS
®
Text Miner
SAS
®
Text Miner for Desktop
SAS
®
Underwriting Risk Management for P&C Insurance
SAS
®
Visual Analytics
SAS
®
Visual Data Mining and Machine Learning
SAS
®
Visual Forecasting
SAS
®
Visual Investigator
SAS
®
Visual Scenario Designer
SAS
®
Visual Statistics
SAS
®
Visual Text Analytics
SAS
®
Viya
®
SAS
®
Viya
®
Advanced
SAS
®
Viya
®
Enterprise
SAS
®
Viya
®
on Microsoft Azure
SAS
®
Viya
®
Programming
SAS
®
Viya
®
with SingleStore
SAS
®
Web Report Studio
SAS
®
Web Report Viewer
SAS
®
Workow Manager
89
^
CONTENTS
QUALITY IMPERATIVE
Glossary
• Accessibility: The degree to which a product, service or environment is usable and
available to people with disabilities, including those with visual, auditory, motor or
cognitive impairments. Accessibility ensures that products and services can be
accessed and used by the widest possible audience. The SAS Disability Support
Center contains various resources for SAS users with disabilities.
• Advanced Analytics: A set of techniques and methods used to analyze and interpret
complex data sets to uncover insights and gain deeper understanding of business
operations and customer behavior. Predictive modeling, data mining, machine learning
and statistical analysis are used to identify patterns, trends and anomalies in the data.
Advanced analytics is used to support decision-making, optimize processes and drive
business growth.
• Articial Intelligence (AI): The theory and development of computer systems able
to perform tasks that normally require human intelligence, such as visual perception,
speech recognition, decision-making and translation between languages.
• Business Continuity Management: A holistic management process including
advanced planning and preparation of an organization to maintaining business
functions or quickly resuming after a disaster has occurred.
• Computer Vision: A eld of articial intelligence (AI) that enables computers and
systems to derive meaningful information from digital images, videos and other visual
inputs and then take actions or make recommendations based on that information.
• Continuous Integration/Continuous Delivery or Continuous Deployment (CI/CD):
A software development practice that involves automating the building, testing and
deployment of software applications. CI/CD is used to improve the efciency, quality
and reliability of software development by enabling faster feedback, reducing the risk
of errors and increasing the speed of delivery. It involves the use of various tools and
practices such as version control systems, automated testing frameworks and
deployment pipelines to automate the software development process and accelerate
the time to market for new features and improvements.
• Continuous Delivery (CD): A software development practice where code changes are
automatically built, tested and deployed to production or other environments, after
passing a series of automated tests and manual reviews. The goal of continuous
delivery is to deliver software faster and more reliably, while reducing the risk of
errors and minimizing the time to market for new features and improvements.
• Continuous Deployment: A software development practice where code changes are
automatically built, tested and deployed to production or other environments, without
any human intervention or review. The goal of continuous deployment is to automate
the entire software delivery process, from code changes to production deployment,
enabling organizations to release software faster and more frequently while
maintaining high quality and reliability.
• Continuous Integration (CI): A software development practice where developers
frequently integrate their code changes into a shared repository, triggering an
automated build and test process to detect integration issues early in the
development cycle. The goal of continuous integration is to improve code quality,
reduce integration risks and accelerate the software development process.
• Data Lineage: The process of understanding, recording and visualizing data as it
ows from data sources to consumption. This includes all transformations the data
underwent along the way, such as how the data was transformed, what changed and
why changes were made.
90
^
CONTENTS
QUALITY IMPERATIVE
• Deep Learning: A subeld of machine learning that involves training articial neural
networks to learn and recognize patterns in data. It involves multiple layers of
interconnected nodes that process information in a hierarchical manner to extract
features and make predictions. Deep learning is used in various applications, such
as computer vision, speech recognition, natural language processing and robotics.
• DevOps: A software development methodology that emphasizes collaboration and
communication between development teams and IT operations teams, with the goal
of delivering high-quality software products more rapidly and efciently. DevOps
(development + operations) involves integrating software development, testing,
deployment and operations into a single, continuous process to streamline the
development and deployment of software.
• Forecasting: A eld of articial intelligence used to make scientic predictions about
the future without requiring oversight.
• IDeaS: A wholly owned subsidiary of SAS Institute that produces a suite of revenue
management software and services for the hospitality industry.
• JMP Statistical Discovery LLC: A wholly owned subsidiary of SAS Institute that
produces the JMP® suite of software used for interactive, visual statistical data
analysis.
• Machine Learning: A subeld of articial intelligence that involves the use of
algorithms and statistical models to enable computers to learn from data without
being explicitly programmed. It involves building models that can make predictions
or decisions based on patterns in the data. Machine learning is used in various
applications, such as image recognition, natural language processing, recommender
systems and fraud detection.
• Model Governance: The process for how an organization controls access, implements
policy and tracks activity for models, and helps accountability and traceability to
machine learning models.
• Multivendor Architecture (MVA): The underlying design of SAS Foundation, which
enables SAS code to be compiled for, and run on, diverse host platforms without
requiring any modications.
• Natural Language Processing: An interdisciplinary subeld of linguistics, computer
science and articial intelligence concerned with the interactions between computers
and human language, in particular how to program computers to process and analyze
large amounts of natural language data.
• Numerical Accuracy: The degree to which the numerical results obtained from a
computation or measurement are correct and reliable. It is a measure of how closely
the calculated or measured values match the true values or expected values.
• Optimization: The application of articial intelligence (AI) technologies, such as
machine learning and advanced analytics to automate problem-solving and processes
in network and IT operations, and to enhance network design and optimization
capabilities.
• Reliability: The degree to which a system, product or service consistently performs
its intended function over time and under various conditions. It is a measure of the
dependability and consistency of a system, and it is often evaluated in terms of factors
such as uptime, downtime, mean time between failures and mean time to repair.
• SAS Component Language (SCL): A programming language designed to facilitate the
development of interactive applications using SAS.
• Threaded Kernel (TK) Libraries: Lightweight, threadsafe services upon which
concurrent/threaded code may be written, allowing SAS products written in SAS/C®
to take advantage of servers with multiple processors.
91
^
CONTENTS
QUALITY IMPERATIVE
What’s New
May 2024:
• Added API Standard to include information about SAS’ approach to developing
secure and robust APIs.
• Added R&D Engineering Standards to include information about SAS’ R&D
engineering requirements.
• Updated Third Party Software and Open Source Contributions to include
additional information.
• Updated Appendix 3: Installation and Operational Qualication for SAS
®
9.4 and
SAS Viya with information about SAS Viya Operational Qualication (OQ) tests.
• Various minor editorial and content updates.
November 2023:
• Expanded Protecting Privacy to include more information about SAS’ approach to
individual privacy rights and privacy and data protection laws.
• Added Environmentally Responsible IT section to reect the SAS IT organization’s
approach to corporate environmental sustainability.
• Expanded Software Globalization to reect current denitions and approaches to
SAS’ software globalization efforts.
• Updated Maintenance and Support to include information about SAS Viya platform
hot xes and maintenance releases.
• Various minor editorial and content updates.
May 2023:
• Added Glossary and What’s New sections.
• Updated Our Employees: Employee Training and Appendix 4: Employee Training
to reect updates to training processes.
• Updated Our Employees: Quality Starts With Communication to reect updates
to internal communication processes.
• Various minor editorial and content updates.
Release Information
The version of this paper is May 2024.
Unless otherwise indicated, this document relates only to SAS 9.4, SAS Viya, and
the products that are available with SAS 9.4 and SAS Viya. It also relates to services
from the date of this paper forward. Quality processes are continually evolving.
Therefore, SAS reserves the right to modify the processes described in this
document at any time. If you are using SAS 9.4 and SAS Viya and have questions
about processes in those releases, send email to [email protected].
Learn more about SAS solutions at sas.com.
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries.
®
indicates USA
registration. Other brand and product names are trademarks of their respective companies. Copyright © 2024, SAS Institute Inc. All rights reserved. 106810_G275440.0524