Draft NIST SP 800-204C, Implementation of DevSecOps for a

Withdrawn Draft

Warning Notice

The attached draft document has been withdrawn and is provided solely for historical purposes.

It has been superseded by the document identified below.

Withdrawal Date March 8, 2022

Original Release Date September 29, 2021

Superseding Document

Status Final

Series/Number NIST Special Publication 800-204C

Title Implementation of DevSecOps for a Microservices-based

Application with Service Mesh

Publication Date March 2022

DOI https://doi.org/10.6028/NIST.SP.800-204C

CSRC URL https://csrc.nist.gov/publications/detail/sp/800-204c/final

Additional Information

Draft NIST Special Publication 800-204C 1

Implementation of DevSecOps for a

Microservices-based Application with

Service Mesh

Ramaswamy Chandramouli 9

This publication is available free of charge from: 13

https://doi.org/10.6028/NIST.SP.800-204C-draft14

Draft NIST Special Publication 800-204C 15

Implementation of DevSecOps for a

Microservices-based Application with

Service Mesh

Ramaswamy Chandramouli 21

Computer Security Division 22

Information Technology Laboratory 23

This publication is available free of charge from: 30

https://doi.org/10.6028/NIST.SP.800-204C-draft 31

September 2021 34

U.S. Department of Commerce 39

Gina M. Raimondo, Secretary 40

National Institute of Standards and Technology 42

James K. Olthoff, Performing the Non-Exclusive Functions and Duties of the Under Secretary of Commerce 43

for Standards and Technoogy & Director, National Institute of Standards and Technology 44

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

Authority 45

This publication has been developed by NIST in accordance with its statutory responsibilities under the 46

Federal Information Security Modernization Act (FISMA) of 2014, 44 U.S.C. § 3551 et seq., Public Law 47

(P.L.) 113-283. NIST is responsible for developing information security standards and guidelines, including 48

minimum requirements for federal information systems, but such standards and guidelines shall not apply 49

to national security systems without the express approval of appropriate federal officials exercising policy 50

authority over such systems. This guideline is consistent with the requirements of the Office of Management 51

and Budget (OMB) Circular A-130. 52

Nothing in this publication should be taken to contradict the standards and guidelines made mandatory and 53

binding on federal agencies by the Secretary of Commerce under statutory authority. Nor should these 54

guidelines be interpreted as altering or superseding the existing authorities of the Secretary of Commerce, 55

Director of the OMB, or any other federal official. This publication may be used by nongovernmental 56

organizations on a voluntary basis and is not subject to copyright in the United States. Attribution would, 57

however, be appreciated by NIST. 58

National Institute of Standards and Technology Special Publication 800-204C 59

Natl. Inst. Stand. Technol. Spec. Publ. 800-204C, 40 pages (September 2021) 60

CODEN: NSPUE2 61

This publication is available free of charge from: 62

https://doi.org/10.6028/NIST.SP.800-204C-draft 63

Certain commercial entities, equipment, or materials may be identified in this document in order to describe an 64

experimental procedure or concept adequately. Such identification is not intended to imply recommendation or 65

endorsement by NIST, nor is it intended to imply that the entities, materials, or equipment are necessarily the best 66

available for the purpose. 67

There may be references in this publication to other publications currently under development by NIST in accordance 68

with its assigned statutory responsibilities. The information in this publication, including concepts and methodologies, 69

may be used by federal agencies even before the completion of such companion publications. Thus, until each 70

publication is completed, current requirements, guidelines, and procedures, where they exist, remain operative. For 71

planning and transition purposes, federal agencies may wish to closely follow the development of these new 72

publications by NIST. 73

Organizations are encouraged to review all draft publications during public comment periods and provide feedback to 74

NIST. All NIST Computer Security Division publications, other than the ones noted above, are available at 75

http://csrc.nist.gov/publications. 76

Public comment period: September 29, 2021 through November 1, 2021 77

National Institute of Standards and Technology 78

Attn: Computer Security Division, Information Technology Laboratory 79

100 Bureau Drive (Mail Stop 8930) Gaithersburg, MD 20899-8930 80

Email: sp800-204c-comm[email protected] 81

All comments are subject to release under the Freedom of Information Act (FOIA). 82

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

Reports on Computer Systems Technology 84

The Information Technology Laboratory (ITL) at the National Institute of Standards and 85

Technology (NIST) promotes the U.S. economy and public welfare by providing technical 86

leadership for the Nation’s measurement and standards infrastructure. ITL develops tests, test 87

methods, reference data, proof of concept implementations, and technical analyses to advance the 88

development and productive use of information technology. ITL’s responsibilities include the 89

development of management, administrative, technical, and physical standards and guidelines for 90

the cost-effective security and privacy of other than national security-related information in federal 91

information systems. 92

Abstract 93

Cloud-native applications have evolved into a standardized architecture consisting of multiple 94

loosely coupled components called microservices (implemented as containers), supported by 95

code for providing application services called service mesh. Both of these components are hosted 96

on a container orchestration and resource management platform, which is called a reference 97

platform in this document. Due to security, business competitiveness, and its inherent structure 98

(loosely coupled application components), this class of applications needs a different application 99

development, deployment, and runtime paradigm. DevSecOps (consisting of three acronyms for 100

Development, Security, and Operations, respectively) has been found to be a facilitating 101

paradigm for these applications with primitives such as Continuous Integration, Continuous 102

Delivery, and Continuous Deployment (CI/CD) pipelines. These pipelines are workflows for 103

taking the developer’s source code through various stages, such as building, testing, packaging, 104

deployment, and operations supported by automated tools with feedback mechanisms. In 105

addition to the application code, and the code for application services (service mesh), the 106

architecture has functional elements for infrastructure (computing, networking, and storage 107

resources), runtime policies (authentication, authorization etc.) and continuous monitoring of the 108

health of the application (Observability), which can be deployed through declarative codes. 109

Thus, separate CI/CD pipelines can be created for all of the five code types. 110

The objective of this document is to provide guidance for the implementation of DevSecOps 111

primitives for a reference platform hosting a cloud-native application with the functional 112

elements described above. The benefits of this approach for high security assurance and for 113

enabling continuous authority to operate (C-ATO) are also discussed. 114

115

Keywords 116

container orchestration and resource management platform; DevSecOps; CI/CD pipelines; 117

infrastructure as code; policy as code; observability as code; GitOps; workflow models; static 118

AST; dynamic AST; interactive AST; SCA. 119

120

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

iii

Acknowledgments 121

The author would like to express their first thanks to Mr. David Ferraiolo of NIST for initiating 122

this effort to provide a targeted guidance for the implementation of DevSecOps primitives for the 123

development, deployment, and monitoring of services in microservices-based applications with 124

service mesh infrastructure. Sincere thanks to Mr. Nicolas Chaillan, CSO US Air Force, for a 125

detailed and insightful review and feedback. Thanks are also due to Zack Butcher of Tetrate, Inc. 126

for suggesting the title for this document. The author also expresses thanks to Isabel Van Wyk of 127

NIST for her detailed editorial review. 128

Call for Patent Claims 129

This public review includes a call for information on essential patent claims (claims whose use 130

would be required for compliance with the guidance or requirements in this Information 131

Technology Laboratory (ITL) draft publication). Such guidance and/or requirements may be 132

directly stated in this ITL Publication or by reference to another publication. This call also 133

includes disclosure, where known, of the existence of pending U.S. or foreign patent applications 134

relating to this ITL draft publication and of any relevant unexpired U.S. or foreign patents. 135

ITL may require from the patent holder, or a party authorized to make assurances on its behalf, 136

in written or electronic form, either: 137

a) assurance in the form of a general disclaimer to the effect that such party does not hold 138

and does not currently intend holding any essential patent claim(s); or 139

b) assurance that a license to such essential patent claim(s) will be made available to 140

applicants desiring to utilize the license for the purpose of complying with the guidance 141

or requirements in this ITL draft publication either: 142

i. under reasonable terms and conditions that are demonstrably free of any unfair 143

discrimination; or 144

ii. without compensation and under reasonable terms and conditions that are 145

demonstrably free of any unfair discrimination. 146

Such assurance shall indicate that the patent holder (or third party authorized to make assurances 147

on its behalf) will include in any documents transferring ownership of patents subject to the 148

assurance, provisions sufficient to ensure that the commitments in the assurance are binding on 149

the transferee, and that the transferee will similarly include appropriate provisions in the event of 150

future transfers with the goal of binding each successor-in-interest. 151

The assurance shall also indicate that it is intended to be binding on successors-in-interest 152

regardless of whether such provisions are included in the relevant transfer documents. 153

Such statements should be addressed to: sp800-204c-comm[email protected]v 154

155

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

Executive Summary 156

Cloud-native applications have evolved into a standardized architecture consisting of the 157

following components: 158

• Multiple loosely coupled components called microservices (implemented as containers) 159

• An application services infrastructure called Service Mesh (providing services such as 160

secure communication, authentication, and authorization for users, services, and devices) 161

162

Due to security, business competitiveness, and its inherent structure (loosely coupled application 163

components), this class of applications needs a different application, deployment, and runtime 164

monitoring paradigm – collectively called the software life cycle paradigm. DevSecOps 165

(consisting of three acronyms for Development, Security, and Operations, respectively) is one of 166

the facilitating paradigms for the development, deployment, and operation of these applications 167

with primitives such as Continuous Integration, Continuous Delivery, and Continuous 168

Deployment (CI/CD) pipelines. 169

170

CI/CD pipelines are workflows for taking the developer’s source code through various stages, 171

such as building, functional testing, security scanning for vulnerabilities, packaging, and 172

deployment supported by automated tools with feedback mechanisms. In addition to the 173

application code and the code for providing application services, this architecture may be made 174

up of functional elements for infrastructure, runtime policies (such as zero trust policy), and 175

continuous monitoring of the health of the application, the last three of which can be deployed 176

through declarative codes. Thus, separate CI/CD pipelines can be created for all five code types. 177

The runtime behavior of each of these code types is also described to highlight the roles that they 178

play in the overall execution of the application. 179

180

Though cloud-native applications have a common architectural stack, the platform on which the 181

components of the stack run may vary. The platform is an abstraction layer over a physical (bare 182

metal) or virtualized (e.g., virtual machines, containers) infrastructure. In this document, the 183

chosen platform is a container orchestration and resource management platform (e.g., 184

Kubernetes). To unambiguously refer to this platform or application environment throughout this 185

document, it is called the Reference Platform for DevSecOps Primitives, or simply the reference 186

platform. 187

188

The objective of this document is to provide guidance for the implementation of DevSecOps 189

primitives for the reference platform. The benefits of this implementation for high security 190

assurance and the use of the artifacts within the pipelines for providing continuous authority to 191

operate (C-ATO) using risk management tools and dashboard metrics are also described. 192

193

194

195

196

197

198

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

199

Table of Contents 200

Executive Summary ..................................................................................................... iv 201

1 Introduction ............................................................................................................ 1 202

1.1 Scope .............................................................................................................. 1 203

1.2 Related DevSecOps Initiatives ....................................................................... 2 204

1.3 Target Audience.............................................................................................. 3 205

1.4 Relationship to Other NIST Guidance Documents .......................................... 3 206

1.5 Organization of this document ........................................................................ 4 207

2 Reference Platform for the Implementation of DevSecOps Primitives ............. 5 208

2.1 Container Orchestration and Resource Management Platform ......................... 5 209

2.1.1 Security Limitations of Orchestration Platform ...................................... 5 210

2.2 Service Mesh Software Architecture ............................................................... 6 211

2.2.1 Data Plane ............................................................................................ 6 212

2.2.2 Control Plane ......................................................................................... 7 213

3 DevSecOps – Organizational Preparedness, Key Primitives, and 214

Implementation .............................................................................................................. 9 215

3.1 Organizational Preparedness for DevSecOps ................................................ 9 216

3.2 Seamless Evolution from DevOps to DevSecOps .......................................... 9 217

3.3 DevSecOps – Key Primitives and Implementation Tasks ............................... 10 218

3.3.1 Concept of Pipelines and the CI/CD pipeline ....................................... 10 219

3.3.2 Building Blocks for CI/CD pipelines ...................................................... 12 220

3.3.3 Designing and Executing the CI/CD pipeline ........................................ 13 221

3.3.4 Strategies for Automation ..................................................................... 15 222

3.3.5 Requirements for Automation Tools in CI/CD Pipelines ....................... 16 223

4 Implementing DevSecOps Primitives for the Reference Platform ................... 17 224

4.1 Code Types in Reference Platform and Associated CI/CD Pipelines .............. 17 225

4.2 CI/CD Pipeline for Application Code and Application Services Code .............. 19 226

4.3. CI/CD Pipeline for Infrastructure as Code ...................................................... 19 227

4.3.1 Comparison of Configuration and Infrastructure ................................... 20 228

4.4 CI/CD Pipeline for Policy as Code ................................................................... 20 229

4.5 CI/CD Pipeline for Observability as Code ....................................................... 21 230

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

4.6 Securing the CI/CD Pipeline............................................................................ 21 231

4.7 Workflow Models in CI/CD Pipelines ............................................................... 22 232

4.7.1 GitsOps Workflow Model for CI/CD – A Pull-based Model ........................... 22 233

4.8 Security Testing – Common Requirement for CI/CD Pipelines for All Code 234

Types .................................................................................................................... 23 235

4.8.1 Functional and Coverage Requirements for AST tools ........................ 24 236

4.9 Benefits of DevSecOps Primitives to Application Security in the Service Mesh237

25 238

4.10 Leveraging DevSecOps for Continuous Authorization to Operate (c-ATO) ... 26 239

5 Summary and Conclusion ................................................................................... 28 240

References ................................................................................................................... 29 241

242

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

1 Introduction 243

Cloud-native applications are made up of multiple loosely coupled components (called 244

microservices implemented as containers), operate in perimeter-less network environments 245

requiring zero trust concepts (on-premises or cloud), and are accessed by users from a diverse set 246

of locations (e.g., campus, home office, etc.). Cloud-native applications do not just refer to 247

applications that run in the cloud. They also refer to the class of applications with design and 248

runtime architectures, such as microservices, and a dedicated infrastructure for providing all 249

application services, such as security. The incorporation of zero trust principles into this class of 250

application provides techniques where access to all protected resources is enforced through 251

identity-based protection, as well as network-based protections (e.g., micro-segmentation) where 252

applicable. 253

Cloud-native applications require agile and secure updates and deployment techniques for 254

business reasons as well as the necessary resilience to respond to cybersecurity events. Hence, 255

they call for a different application development, deployment, and runtime monitoring paradigm 256

(collectively called the software life cycle paradigm) than the ones used for traditional monolithic 257

or multi-tier applications. DevSecOps (Development, Security, and Operations) is a facilitating 258

paradigm for this class of applications since it facilitates agile and secure development, delivery, 259

deployment, and operations through (a) primitives, such as Continuous Integration, Continuous 260

Delivery, and Continuous Deployment (CI/CD) pipelines (explained in section 3); (b) security 261

testing throughout the life cycle; and (c) continuous monitoring during runtime, all of which are 262

supported by automation tools. Thus, DevSecOps has the necessary primitives and other building 263

blocks to meet the design goals of cloud-native applications. 264

1.1 Scope 265

DevSecOps primitives can, in theory, be applied to many application architectures but are best 266

suited to microservices-based ones, which permit agile development paradigms due to the fact 267

that the application is made up of relatively small, loosely coupled modules called microservices. 268

Even within microservices-based architectures, the implementation of DevSecOps primitives can 269

take on different forms, depending on the platform. In this document, the chosen platform is a 270

container orchestration and resource management platform (e.g., Kubernetes). The platform is an 271

abstraction layer over a physical (bare metal) or virtualized (e.g., virtual machines, containers) 272

infrastructure. To unambiguously refer to this platform or application environment throughout 273

this document, it is called the Reference Platform for DevSecOps Primitives, or simply the 274

reference platform. 275

276

Before describing the implementation of DevSecOps primitives for the reference platform, it is 277

assumed that the following due diligence is applied with respect to deployment of the service 278

mesh component [1]: 279

• Secure design patterns for deploying and managing service mesh-based components for 280

infrastructure (e.g., network routing), policy enforcement, and monitoring 281

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

• Tests to prove that these service mesh components work as intended in a variety of scenarios 282

for all aspects of the application, such as ingress, egress, and inside services. 283

The guidance provided for implementation of DevSecOps primitives for the reference platform is 284

agnostic to (a) the tools used in DevSecOps pipelines and (b) the service mesh software, which 285

provides application services, though examples from service mesh offerings, such as Istio, are 286

used to link them to real-world application artifacts (e.g., containers, policy enforcement 287

modules, etc.). 288

The reference platform, along with DevSecOps implementation components, can therefore be 289

looked upon as having the following functional elements or code types: 290

1. The application code, which contains the application logic for transactions and database 291

access 292

2. The service mesh code, which provides application services such as network routes, 293

network resiliency services (e.g., load balancing, retries etc.), and security services 294

3. Infrastructure (consisting of computing, networking, and storage resources) as code 295

4. Policy as code for generating rules and configuration parameters for realizing security 296

objectives, such as zero trust through security controls (e.g., authentication, 297

authorization) during runtime 298

5. Observability as code for (a) triggering software related to logging, tracing 299

(communication pathways involved in executing application request), and monitoring for 300

recording all transactions and for (b) keeping track of application states during runtime. 301

302

This document covers the implementation of pipelines or workflows associated with all five code 303

types listed above. Out of them, the infrastructure as code, policy as code, and observability as 304

code belong to a special class called declarative code. A declarative code is written in a special 305

purpose language called declarative language that declares the requirements, and an associated 306

tool converts them into artifacts that make up a runtime instance. For example, in the case of 307

infrastructure as code (IaC), the declarative language models the infrastructure as a series of 308

resources. The associated configuration management tool pulls together these resources and 309

generates what are known as Manifests that define the final shape and state of the platform 310

(runtime instance) associated with the defined resources. These manifests are stored in servers 311

associated with a configuration management tool and are used by the tool to create compiled 312

configuration instructions for the runtime instance on the designated platform. Manifests are 313

generally encoded in platform-neutral representations (e.g., JSON) and fed to platform resource 314

provisioning agents through REST APIs. 315

316

1.2 Related DevSecOps Initiatives 317

DoD Enterprise DevSecOps Initiative: The Department of Defense (DOD) Enterprise 318

DevSecOps initiative spans many open-source projects with many state-of-practice ingredients. 319

The artifacts developed under this initiative are used by 12 U.S. federal agencies, five nations, 320

and a substantial number of DoD contractors and vendors. The open-source projects under this 321

initiative include but are not limited to [2]: 322

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

• The Iron Bank, a repository of DOD-vetted hardened container images 323

• Platform One, the DevSecOps platform that the department created for software deployments 324

from jets to bombers to space to clouds; it is based on the concept of continuous authority to 325

operate (cATO), which updated the DOD’s authorization process to accommodate the speed 326

and frequency of modern continuous software deployments 327

328

The DoD Enterprise DevSecOps initiative has the following state-of-practice ingredients: 329

1. Abstracted: To avoid drifts, be agnostic to the environment (e.g., cloud, on-premise, 330

classified, disconnected), and prevent lock-ins with cloud or platform layers, CNCF-331

compliant Kubernetes and OCI-compliant containers are leveraged – open-source stacks 332

with U.S. observation of code and continuous scanning. 333

2. GitOps/Infrastructure as Code (IaC): No drift, everything is code (including 334

configuration, networking etc.), and the entire stack is automatically instantiated. 335

3. Continuous Integration/Continuous Delivery pipeline (CI/CD): Fully containerized 336

and using Infrastructure as Code (IaC) 337

4. Hardened Containers: Hardened “Lego blocks” to bring options to development teams 338

(one size fits all lead to shadow IT) 339

5. Software Testing: Mandated high-test coverage 340

6. Baked-in Security: Mandated static/dynamic code analysis, container security, bill of 341

material (supply chain risk), etc. 342

7. Continuous Monitoring: 343

• Centralized logging and telemetry 344

• Automated alerting 345

• Zero trust, leveraging service mesh as sidecar (part of SCSS) down to the 346

container level 347

• Behavior detection (automated prevention leveraging AI/ML capabilities) 348

• CVE signatures scanning 349

8. Chaos engineering: Dynamically kills/restarts container with moving target defense 350

1.3 Target Audience 351

352

Since DevSecOps primitives span development (build and test for security, package, delivery, 353

deployment, continuous monitoring) and ensure secure states during runtime, the target audience 354

for the recommendations in this document includes software development, operations, and 355

security teams. 356

357

1.4 Relationship to Other NIST Guidance Documents 358

Since the Reference Platform is made up of container orchestration and resource management 359

platform and service mesh software, the following publications offer guidance for securing this 360

platform as well as background information for the contents of this document: 361

● Special Publication (SP) 800-204, Security Strategies for Microservices-based Application 362

Systems [3], discusses the characteristics of microservices-based applications and the overall 363

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

security requirements and strategies for addressing those requirements. 364

● SP 800-204A, Building Secure Microservices-based Applications Using Service-Mesh 365

Architecture [4], provides deployment guidance for various security services (e.g., 366

establishment of secure sessions, security monitoring, etc.) for a microservices-based 367

application using a dedicated infrastructure (i.e., a service mesh) that uses service proxies that 368

operate independent of the application code. 369

● SP 800-204B [5], Attribute-based Access Control for Microservices-based Applications 370

Using a Service Mesh, provides deployment guidance for building an authentication and 371

authorization framework within the service mesh that meets the security requirements, such 372

as (1) zero trust by enabling mutual authentication in communication between any pair of 373

services and (2) a robust access control mechanism based on an access control such as 374

attribute-based access control (ABAC) that can be used to express a wide set of policies and 375

is scalable in terms of user base, objects (resources), and deployment environment. 376

1.5 Organization of this document 377

The organization of the rest of the document is as follows: 378

● Chapter 2 gives a brief description of the reference platform for which guidance for 379

implementation of DevSecOps primitives is provided. 380

● Chapter 3 introduces the DevSecOps primitives (i.e., pipelines), the methodology for 381

designing and executing the pipelines, and the role that automation plays in the execution. 382

● Chapter 4 covers all facets of pipelines, including (a) Common issues to be addressed for 383

all pipelines, (b) descriptions of the pipelines for the four code types in the reference 384

platform that are listed in Section 1.1, and (c) the benefit of DevSecOps for security 385

assurance for the entire application environment (the reference platform with five code 386

types, thus carrying the DevSecOps implementation) during the entire life cycle, 387

including the “Continuous Authority to Operate (CAT).” 388

● Chapter 5 provides a summary and conclusion. 389

390

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

2 Reference Platform for the Implementation of DevSecOps Primitives 391

As stated in Section 1.1, the reference platform is a container orchestration and management 392

platform. In modern application environments, the platform is an abstraction layer over a physical 393

(bare metal) or virtualized (e.g., virtual machines, containers) infrastructure. Before the 394

implementation of DevSecOps primitives, the platform simply contains the application code, 395

which contains the application logic and the service mesh code, which in turn provides 396

application services. This section will consider the following: 397

• A container orchestration and resource management platform that houses both the 398

application code and most of the service mesh code 399

• The service mesh software architecture 400

401

2.1 Container Orchestration and Resource Management Platform 402

Since microservices are implemented as containers, a container orchestration and resource 403

management platform is used for deployment, operations, and upgrading services. Kubernetes is 404

one such open-source container orchestration and resource management platform that is widely in 405

use. 406

A typical orchestration and resource management platform consists of various logical (forming 407

the abstraction layer) and physical artifacts for the deployment of containers. For example, in 408

Kubernetes, containers run inside the smallest unit of deployment called a pod. A pod can 409

theoretically host a group of containers, though usually, only one container runs inside a pod. A 410

group of pods are defined inside what is known as a node, where a node can be either a physical 411

or virtual machine (VM). A group of nodes constitutes a cluster. Usually, multiple instances of a 412

single microservice are needed to distribute the workload to achieve the desired performance 413

level. A cluster is a pool of resources (nodes) that is used as a means to distribute the workload of 414

microservices. One of the techniques used is horizontal scaling, where microservices that are 415

accessed more frequently are allocated more instances or allocated to nodes with higher resources 416

(e.g., CPUs and/or memory). 417

2.1.1 Security Limitations of Orchestration Platform 418

Microservices-based applications require several application services, including security services 419

such as authentication and authorization, as well as the generation of metrics for individual pods 420

(monitoring), consolidated logging (to ascertain causes of failures of certain requests), tracing 421

(sequence of service requests within the application), traffic control, caching, secure ingress, 422

service-to-service (east/west traffic), and egress communication. 423

Taking the example of secure communications between Kubernetes containers, these are not 424

natively secure since there is no easy way to enforce TLS between pods as this would require in 425

maintaining hundreds of TLS certificates. Pods that communicate do not apply identity and 426

access management between themselves. Though there are tools that can be implemented to act 427

as a firewall between pods, such as the Kubernetes Network Policy, this is a layer 3 solution 428

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

rather than a layer 7 solution, which is what most modern firewalls are. This means that while 429

one can know the source of traffic, one cannot peek into the data packets to understand what they 430

contain. Further, it does not allow for making vital metadata-driven decisions, such as routing on 431

a new version of a pod based on an HTTP header. There are Kubernetes ingress objects that 432

provide a reverse proxy based on layer 7, but they do not offer anything more than simple traffic 433

routing. Kubernetes does offer different ways of deploying pods that do some form of A/B 434

testing or canary deployments, but they are done at the connection level and provide no fine-435

grained control or fast failback. For example, if a developer wants to deploy a new version of a 436

microservice and pass 10 % of traffic through it, they will have to scale the containers to at least 437

10 – nine for the old version and one for the new version. Further, Kubernetes cannot split the 438

traffic intelligently and instead balances loads between pods in a round-robin fashion. Every 439

Kubernetes container within a pod has a separate log, and a custom solution over Kubernetes 440

must be implemented to capture and consolidate them. 441

Although the Kubernetes dashboard offers monitoring features on individual pods and their 442

states, it does not expose metrics that describe how application components interact with each 443

other or how much traffic flows through each of the pods. Consolidated logging is required to 444

determine error conditions that cause an application request or transaction to fail. Tracing is 445

required to trace the sequence of containers that are invoked as part of any application request 446

based on the application logic that underlies a transaction. Since traffic flow cannot be traced 447

through Kubernetes pods out of the box, it is unclear where on the chain the failure for the 448

request occurred. 449

This is where the service mesh software can provide the needed application services and much 450

more. 451

2.2 Service Mesh Software Architecture 452

Having looked at the various application services required by microservices-based applications, 453

consider the architecture of service mesh software that provides those services. The service mesh 454

software consists of two main components: the data plane and the control plane. 455

456

2.2.1 Data Plane 457

The data plane component performs three different functions: 458

1. Secure networking functions 459

2. Policy enforcement functions 460

3. Observability functions 461

The primary component of the data plane that performs all three functions listed above is called 462

the sidecar proxy. This layer 7 (L7) proxy runs in the same network namespace (which, in this 463

platform, is the same pod) as the microservice for which it performs proxy functions. There is a 464

proxy for every microservice to ensure that a request from a microservice does not bypass its 465

associated proxy and that each proxy is run as a container in the same pod as the application 466

microservice. Both containers have the same IP address and share the same IP Table rules. That 467

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

makes the proxy take complete control over the pod and handle all traffic that passes through it 468

[6,7]. 469

The first category of functions (secure networking) includes all functions related to the actual 470

routing or communication of messages between microservices. The functions that come under 471

this category are service discovery, establishing a secure (TLS) session, establishing network 472

paths and routing rules for each microservice and its associated requests, authenticating each 473

request (from a service or user), and authorizing the request. 474

With the example of establishing a mutual TLS session, the proxy that initiates the 475

communication session will interact with the module in the control plane of the service mesh to 476

check whether it needs to encrypt traffic through the chain and establish mutual TLS with the 477

backend or target pod. Enabling this functionality using mutual TLS requires every pod to have a 478

certificate (i.e., a valid credential). Since a good-sized microservice application (consisting of 479

many microservices) may require hundreds of pods (even without horizontal scaling of individual 480

microservices through multiple instances), this may involve managing hundreds of short-lived 481

certificates. This, in turn, requires each microservice to have a robust identity and the service 482

mesh to have an access manager, a certificate store, and certificate validation capability. In 483

addition, mechanisms for identifying and authenticating the two communicating pods are 484

required for supporting authentication policies. 485

Other kinds of proxies include ingress proxies [8] that intercept the client calls into the first entry 486

point of application (first microservice that is invoked) and egress proxies that handle a 487

microservice’s request to application modules residing outside of the platform cluster. 488

The second category of functions that the data plane performs is enforcement of the policies 489

defined in the control plane through configuration parameters in the proxies (policy enforcement 490

service). 491

The third category of functions that service proxies perform sometimes in collaboration with 492

application service containers are to gather telemetry data, which helps to monitor the health and 493

state of the services, transfer logs associated with a service to the log aggregation module in the 494

control plane, and append necessary data to application request headers to facilitate the tracing of 495

all requests associated with a given application transaction. The application response is conveyed 496

by proxies back to its associated calling service in the form of return codes, a description of 497

response, or the retrieved data. 498

2.2.2 Control Plane 499

The control plane has several components. While the data plane of the service mesh mainly 500

consists of proxies running as containers within the same pod as application containers, the 501

control plane components run in their own pods, nodes, and associated clusters. The following 502

are the various functions of the control plane [7] (components in the Istio service mesh that 503

perform these functions are given in parentheses): 504

1. Service discovery and configuration of the Envoy sidecar proxies (Pilot) 505

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

2. Automated key and certificate management (Citadel) 506

3. API for policy definition and the gathering of telemetry data (Mixer) 507

4. Configuration ingestion for service mesh components (Galley) 508

5. Management of an inbound connection to the service mesh (Ingress Gateway) 509

6. Management of an outbound connection from the service mesh (Egress Gateway) 510

7. Inject sidecar proxies into those pods, nodes, or namespaces where application 511

microservice containers are hosted (Sidecar Injector) 512

Overall, the control plane helps the administrator populate the data plane component with the 513

configuration data that is generated from the policies resident in the control plane. The policies 514

for function 3 above may include network routing policies, load balancing policies, policies for 515

blue-green deployments, canary rollouts, timeout, retry, and circuit-breaking capabilities. These 516

last three are collectively called by the special name of resiliency capabilities of the networking 517

infrastructure services. Last but not the least are security-related policies (e.g., authentication and 518

authorization policies, TLS establishment policies, etc.). These policy rules are parsed by a 519

module that converts them into configuration parameters for use by executables in data plane 520

proxies that enforce those policies. 521

The service mesh is container orchestration platform-aware, interacts with the API server that 522

provides a window into application services installed in various platform artifacts (e.g., pods, 523

nodes, namespaces), monitors it for new microservices, and automatically injects sidecar 524

containers into the pods containing these new microservices. Once the service mesh inserts the 525

sidecar proxy containers, operations and security teams can enforce policies on the traffic and 526

help secure and operate the application. These teams can also configure the monitoring of 527

microservices applications without interfering with the functioning of the applications. 528

The provisioning of infrastructure, policy generation, and observability services can be automated 529

using declarative code that is part of DevSecOps pipelines. The development team can 530

concentrate their efforts on efficient development paradigms, such as code modularity and 531

structuring, without worrying about the security and management details of their implementation. 532

533

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

3 DevSecOps – Organizational Preparedness, Key Primitives, and 534

Implementation 535

DevSecOps incorporates security into the software engineering process early on. It integrates and 536

automates security processes and tooling into all of the development workflow (or pipeline as later 537

explained) in DevOps so that it is seamless and continuous. In other words, it can be looked upon 538

as a combination of the three processes: Development + Security + Operations [9]. 539

This section discusses the following aspects of DevSecOps: 540

• Organizational preparedness for DevSecOps 541

• Seamless evolution from DevOps to DevSecOps (for organizations that already have a 542

DevOps team in place) 543

• Fundamental building blocks or key primitives for DevSecOps 544

545

3.1 Organizational Preparedness for DevSecOps 546

DevSecOps is a software development, deployment, and life cycle management methodology that 547

involves a shift from one large release for an entire application or platform to the Continuous 548

Integration, Continuous Delivery, and Continuous Deployment (CI/CD) approach. This shift, in 549

turn, requires changes in the structure of a company’s IT department and its workflow. The most 550

pronounced change involves organizing a DevSecOps group that consists of software developers, 551

security specialists, and IT operations experts for each portion of the application (i.e., the 552

microservice). This smaller team not only promotes efficiency and effectiveness in initial agile 553

development and deployment but also in subsequent life cycle management activities, such as 554

monitoring behavior, developing patches, fixing bugs, or scaling the application. This cross-555

functional team with expertise in three areas forms a critical success factor for introducing 556

DevSecOps in an organization. 557

3.2 Seamless Evolution from DevOps to DevSecOps 558

559

DevOps is an agile, automated development and deployment process that uses primitives called 560

CI/CD pipelines aided by automated tools to take the software from the build phase to 561

deployment phase. These pipelines are nothing but workflows that take the developer’s source 562

code through various stages, such as building, testing, packaging, and deployment supported by 563

automated tools with feedback mechanisms. DevOps is primarily a development/deployment 564

pipeline where security assurance is provided through the use of some basic static application 565

security tools (SAST), dynamic security testing tools (DAST), and software composition analysis 566

(SCA) tools. Thus, a DevOps platform only runs during the build and deployment phases of the 567

software life cycle. 568

In a DevSecOps platform, security assurance is provided through built-in design features, such as 569

zero trust, during the build and deployment phases in addition to the use of a comprehensive set 570

of security testing tools, such as DAST and SCA tools. Security assurance is also provided during 571

the runtime/operations phase by continuous behavior detection/prevention tools that use 572

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

sophisticated techniques, such as artificial intelligence (AI) and machine learning (ML). 573

Therefore, a DevSecOps platform not only runs during build and deployment phases but also 574

during runtime/operations phase. DevSecOps pipelines also run in a production environment as 575

opposed to DevOps, which runs in a separate environment for testing/staging since workflows 576

associated with the latter terminate in the deployment phase. 577

Another distinguishing characteristic of a DevSecOps platform is that the security tools (e.g., 578

SAST, DAST, and SCAs) are so tightly integrated with IDEs and other DevOps tools that they 579

perform application security analysis, such as identifying vulnerabilities and bugs through 580

efficient scanning in the background. This is often invisible to developers without making them 581

call separate APIs for running these tools [10]. 582

In summary, DevSecOps has the following distinguishing features as compared to DevOps: 583

• Security testing and the robust incorporation of security controls are integral to all 584

pipelines instead of being relegated to a separate task or phase. 585

• The DevSecOps platform operates during runtime (in production), providing real-time 586

assurance of security through correction mechanisms and enabling the certification of 587

continuous authority to operate (C-ATO). 588

589

3.3 DevSecOps – Key Primitives and Implementation Tasks 590

591

The key primitives and implementation tasks involved are: 592

• Concept of pipelines and the CI/CD pipeline 593

• Building blocks for the CI/CD pipeline 594

• Designing and executing the CI/CD pipeline 595

• Strategies for automation 596

• Requirements for automation tools in the CI/CD pipeline 597

598

3.3.1 Concept of Pipelines and the CI/CD pipeline 599

DevSecOps is a methodology or framework for agile application development, deployment, and 600

operations for cloud-native applications [11] and is made up of stages just like any other 601

methodology. The sequence and flow of information through the stages is called workflow, 602

where some stages can be executed in parallel while some others have to follow a sequence. Each 603

stage may require the invocation of a unique job to execute the activities in that stage. 604

A unique concept that DevSecOps introduces in the process workflow is the concept of 605

“pipelines” [12]. With pipelines, there is no need to individually write jobs for each stage of the 606

process. Instead, there is only one job that starts from the initial stage, automatically triggers the 607

activities pertaining to other stages (both sequential and parallel), and creates an error-free smart 608

workflow. 609

The pipeline in DevSecOps is called the CI/CD pipeline based on the overall tasks it 610

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

accomplishes and the two individual stages it contains. CI stands for the continuous integration 611

stage. CD can denote either the continuous delivery or continuous deployment stage. Depending 612

on this latter stage, CI/CD can involve the following tasks: 613

• Build, Test, Secure, and Deliver (the tested modified code is delivered to the staging area) 614

• Build, Test, Secure, Deliver, and Deploy (the code in the stage area is automatically 615

deployed) 616

617

In the former, automation ends at the delivery stage, and the next task of deployment of the 618

modified application in the hosting platform infrastructure is performed manually. In the latter, 619

the deployment is also automated. Automation of any stage in the pipeline is enabled by tools that 620

express the pipeline stage as code. 621

The workflow process for a CI/CD pipeline is depicted in Figure 1 below: 622

623

Figure 1: CI/CD pipeline workflow [13] 624

Figure 3.1. CI/CD pipeline workflow [13] 625

The unit and integration tests shown in the diagram use SAST, DAST, and SCA tools described 626

in Section 3.2. These tests and associated tools are common to DevOps and DevSecOps. 627

Continuous integration involves developers frequently merging code changes into a central 628

repository where automated builds and tests run. Build is the process of converting the source 629

code to executable code for the platform on which it is intended to run. In the CI/CD pipeline 630

software, the developer’s changes are validated by creating a build and running automated tests 631

against the build. This process avoids the integration challenges that can happen when waiting for 632

release day to merge changes into the release branch [14]. 633

Continuous delivery is the next stage after continuous integration where code changes are 634

deployed to a testing and/or staging environment after the build stage. Continuous delivery to a 635

production environment involves the designation of a release frequency – daily, weekly, 636

fortnightly, or some other period – based on the nature of the software or the market in which the 637

organization operates. This means that on top of automated testing, there is a scheduled release 638

process, though the application can be deployed at any time by clicking a button. The deployment 639

process in continuous delivery is characterized as manual, but tasks such as the migration of code 640

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

to a production server, the establishment of networking parameters, and the specification of 641

runtime configuration data may be performed by automated scripts. 642

Continuous deployment is similar to continuous delivery except that releases happen 643

automatically [14], and changes to code are available to customers immediately after they are 644

made. The distinction between continuous delivery and continuous deployment is shown in 645

Figure 2 below. 646

647

Figure 2 - Distinction between Continous Delivery and Continous Deployment [14] 648

3.3.2 Building Blocks for CI/CD pipelines 649

The primary software for defining CI/CD pipeline resources, building the pipelines, and 650

executing those pipelines is the CI/CD pipeline software. There may be slight variations in the 651

architecture of this class of software depending on the particular offering. The following is an 652

overview of the landscape in which CI/CD tools (pipeline software) operate: 653

• Some CI/CD tools natively operate on the platform on which the application and the 654

associated resources needed are hosted (i.e., container orchestration and resource 655

management platform, such as Kubernetes), while others need to be integrated into the 656

application hosting platform through its API. An example of a former class of tool is 657

Tekton, and examples for the latter class include Jenkins, Travis, Bamboo, and CircleCI. 658

Some advantages of using the CI/CD tools that are native to the application hosting 659

platform are: 660

(a) It makes it easier to deploy, maintain, and manage the CI/CD tool itself. 661

(b) Every pipeline defined by the CI/CD tool becomes another platform-native 662

resource and is managed the same way. In fact, all entities required for executing 663

pipelines, such as Tasks and Pipelines (which then act as blueprints for other 664

entities, such as TaskRuns and PipelineRuns, respectively), can be created as 665

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

CustomResourceDefinitions (CRDs) built on top of resources native to the platform 666

[15]. Software with this type of architecture may be used by other CI/CD pipeline 667

software offerings to facilitate faster defining of pipelines. 668

• Some CI/CD tool offerings (e.g., GitLab, Jenkins X) integrate with a Git Repository – one 669

for each application and for each environment (staging/production). All changes in 670

application modules, infrastructure, or configuration are made and stored in these Git 671

repositories. The CI/CD pipeline software is connected to the Git repositories through Git 672

webhooks and gets activated on commits (push workflow model) or pull requests in these 673

repositories. 674

• Some CI/CD tools perform CD functions alone for the native platform (e.g., Jenkins X for 675

Kubernetes platform) or for multiple technology stacks (e.g., Spinnaker for multi-cloud 676

deployment). The difficulty with some in this class of tools is that they may lack native 677

tools for completing the CI functions (e.g., tools to test code, build application images, or 678

push them to registry). 679

3.3.3 Designing and Executing the CI/CD pipeline 680

The purpose of creating the CI/CD pipeline is to enable frequent updates to source code, rebuilds, 681

and the automatic deployment of updated modules into the production environment. The key 682

tasks involved are [16]: 683

• Setting up the source code repository: Set up a repository (e.g., GitHub or GitLab) for 684

storing application source code with proper version control. 685

• Build process: Configure and execute the build process for generating the executables (for 686

those portions of the code that need to be updated) using an automated code build tool. 687

• Securing the process: Ensure that the build is free of static and dynamic vulnerabilities 688

through unit testing with SAST and DAST tools. 689

• Creating the deployment environment: Create the environment to deploy the application 690

using an automated deployment tool. 691

• Creating the delivery pipeline: Create a pipeline that will automatically build and deploy 692

the application using an automated pipeline tool. 693

• Test the code and execute the pipeline: Test the code in the pipeline using an automated 694

testing tool, and execute the pipeline to complete the deployment of updated code. 695

To reiterate, the three primary stages of the CI/CD process are build/test, ship/package, and 696

deploy. The following features transform this into a pipeline: 697

• When an update is made to the source code for a service, the code changes pushed to the 698

source code repository trigger the code building tool. 699

• The code building tool, which is usually an IDE, is often integrated with security testing 700

tools (e.g., static vulnerability analysis tool) to facilitate the generation of secure compiled 701

code artifacts, thus integrating security into the CI pipeline. 702

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

• The generation of compiled code artifacts in code building tools triggers the 703

shipping/package tool, which may be integrated with its own set of tools (e.g., dynamic 704

vulnerability analysis or dynamic penetration testing tools, software composition analysis 705

tools for identifying vulnerabilities in the attached libraries) and also creates the 706

configuration parameters relevant to the deployment environment. 707

• The output of the shipping/packaging tool is then automatically fed to the CD tool, which 708

then deploys the package into the desired environment (e.g., staging, production etc.) [17]. 709

The workflow of the CI/CD pipeline should not create the impression that there is no human 710

element involved. The following teams provide input into the CI/CD pipeline [18]: 711

• Development team incorporates new features into the application 712

• Build or integration team creates new CI pipelines and introduces new CI tools and testing 713

tools 714

• Security team conducts audits and patching 715

• Infrastructure team creates, maintains, and upgrades the infrastructure 716

• QA team develops integration test cases 717

• Deployment team/release team creates pipelines and packages for various environments 718

(UAT/PreProd/Prod) and performs configuration and provisioning appropriate for these 719

environments 720

Some of the many activities performed by these teams include the customization, update, and 721

enhancement of the tools employed in the CI/CD pipeline (e.g., updating the static vulnerability 722

analysis tool with the latest database of known vulnerabilities etc.). Caution should be exercised 723

during manual operations so that they do not block pipelines. Targets for mean time to production 724

should be set up while also mitigating risks, such as insider threats, through the use of “merge 725

requests” and multiple approvers in merge requests. 726

727

Figure 2 - Examples of Tools involved in CI/CD Pipeline Tasks 728

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

This pipeline is designed, maintained, and executed by the post-release team who – in addition to 729

monitoring functions – performs other processes, such as compliance management, backup 730

processes, and asset tracking [19]. 731

732

3.3.4 Strategies for Automation 733

Compared to other models of software development, which involve a linear progression from 734

coding to release, DevOps uses a forward process with a delivery pipeline (i.e., build/secure, 735

ship/package, and release) and a reverse process with a feedback loop (i.e., plan and monitor) that 736

form a recursive workflow. The role of automation in these activities is to improve this workflow. 737

Continuous integration emphasizes testing automation to ensure that the application is not broken 738

whenever new commits are integrated into the main branch. Automation results in the following 739

benefits: 740

• Generation of data regarding software static and runtime flows 741

• Reduction of development and deployment times 742

• Better utilization of the team’s expertise by assigning routine tasks to the tools 743

The following strategies are recommended [22] for automation so as to facilitate better utilization 744

of organizational resources and derive the greatest benefit in terms of an efficient, secure 745

application environment. 746

Choice of Activities to Automate: For example, the following are productive candidates for the 747

automation of testing activities. 748

• Testing of modules whose functions are subject to regulatory compliance (e.g., PCI-DSS, 749

HIPAA, Sarbanes-Oxley) 750

• Tasks that are repetitive with moderate to high frequency 751

• Testing of modules that perform time-sequenced operations, such as message publishers 752

and message subscribers 753

• Testing of workflows (e.g., request tracing) involving transactions that span multiple 754

services 755

• Testing of services that are resource-intensive and likely to constrict operations 756

After choosing the candidates for automation based on the above criteria, the usual risk analysis 757

must be applied to choose a subset that provides an optimum return on investment and maximizes 758

desirable security metrics (e.g., defense in depth). Some recommended strategies include: 759

• Using the cost-benefit ratio in hours saved per year to prioritize which processes to 760

automate [22] 761

• Using key performance indicators (KPI) (e.g., mean time to identify faults or problems, 762

rectify, or recover) as markers to refine the DevSecOps processes [22] 763

• Based on the application, applying different weights to infrastructure services (e.g., 764

authorization and other policies enforcement, monitoring of system states to ensure secure 765

runtime states, network resilience in terms of system availability, latency, etc.) to 766

determine the allocation of resources to DevSecOps processes 767

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

3.3.5 Requirements for Automation Tools in CI/CD Pipelines 768

The main utility of a performance testing tool is to identify the root cause of an issue, such as 769

application response time or a bottleneck operation. The desirable features of this class of tool 770

are: 771

• Easy integration into developers’ CI/CD pipelines, which enables this class of tools to be 772

an integral part of the DevSecOps process rather than one managed by a central platform 773

engineering or site reliability engineering team that could become hamper deployments 774

• An accessible interface that can be used by all participants involved in the software 775

release workflow and have features that analysts/engineers can use to perform insightful 776

analyses into specific datasets 777

• Coverage for a wide variety of back-end infrastructure components in multiple clouds as 778

well as on premises; this abstraction feature is now natively present in container 779

orchestration and resource management platforms (e.g., Kubernetes) or through stacks 780

(e.g., vSphere or OpenStack) 781

The security automation tools for various functions (e.g., static vulnerability analysis, dynamic 782

vulnerability analysis, software composition analysis) used in CI/CD pipelines need to have 783

different interface and alerting/reporting requirements since they have to operate seamlessly 784

depending on the pipeline stage (e.g., build, package, release) at which they are used. These 785

requirements are: 786

• Security automation tools should work with integrated development environment (IDE) 787

tools and help developers prioritize and remediate static vulnerabilities. These capabilities 788

are needed to facilitate developer adoption and improve productivity. 789

• Security automation tools should be flexible to support specific workflows and provide 790

scaling capabilities for security services. 791

• Tools that perform static vulnerability checks at the build phase ensure safe data flows, 792

and those that perform dynamic vulnerability checks ensure safe application states during 793

runtime. 794

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

4 Implementing DevSecOps Primitives for the Reference Platform 795

Various CI/CD pipelines are involved in the reference platform (i.e., microservices-based 796

application with service mesh that provides infrastructure services). Though the reference 797

application is a microservices-based application, the DevSecOps primitives can be applied to 798

monolithic applications as well as applications that are both on-premises and cloud-based (hybrid 799

cloud, single public cloud, and multi-cloud). 800

801

The application architecture consists of other functional elements (in addition to elements for 802

housing executable application code and providing application services) such as for infrastructure 803

resources, runtime policies, and continuous monitoring of the health of the application, which can 804

all be deployed through declarative codes. Hence, separate CI/CD pipelines can be created for all 805

of these code types as well. These five code types will first be considered in the context of the 806

reference platform followed by separate sections that will describe the associated CI/CD 807

pipelines. 808

1. Code types in the reference platform and associated CI/CD pipelines (Section 4.1) 809

2. CI/CD pipeline for application code and application services code (Section 4.2) 810

3. CI/CD pipeline for infrastructure as code (IaC) (Section 4.3) 811

4. CI/CD pipeline for policy as code (Section 4.4) 812

5. CI/CD pipeline for observability as code (Section 4.5) 813

814

Implementation issues for all CI/CD pipelines irrespective of code types will be addressed in the 815

following sections: 816

• Securing the CI/CD pipelines (Section 4.6) 817

• Workflow models in the CI/CD pipelines (Section 4.7) 818

• Security testing in the CI/CD pipelines (Section 4.8) 819

820

This section will also consider the overall benefits of DevSecOps with a subsection on specific 821

advantages in the context of the reference platform and the ability to leverage DevSecOps for 822

continuous authorization to operate (C-ATO) in Sections 4.9 and 4.10, respectively. 823

4.1 Code Types in Reference Platform and Associated CI/CD Pipelines 824

The constituent components of the code types in the reference platform are: 825

1. Microservices-based application component (implemented as a series of containers), 826

which contains the application logic (e.g., interacting with data, performing transactions 827

etc.) and the application code 828

2. Infrastructure component (containing computer, networking, and storage resources) 829

whose constituents can be provisioned using infrastructure as code 830

3. Service mesh component (implemented through a combination of control plane modules 831

and service proxies), which provides application services, enforces policies (e.g., 832

authentication and authorization), and contains application services code and policy as 833

code 834

4. Monitoring component (modules involved in ascertaining the parameters that indicate the 835

health of the application), which performs functions (e.g., log aggregation, the generation 836

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

of metrics, the generation of displays for dashboard, etc.) and contains observability as 837

code 838

The DevSecOps methodology for developing, testing, delivering, and deploying these five types 839

of codes (i.e., application, application services, infrastructure, policy, and monitoring) requires 840

the corresponding pipelines: 841

• CI/CD pipelines for application code and application services code – the former contains 842

the data and application logic for a specific set of business transactions while the latter 843

contains code for all services such as network connections, load balancing, network 844

resilience etc. 845

• CI/CD pipeline for infrastructure as code (IaC) – The process of writing the code for 846

provisioning and configuring the steps of infrastructure resources to automate application 847

deployment in a repeatable and consistent manner [23]. This code is written in a 848

declarative language and – when executed – provisions, de-provisions, and configures the 849

infrastructure for the application that is being deployed. This type of code is like any other 850

code found in an application’s microservice except that it provides an infrastructure 851

service (e.g., provisioning a server) rather than a transaction service (e.g., payment 852

processing for an online retail application). 853

• CI/CD pipeline for policy as code – Describes many policies, including security policies, 854

as executable modules [24]. One example is the authorization policy, the code for which 855

contains verbs or artifacts specific to the policy (e.g., allow, deny etc.) and to the domain 856

where it applies (e.g., REST API with verbs such as method [GET, PUT, etc.], path, etc.). 857

This code can be written in a special-purpose policy language, such as Rego or WASM, 858

or languages used in regular applications, such as Go. This code may have some overlap 859

with the configuration code of IaC. However, for implementing policies associated with 860

critical security services that are specific to the application domain, a separate policy as 861

code that resides in the policy enforcement points (PEPs) of the reference platform is 862

required. 863

• CI/CD pipeline for observability as code – The ability to infer a system’s internal state 864

and provide actionable insights into when and, more importantly, why errors occur within 865

a system. It is a full-stack observability that includes monitoring and analytics, as well as 866

offers key insights into the overall performance of applications and the systems hosting 867

them. In the context of the reference platform, observability as code is the portion of the 868

code that creates agencies in proxies and creates functionality for gathering three types of 869

data (i.e., logs, traces, and telemetry) from microservices applications [25]. This type of 870

code also supplies or transfers data to the external tools (e.g., log aggregation tool that 871

aggregates log data from individual microservices, provides analysis of tracing data for 872

bottleneck services, generates metrics that reflect the application health from telemetry 873

data, etc.). Brief descriptions of the three functions enabled by observability as code are: 874

1. Logging captures detailed error messages, as well as debugs logs and stack 875

traces for troubleshooting. 876

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

2. Tracing follows application requests as they wind through multiple 877

microservices to complete a transaction in order to identify an issue or 878

performance bottleneck in a distributed or microservices-based ecosystem. 879

3. Monitoring, or metrics, gathers telemetry data from applications and 880

services. 881

The policy and observability code types span the following components of the service mesh. 882

• Proxies (ingress, sidecar, and egress): These house encoding policies related to session 883

establishment, routing, authentication, and authorization functions. 884

• Control plane of the service mesh: This houses code for relaying telemetry information 885

from services captured and sent by proxies to specialized monitoring tools, authentication 886

certificate generation and maintenance, updating policies in the proxies, monitoring 887

overall configuration in the service orchestration platform for generating new proxies, and 888

deleting obsolete proxies associated with discontinued microservices. 889

• External modules: These house modules that perform specialized functions at the 890

application and enterprise levels (e.g., such as the centralized authorization or entitlement 891

server, the centralized logger, monitoring/alerting server status through dashboards, etc.) 892

and build a comprehensive view of the application status. These modules are called by 893

code from the proxies or the control plane. 894

4.2 CI/CD Pipeline for Application Code and Application Services Code 895

Application code and application services code reside in the container orchestration and resource 896

management platform, and the CI/CD software that implements the workflows associated with it 897

usually reside in the same platform. This pipeline should be protected using the steps outlined in 898

Section 4.6, and the application code under the control of this pipeline should be subject to the 899

security testing described in Section 4.8. Additionally, the orchestration platform on which the 900

application resides should itself be protected using a runtime security tool (e.g., Falco) [32] that 901

can read OS kernel logs, container logs, and platform logs in real-time and process them against a 902

threat detection rules engine to alert users of malicious behavior (e.g., creation of a privileged 903

container, reading of a sensitive file by an unauthorized user, etc.). They usually come with a set 904

of default (predefined) rules over which custom rules can be added. Installing them on the 905

platform spins up agents for each node in the cluster, which can monitor the containers running in 906

the various pods of that node. The advantage of this type of tool is that it complements the existing 907

platform’s native security measures, such as access control models and pod security policies, that 908

prevent violations of security by actually detecting them when they occur [32]. 909

4.3. CI/CD Pipeline for Infrastructure as Code 910

Infrastructure as code (IaC) involves codifying all software deployment tasks (allocation of type 911

of servers, such as bare metal, VMs or containers, resource content of servers) and the 912

configuration of these servers and their networks. The software containing this code type is also 913

called a resource manager or deployment manager. In other words, IaC software automates 914

management of the whole IT infrastructure life cycle (provisioning and de-provisioning of 915

resources) and enables a programmable infrastructure. The integration of this software as part of 916

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

the CI/CD pipeline not only results in agile deployment and maintenance but also in a robust 917

application platform that is secure and meets performance needs. 918

919

The conventional approach to allocating infrastructure for applications consists of initially 920

provisioning compute and networking resources with configuration parameters and ongoing tasks 921

such as patch management (e.g., OS and libraries), establishing conformity to compliance 922

regulations (e.g., data privacy), and making drift correction (where the current configuration no 923

longer provides the intended operational state). 924

925

Infrastructure as code (IaC) is a declarative style of code that encodes computer instructions that 926

encapsulate the parameters necessary to deploy virtual infrastructure on a public cloud service or 927

private data center via a service’s management APIs [33]. Depending on the particular IaC tool, 928

this language can either be a scripting language (e.g., Go, JavaScript, Python, TypeScript, etc.) or 929

a proprietary configuration language (e.g., HCL) that may or may not be compatible with 930

standardized languages (e.g., JSON). The basic unit of these instructions is called “configuration” 931

and tells the system how to provision and manage infrastructure (whether that is an individual 932

compute instance or a complete server, such as physical servers or virtual machines), containers, 933

storage, network connections, connection topology, and load balancers. [34]. In some cases, the 934

infrastructure may be short-lived or ephemeral, and the lifespan of the infrastructure (whether 935

immutable or mutable) does not warrant continued configuration management. Provisioning 936

could be tied to individual commits of application code using tools that can connect application 937

code and infrastructure code in way that is logical, expressive, and familiar to development and 938

operations teams, where application code increasingly defines the infrastructure resource 939

requirements for a cloud application [35]. 940

941

4.3.1 Comparison of Configuration and Infrastructure 942

943

Infrastructure is often confused with configuration [34], which maintains computer systems, 944

software, dependencies, and settings in a desired, consistent state. For example, putting a newly 945

purchased server onto a rack and connecting it to the switches so that it is connected to the existing 946

networks (or launching a new virtual machine and assigning network interfaces to it) belongs to 947

the definition of “infrastructure.” In contrast, after the server is launched, installing an HTTP 948

server and configuring it belongs to configuration management. In physical data centers, specific 949

teams purchase servers, install servers, and connect networking cables with the underlying 950

infrastructure in mind. 951

952

4.4 CI/CD Pipeline for Policy as Code 953

Policy as code involves codifying all policies and running them as part of the CI/CD pipeline so 954

that they become an integral part of the application runtime. Examples of policy categories 955

include authorization policies, networking policies, and implementation artifact policies (e.g., 956

container policies). Policy management capabilities in a typical “policy as code software” may 957

come with a set of predefined policy categories and policies and also support the definition of 958

new policy categories and associated policies by providing policy templates [36]. 959

Some examples of policy categories and associated policies are given in 960

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

Table 1 below. 961

Table 1: Policy Categories and Example Policies 962

963

Policy Category

Example Policies

Networking policies and zero trust policies

• Blocking designated ports

• Designating ingress host names

Implementation artifact policies (e.g.,

container policies)

• Ensuring that containers do not run as root

• Blocking privilege escalation for

containers

Storage policies

• Set persistent volume sizes

• Set persistent volume reclaim policies

Access control policies

• Ensure that policies cover all data objects

• Ensure that there are no policy conflicts

Supply chain policies

• Allow only approved container registries

• Allow only certified libraries

The policies defined in the “policy as code software” may translate into the following in the 964

application infrastructure: 965

• Runtime configuration parameters 966

• Policy-enforcing executable (e.g., WASM in service proxies) 967

• Triggers for calling an external policy decision module (e.g., calling an external 968

authorization server for an allow/deny decision based on the evaluation of access control 969

policies relevant to the current access request) 970

4.5 CI/CD Pipeline for Observability as Code 971

Observability as code deploys a monitoring agent in each of the application’s service components 972

to collect the three types of data (described in Section 4.1), send them to specialized tools that 973

correlate them, perform analysis, and display the analyzed consolidated data on dashboards to 974

present an overall application-level picture [20]. An example of such consolidated data are log 975

patterns, which provide a view of log data that is presented after the log data are filtered using 976

some criterion (e.g., a service or an event). The data are grouped into clusters based on common 977

patterns (e.g., based on timestamp or range of IP addresses) for easy interpretation. Unusual 978

occurrences are identified, and those findings can then be used to steer and accelerate further 979

investigation [21]. 980

4.6 Securing the CI/CD Pipeline 981

There are some common implementation issues to be addressed for CI/CD pipelines irrespective 982

of code type. The first – security – involves securing the artifacts, including hardening the servers 983

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

and nodes used for generating the build and establishing the authenticity of the build components. 984

Securing the processes involves the assignment of roles for operating the build tasks. Automation 985

tools (e.g., Git Secrets) are available for this purpose. The following security tasks should be 986

performed when securing the CI pipeline [26]: 987

● Authentication and validation in code and build repositories 988

● Logging all activities associated with code and build updates 989

● Sending build reports to developers and stopping everything if a build fails 990

● Sending build reports to security and stopping everything if the audit or validation fails 991

Securing the CD pipeline involves the following: 992

• Multi-party signing with TUF and in-toto to sign each CD pipeline phase 993

• Signing the final artifacts with multiple phase keys to ensure that no one bypasses the 994

pipeline 995

4.7 Workflow Models in CI/CD Pipelines 996

The next common issue involves workflow models. All CI/CD pipelines can have two types of 997

workflow models, which depend on the automated tools that are deployed as part of the pipeline. 998

1. Push-based model 999

2. Pull-based model 1000

1001

In the CI/CD tools that support the push-based model, changes made in one stage or phase of the 1002

pipeline trigger changes in the subsequent stages and phases. For example, through a series of 1003

encoded scripts, the new builds in the CI system trigger changes to the CD portion of the pipeline 1004

and thus change the deployment infrastructure (e.g., Kubernetes cluster). The security downside 1005

of using the CI system as the basis for change in deployments is the possibility of exposing 1006

credentials outside of the deployment environment in spite of best efforts to secure the CI scripts, 1007

which operate outside of the trusted domain of the deployment infrastructure. Since CD tools 1008

have the keys to production systems, push-based models are rendered insecure. A pull-based 1009

model, which typically uses a GitOps repository for storing the source code and builds, is 1010

therefore highly recommended. 1011

In a pull-based workflow model, an operator that pertains to the deployment environment (e.g., 1012

Kubernetes Operator, Flux, ArgoCD) pulls new images from inside of the environment as soon as 1013

the operator observes that a new image has been pushed to the registry. The new image is pulled 1014

from the registry, the deployment manifest is automatically updated, and the new image is 1015

deployed in the environment (e.g., cluster). Thus, the convergence of the actual deployment 1016

infrastructure state with the state declaratively described in the Git deployment repository is 1017

achieved. Additionally, the deployment environment credentials (e.g., cluster credentials) are not 1018

exposed outside of the production environment. 1019

4.7.1 GitsOps Workflow Model for CI/CD – A Pull-based Model 1020

The GitOps workflow model is an improvement on the CI/CD pipeline (for the delivery portion 1021

of the pipeline), which uses a pull-based workflow model instead of the push-based model 1022

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

supported by many CI/CD tools. In this model, the CI portion of the pipeline is unchanged since 1023

the CI engine (e.g., Jenkins, GitLab CI) is still used for creating builds for the changed code, 1024

regression testing, and integrating/merging with the main source code in the relevant repositories, 1025

though it is not used to trigger continuous delivery (push updates directly) in the pipeline. 1026

Instead, a separate GitOps operator manages the deployment based on updates to the main (trunk) 1027

code. 1028

An operator (eg. Flux, ArgoCD) is an actor managed by an orchestration platform and can 1029

inherit the cluster’s configuration, security, and availability. The use of this actor improves 1030

security because an agent that lives inside of the cluster listens for updates to all code and image 1031

repositories that it is allowed to access and pulls images and configuration updates into the 1032

cluster. The pull approach used by the agent has the following security features: 1033

• ONLY carry out operations permitted by authorization policies defined in the 1034

orchestration platform; trust is shared with the cluster and not managed separately 1035

• Bind natively to all orchestration platform objects, and know whether operations have 1036

completed or need to be retried 1037

4.8 Security Testing – Common Requirement for CI/CD Pipelines for All Code Types 1038

The last common issue is security testing. Whatever the code type is, the CI/CD pipelines of 1039

DevSecOps for microservices-based infrastructure with service mesh should include application 1040

security testing (AST) enabled by either automated tools or offered as a service. These tools 1041

analyze and test applications for security vulnerabilities. According to Gartner, there are four 1042

main AST technologies [27]: 1043

1. Static AST (SAST) tools – analyze an application’s source, bytecode, or binary code for 1044

security vulnerabilities, typically at the programming and/or testing software life cycle 1045

(SLC) phases. Specifically, this technology involves techniques that look through the 1046

application in a commit and analyze its dependencies [28]. If any dependencies contain 1047

issues or known security vulnerabilities, a commit will be marked as insecure and will not 1048

be allowed to proceed to deployment. 1049

2. Dynamic AST (DAST) tools – analyze applications in their dynamic, running state during 1050

testing or operational phases. They simulate attacks against an application (typically web-1051

enabled applications, services, and APIs), analyze the application’s reactions, and 1052

determine whether it is vulnerable. In particular, DAST tools go one step further than 1053

SAST and spin up a copy of the production environment inside the CI job in order to scan 1054

the resulting containers and executables [28]. The dynamic aspect helps the system catch 1055

dependencies that are being loaded at launch time, such as those would not be caught by 1056

SAST. 1057

3. Interactive AST (IAST) tools – combine elements of DAST with the instrumentation of 1058

the application under test. They are typically implemented as an agent within the test 1059

runtime environment (e.g., instrumenting the Java Virtual Machine [JVM] or .NET CLR) 1060

that observes operations or identifies and attacks vulnerabilities. 1061

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

4. Software composition analysis (SCA) tools – are used to identify open-source and third-1062

party components in use in an application, their known security vulnerabilities, and 1063

typically adversarial license restrictions. 1064

4.8.1 Functional and Coverage Requirements for AST tools 1065

In general, the overall metrics that testing tools (including the specific class of AST tools) should 1066

satisfy are [29]: 1067

• Increase the quality of application releases 1068

• Integrate with the tools that developers are already using 1069

• Be as few test tools as possible but provide the necessary coverage risk 1070

• Lower-level unit tests at the API and microservices level should have sufficient visibility 1071

to determine coverage 1072

• Higher-level UI/UX and system tests 1073

• Have deep code analysis capabilities to detect runtime flaws 1074

• Increase the speed at which the releases can be done 1075

• Be cost-efficient 1076

The functional requirements for AST tools in particular are that perform the following types of 1077

scans [30]: 1078

• Vulnerability scans: Probe applications for security weaknesses that could expose them 1079

to attacks 1080

• Container image scans: Analyze the contents and build process of a container image in 1081

order to detect security issues, vulnerabilities, or deficient practices 1082

• Regulatory/compliance scans: Assess adherence to specific compliance requirements 1083

The vulnerability scans are to be performed whenever the code in GitHub is revised to ensure that 1084

the current revision does not contain any vulnerable dependencies [31]. 1085

The desirable features of AST tools and/or services, along with techniques for behavioral 1086

analysis, are [27]: 1087

• Analyze source, byte, or binary code 1088

• Observe the behavior of apps to identify coding, design, packaging, deployment, and 1089

runtime conditions that introduce security vulnerabilities 1090

Scanning application code for security vulnerabilities and misconfiguration as part of CI/CD 1091

pipeline tasks should involve the following artifacts: 1092

• Container images should be scanned for vulnerabilities. 1093

• Filesystems should be scanned for both vulnerabilities and misconfigurations. 1094

• Git repositories should be scanned for both vulnerabilities and misconfigurations. 1095

Container images include OS packages (e.g., Alpine, UBI, RHEL, CentOS, etc.) and language-1096

specific packages (e.g., Bundler, Composer, npm, yarn, etc.) 1097

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

Scanning Infrastructure-as-code for security vulnerabilities reduces the operations workload by 1098

preventing those vulnerabilities from making it to production, although it cannot replace runtime 1099

security since there will always be the risk of drift. The infrastructure-as-code files can be found 1100

in the following: 1101

• In the container orchestration platform itself to facilitate deployments (e.g., Kubernetes 1102

YAML infrastructure-as-code files) 1103

• In the dedicated infrastructure-as-code files found as part of CI/CD pipeline software 1104

(e.g., HashiCorp Terraform infrastructure-as-code files, AWS CloudFormation 1105

infrastructure-as-code files) 1106

Both application services code, policy-as-code, and observability-as-code files can be found in 1107

the data plane and control plane components of the dedicated application services component 1108

(e.g., service mesh) and should be scanned for both security vulnerabilities (e.g., information 1109

leakage in authorization policies) and misconfiguration. 1110

4.9 Benefits of DevSecOps Primitives to Application Security in the Service Mesh 1111

The benefits of DevSecOps include [37]: 1112

• Streamlined software delivery and deployment processes 1113

• Reduction of the number of silos between IT groups 1114

• Reduction of attack surfaces by implementing zero trust down to the container level 1115

• Introduction of moving target defenses by threating containers as cattle versus pet and 1116

going back to an immutable state 1117

• Reduction of lateral movement capabilities by leveraging zero trust and continuous 1118

monitoring with modern behavior prevention capabilities 1119

• Increased communications between team members and other stakeholders 1120

• Faster incorporation of feedback, resulting in quicker software improvements 1121

• Automation of repetitive, manual tasks, leading to increased efficiency 1122

• Less downtime and faster time to market 1123

• Infrastructure for continuous software development and delivery (as well as admission 1124

controller and the use of OPA to add guard rails to the runtime) 1125

As already stated, the primary goal of DevSecOps is to have a platform that ensures runtime 1126

security of microservices-based applications. Runtime security is assured through the following: 1127

(a) Validation of every request: Every request from a user or client application (service) is 1128

authenticated and authorized (using mechanisms such as Open Policy Agent [OPA] or any 1129

external authorization engine), and admission controllers that are integral parts of the 1130

platform. While authorization engines provide application domain-specific policy 1131

enforcement, admission controllers provide platform-specific policy relating to end-point 1132

objects of a specific platform (e.g., pods, deployment, namespaces). Specifically, 1133

admission controllers mutate and validate. Mutating admission controllers parse each 1134

request and make changes to the request (mutate) before forwarding it down the chain. An 1135

example is setting default values for specifications that are not set by a user in the request 1136

so as to ensure that workloads running on the cluster are uniform and follow a particular 1137

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

standard defined by the cluster administrator. Another example is adding a specific 1138

resource limit for the pod (if the resource limits are not set for that pod) and then 1139

forwarding it down the chain (mutate the request by adding this field if it is not present in 1140

the request). By doing so, all of the pods in the cluster will always have a resource limit set 1141

according to a specification unless explicitly stated. Validating admission controllers reject 1142

requests that do not follow a particular specification. For example, none of the pod requests 1143

can have security context set to run as root user [38]. 1144

(b) Feedback mechanisms: 1145

• Provide feedback loops to the application hosting platform (e.g., a notification to kill a 1146

pod that contains a malicious container) 1147

• Provide proactive dynamic security by monitoring application configuration (e.g., 1148

monitoring new pods/containers introduced into the application and generating and 1149

injecting proxies to take care of their secure communication needs) 1150

• Enable several security assertions regarding the application: non-bypassable (policies 1151

always enforced under all usage scenarios), trusted and untrusted portions of the 1152

overall application code, absence of credential and privilege leaks, trusted 1153

communication paths, and secure state transitions 1154

• Enable assertions regarding performance parameters (e.g., network resilience 1155

parameters, such as continued operation under failures, redundancy and recoverability 1156

features) 1157

4.10 Leveraging DevSecOps for Continuous Authorization to Operate (c-ATO) 1158

In the reference platform, the runtime status or execution state of the entire application system is 1159

due to a combination of executions of infrastructure code (e.g., networking routes for inter-1160

service communication, resources provisioning code), policy code (e.g., code that specifies 1161

authentication and authorization policies), and session management code (e.g., code that 1162

establishes an mTLS session, code that generates JWT tokens) as revealed by the execution of 1163

observability as code. The observability as code of the service mesh relays the output from the 1164

execution of infrastructure, policy, and session management codes during runtime to various 1165

monitoring tools that generate applicable metrics and log aggregation tools and tracing tools, 1166

which in turn relay their output to a centralized dashboard. The analytics that are integral to the 1167

output of these tools enable system administrators to obtain a comprehensive global view of the 1168

runtime status of the entire application system. It is the runtime performance of a DevSecOps 1169

platform enabled through continuous monitoring with zero trust design features that provides all 1170

of the necessary security assurance for cloud-native applications. 1171

The activities within the DevSecOps pipelines within the service mesh context that enable 1172

continuous ATO are: 1173

• Checking for compliant code: Checking whether the updated infrastructure, policy, 1174

session management, and observability codes are compliant with the chosen risk 1175

management framework (e.g., NIST Risk Management Framework) for the enterprise. 1176

This will involve tools with risk assessment features, such as the capability to generate 1177

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

actionable tasks, specify code-level guidance, and test plans for verifying compliance 1178

[31]. 1179

• Generating a dashboard that displays the runtime status: Using the runtime output of 1180

observability code for generating the dashboard that displays the runtime status of the 1181

overall application 1182

Risk assessment tools should provide complete traceability for all of the artifacts displayed in the 1183

dashboard, as well as the reporting capabilities needed for continuous ATO. 1184

Dashboard generation tools should enable system administrators to analyze macro-level features, 1185

as well as dynamically change the composition of the artifacts to be displayed based on the 1186

evolving system and consumer needs of the environment in which the application operates. 1187

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

5 Summary and Conclusion 1188

This document provides a comprehensive guidance for the implementation of DevSecOps 1189

primitives for a reference platform hosting a cloud-native application. It includes an overview of 1190

the reference platform and describes the basic DevSecOps primitives (i.e., CI/CD pipelines), its 1191

building blocks, the design and execution of the pipelines, and the role of automation in the 1192

efficient execution of workflows in CI/CD pipelines. 1193

1194

The architecture of the reference platform, in addition to the application code and the code for 1195

providing application services, consists of functional elements for infrastructure, runtime policies, 1196

and continuous monitoring of the health of the application, the last three of which can be 1197

deployed through declarative codes with separate CI/CD pipelines types. The runtime behaviors 1198

of these codes, the benefits of the implementation for high-assurance security, and the use of the 1199

artifacts within the pipelines for providing a continuous authority to operate (c-ATO) using risk 1200

management tools and dashboard metrics are also described. 1201

1202

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

References 1203

[1] Garrison J, Nova K (2017) Cloud Native Infrastructure (oreilly.com). Available at 1204

https://www.oreilly.com/library/view/cloud-native-infrastructure/9781491984291/ 1205

1206

[2] Chaillan NM (2020) DoD Enterprise DevSecOps Initiative and Platform One. Available 1207

at https://software.af.mil/dsop/documents 1208

[3] Chandramouli R (2019) Security Strategies for Microservices-based Application Systems. 1209

(National Institute of Standards and Technology, Gaithersburg, MD), NIST Special 1210

Publication (SP) 800-204. https://doi.org/10.6028/NIST.SP.800-204 1211

1212

[4] Chandramouli R, Butcher Z (2020) Building Secure Microservices-based Applications 1213

Using Service-Mesh Architecture. (National Institute of Standards and Technology, 1214

Gaithersburg, MD), NIST Special Publication (SP) 800-204A. 1215

https://doi.org/10.6028/NIST.SP.800-204A 1216

1217

[5] Chandramouli R, Butcher Z, Aradhna C (2021) (National Institute of Standards and 1218

Technology, Gaithersburg, MD), NIST Special Publication (SP) 800-204B. 1219

https://doi.org/10.6028/NIST.SP.800-204B 1220

1221

[6] Agarwal G (2020) How to Manage Microservices on Kubernetes With Istio. Available at 1222

https://medium.com/better-programming/how-to-manage-microservices-on-kubernetes-1223

with-istio-c25e97a60a59 1224

1225

[7] Kubernetes Advocate (2021) Managing Microservices With Istio Service Mesh in 1226

Kubernetes. Available at https://medium.com/avmconsulting-blog/managing-1227

microservices-with-istio-service-mesh-in-kubernetes-36e1fda81757 1228

1229

[8] Ramakani A (2020) Kong API Gateway – From Zero to Production. Available at 1230

https://medium.com/swlh/kong-api-gateway-zero-to-production-5b8431495ee 1231

1232

[9] Srinath K (2020) DevSecOps – Baking Security into Development Process. Available at 1233

https://medium.com/faun/devsecops-baking-security-into-development-process-1234

9579418ad9a7 1235

1236

[10] Rubinstein D (2020) AppSec vs. DevSecOps, and what that means for developers. 1237

Available at https://sdtimes.com/security/appsec-vs-devsecops-and-what-that-means-for-1238

developers/ 1239

1240

[11] Red Hat (2021) What is DevSecOps? Available at 1241

https://www.redhat.com/en/topics/devops/what-is-devsecops 1242

1243

[12] Red Hat (2021) What is CI/CD? Available at 1244

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

https://www.redhat.com/en/topics/devops/what-is-ci-1245

cd#:~:text=CI%2FCD%20is%20a%20method,continuous%20delivery%2C%20and%20c1246

ontinuous%20deployment 1247

1248

[13] Ali M (2018) Continuous Release Practices are evolving, Here is our story. Available at 1249

https://medium.com/the-telegraph-engineering/continuous-release-practices-are-evolving-1250

here-is-our-story-2a4d164e9cac 1251

1252

[14] Pittet S (2021) Continuous integration vs. continuous delivery vs. continuous deployment. 1253

Available at https://www.atlassian.com/continuous-delivery/principles/continuous-1254

integration-vs-delivery-vs-deployment 1255

1256

[15] Wuestkamp K (2020) K8s–native Jenkins-X and Tekton Pipelines. Available at 1257

https://itnext.io/k8s-native-jenkins-x-and-tekton-pipelines-e2b5a61a1d22 1258

1259

[16] aws.amazon.com (2021) Create Continuous Delivery Pipeline. Available at 1260

https://aws.amazon.com/getting-started/hands-on/create-continuous-delivery-1261

pipeline/?trk=gs_card 1262

1263

[17] aws.amazon.com (2021) Setting up a CI/CD pipeline by integrating Jenkins with AWS 1264

CodeBuild and AWS CodeDeploy. https://aws.amazon.com/blogs/devops/setting-up-a-ci-1265

cd-pipeline-by-integrating-jenkins-with-aws-codebuild-and-aws-codedeploy/ 1266

[18] mylocaldevstack.pl (2019) The future of DevOps – Assembly Lines. Available at 1267

https://medium.com/@mylocaldevstack/the-future-of-devops-assembly-lines-1268

40227546d750 1269

[19] Socher R (2020) Best Terraform Tutorial Guides: An Overview. Available at 1270

https://faun.pub/best-terraform-tutorial-guides-an-overview-65a6fcee0a24 1271

[20] Pariseau B (2020) Mendix dumps cluttered DevOps monitoring tools for Datadog. 1272

Available at https://searchitoperations.techtarget.com/news/252497697/Mendix-dumps-1273

cluttered-DevOps-monitoring-tools-for-Datadog?track=NL-1274

1841&ad=938089&asrc=EM_NLN_151848158&utm_medium=EM&utm_source=NLN1275

&utm_campaign=20210315_Kong+Mesh%27s+built-1276

in+Open+Policy+Agent+simplifies+IT+security+management 1277

[21] Boutet R (2018) Log Patterns: Automatically cluster your logs for faster investigation 1278

(datadoghq.com). Available at https://www.datadoghq.com/blog/log-patterns/ 1279

[22] Soni A (2020) GitOps: The Next Big Thing for DevOps and Automation! Available at 1280

https://medium.com/searce/gitops-the-next-big-thing-for-devops-and-automation-1281

2a9597e51559 1282

[23] Fallon A (2020) Understand the role of infrastructure as code in DevOps 1283

(TechTarget.com). Available at 1284

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

https://searchitoperations.techtarget.com/feature/Understand-the-role-of-infrastructure-as-1285

code-in-1286

DevOps?utm_campaign=20210809_The+next+DevSecOps+challenge%3A+People&utm_1287

medium=EM&utm_source=NLN&track=NL-1288

1841&ad=939963&asrc=EM_NLN_174809933 1289

[24] Ahmed M (2020) Introducing Policy As Code: The Open Policy Agent (OPA). Available 1290

at https://www.magalix.com/blog/introducing-policy-as-code-the-open-policy-agent-opa 1291

[25] Singh P (2021) Tackle Kubernetes observability with the right metrics. Available at 1292

https://searchitoperations.techtarget.com/tip/Tackle-Kubernetes-observability-with-the-1293

right-metrics?track=NL-1294

1841&ad=938191&asrc=EM_NLN_153034984&utm_medium=EM&utm_source=NLN1295

&utm_campaign=20210322_DevSecOps+leaves+Excel+in+the+dust 1296

1297

[26] Devopscurry (2021) Securing your CI/CD pipelines with DevSecOps in 2021. Available at 1298

https://medium.com/devopscurry/securing-your-ci-cd-pipelines-with-devsecops-in-2021-1299

1a6a6e34f2e7 1300

[27] Gartner (2021) Gardner D, Horvath M, Zumerle D Magic Quadrant for Application 1301

Security Testing. Available at https://www.gartner.com/doc/reprints?id=1-1302

262TXQZV&ct=210518&st=sb 1303

[28] Lewkowicz J (2021) A guide to automated testing providers. Available at 1304

https://sdtimes.com/test/a-guide-to-automated-testing-providers/ 1305

[29] Lewkowicz J (2021) Automated testing is a must in CI/CE pipelines. Available at 1306

https://sdtimes.com/test/automated-testing-is-a-must-in-ci-cd-pipelines/ 1307

1308

[30] Angel R (2020) How to be a CI/CD Engineer. Available at https://circleci.com/blog/how-1309

to-be-a-cicd-engineer/ 1310

1311

[31] Alexey K (2020) Ultimate guide to CI/CD security and DevSecOps. Available at 1312

https://circleci.com/blog/security-best-practices-for-ci-cd/ 1313

[32] Agarwal G (2020) Kubernetes Security With Falco. Available at 1314

https://betterprogramming.pub/kubernetes-security-with-falco-2eb060d3ae7d 1315

[33] Marko K (2021) Terraform cheat sheet: Notable commands, HCL and more. Available at 1316

https://searchitoperations.techtarget.com/tip/Terraform-cheat-sheet-Notable-commands-1317

HCL-and-1318

more?utm_campaign=20210726_Infrastructure+as+code+still+a+big+security+buzz&utm1319

_medium=EM&utm_source=NLN&track=NL-1320

1841&ad=939808&asrc=EM_NLN_172629823 1321

NIST SP 800-204C (DRAFT) DEVSECOPS FOR A MICROSERVICES-BASED

APPLICATION WITH SERVICE MESH

[34] Guo T (2021) On DevOps 8: Infrastructure as Code: Introduction, Best Practices and 1322

How to choose the Right tool. Available at https://medium.com/4th-coffee/on-devops-8-1323

infrastructure-as-code-introduction-best-practices-and-choosing-the-right-tool-1324

2c8f46d1f34 1325

[35] Pulumi.com (2020) Delivering Cloud Native Infrastructure as Code. Available at 1326

https://cdn2.hubspot.net/hubfs/4429525/Content/Pulumi-Delivering-CNI-as-Code.pdf 1327

1328

[36] Chong T (2021) Product In-Depth: Instantly Secure your Cloud Infrastructure with Policy-as-1329

Code. Available at https://www.magalix.com/blog/instantly-secure-your-cloud-1330

infrastructure-with-policy-as-1331

code?utm_medium=email&_hsmi=123795689&_hsenc=p2ANqtz-1332

HbHGaLCOrCV_WXjrKE-JgWUWUX9Tg1s9ot5HSox2Ts1rgAhwquoZkBKqZYo-1333

OcHJmlUHG8eFhC5Tfu7MW_Dn3i-1334

g3Ng&utm_content=123795689&utm_source=hs_email 1335

[37] Sheldon R (2021) Top 30 DevOps interview questions and answers for 2021. Available at 1336

https://whatis.techtarget.com/feature/Top-30-DevOps-interview-questions-and-1337

answers?utm_campaign=20210712_New+Kubernetes+use+case%253A+Hacking&utm_1338

medium=EM&utm_source=NLN&track=NL-1339

1841&ad=939635&asrc=EM_NLN_170130891 1340

[38] Prasad A (2020) Kubernetes Admission Controllers: Request Interceptors. Available at 1341

https://medium.com/cloudlego/kubernetes-admission-controllers-request-interceptors-1342

47a9b12c5303 1343