AWS CI/CD Pipeline Security & Governance
Secure AWS CI/CD pipelines with CodePipeline, CodeBuild, and CodeDeploy policies, including secrets management and compliance automation.
📋 Prerequisites
- Expert knowledge of AWS CI/CD services (CodeCommit, CodeBuild, CodePipeline, CodeDeploy).
- Advanced proficiency with IAM, including instance profiles, service roles, and trust policies.
- Strong experience with Infrastructure as Code (Terraform/CloudFormation) and build specification files (`buildspec.yml`).
- Familiarity with container security scanning and static analysis (SAST/SCA) tools.
💡 The Pipeline as a Security Control Plane
A CI/CD pipeline is more than just an automation tool; it is a critical piece of security infrastructure. It has privileged access to build artifacts, test environments, and production systems. Securing the pipeline itself with least-privilege roles and embedding automated security checks within its stages ("Shifting Left") transforms it from a potential attack vector into a powerful, automated governance and security enforcement engine.
What You'll Learn
🏷️ Topics Covered
Securing the Pipeline: Least-Privilege IAM Roles
Each AWS service participating in your pipeline must operate under its own fine-grained IAM role. A common anti-pattern is using a single, overly permissive role for the entire pipeline.
📜 JSON: Least-Privilege Role for a CodeBuild Project
This IAM role policy allows a CodeBuild project to perform its core functions and push a container image to a *specific* ECR repository, but nothing more.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CodeBuildCorePermissions",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:us-east-1:123456789012:log-group:/aws/codebuild/my-app-build",
"arn:aws:logs:us-east-1:123456789012:log-group:/aws/codebuild/my-app-build:*"
]
},
{
"Sid": "S3ArtifactsAccess",
"Effect": "Allow",
"Action": [ "s3:PutObject", "s3:GetObject" ],
"Resource": "arn:aws:s3:::my-pipeline-artifacts-bucket/*"
},
{
"Sid": "ECRImagePush",
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage"
],
"Resource": "arn:aws:ecr:us-east-1:123456789012:repository/my-app-repo"
}
]
}"Shifting Left": Integrating Multiple Security Scans
Your CodeBuild stage is the perfect place to automate security checks. By adding these steps to your `buildspec.yml`, you fail the build early if vulnerabilities are detected, preventing insecure code from ever being deployed.
⚙️ YAML: `buildspec.yml` with Software Composition Analysis (SCA)
This phase uses `npm audit` to find and fail the build on high-severity vulnerabilities in Node.js dependencies.
version: 0.2
phases:
install:
runtime-versions:
nodejs: 18
build:
commands:
- echo "Installing dependencies"
- npm install
post_build:
commands:
- echo "Running SCA scan with npm audit"
# Fails the build if any high or critical severity vulnerabilities are found in dependencies
- npm audit --audit-level=high
🛡️ YAML: `buildspec.yml` with IaC Scanning (CFN Guard)
This phase validates CloudFormation templates against a central repository of compliance rules before attempting to deploy them.
version: 0.2
phases:
install:
commands:
- curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/aws-cloudformation/cloudformation-guard/main/install-guard.sh | sh
build:
commands:
- echo "Validating CloudFormation templates"
# Validate all templates in the /iac directory against all rules in the /rules directory
# --show-summary fail ensures the build fails if any rule violations are found
- /root/.guard/bin/cfn-guard validate --data iac/ --rules rules/ --show-summary fail
🐳 YAML: `buildspec.yml` with Container Scanning (Trivy)
This phase builds a Docker image and then uses the open-source tool Trivy to scan it for OS-level vulnerabilities before pushing it to ECR.
version: 0.2
phases:
install:
commands:
- curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
pre_build:
commands:
- aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
build:
commands:
- docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
- docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
post_build:
commands:
- echo "Scanning image with Trivy"
- trivy image --exit-code 1 --severity CRITICAL,HIGH $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
- echo "Pushing image to ECR"
- docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
Advanced Secrets Management in CI/CD
Never store secrets in CodeBuild environment variables. Fetch them at runtime from AWS Secrets Manager via a secure VPC Endpoint to keep traffic off the public internet.
🏗️ HCL: Secure Secrets Access with Terraform
This Terraform configuration creates a secret, a VPC endpoint for Secrets Manager, and a least-privilege IAM policy for CodeBuild to fetch that specific secret.
# Define the data source for the current AWS region
data "aws_region" "current" {}
# 1. The secret itself
resource "aws_secretsmanager_secret" "api_key" {
name = "app/third-party-api-key"
}
# 2. VPC Endpoint for Secrets Manager to keep traffic private
resource "aws_vpc_endpoint" "secretsmanager_vpce" {
vpc_id = aws_vpc.main.id
# Use $$ to escape the ${...} sequence for the Astro renderer
service_name = "com.amazonaws.${data.aws_region.current.name}.secretsmanager"
vpc_endpoint_type = "Interface"
subnet_ids = aws_subnet.private[*].id
security_group_ids = [aws_security_group.allow_https.id]
}
# 3. IAM Policy for CodeBuild role to access the specific secret
data "aws_iam_policy_document" "codebuild_secrets_policy" {
statement {
effect = "Allow"
actions = ["secretsmanager:GetSecretValue"]
resources = [aws_secretsmanager_secret.api_key.arn]
}
}
resource "aws_iam_policy" "secrets_policy" {
name = "CodeBuildGetApiKeySecretPolicy"
policy = data.aws_iam_policy_document.codebuild_secrets_policy.json
}Detective Controls and Auditing for Pipelines
In addition to preventive controls, you need detective controls to continuously audit your CI/CD environment for security drift.
🐍 Python: Audit CodeBuild for Privileged Mode
This Lambda function can be run on a schedule to scan all CodeBuild projects and report any that have the insecure "privileged mode" enabled.
import boto3
import json
import datetime
codebuild = boto3.client('codebuild')
securityhub = boto3.client('securityhub')
aws_account_id = boto3.client('sts').get_caller_identity()['Account']
aws_region = boto3.session.Session().region_name
def audit_codebuild_projects():
findings = []
projects = codebuild.list_projects()['projects']
if not projects:
return
project_details = codebuild.batch_get_projects(names=projects)['projects']
for project in project_details:
if project.get('environment', {}).get('privilegedMode', False):
print(f"FOUND non-compliant project: {project['name']}")
finding = {
'SchemaVersion': '2018-10-08',
'Id': f"{project['arn']}/privileged-mode-check",
'ProductArn': f"arn:aws:securityhub:{aws_region}:{aws_account_id}:product/{aws_account_id}/default",
'GeneratorId': 'CodeBuildAuditor',
'AwsAccountId': aws_account_id,
'Types': ['Software and Configuration Checks/Vulnerabilities/CVE'],
'FirstObservedAt': datetime.datetime.utcnow().isoformat() + 'Z',
'UpdatedAt': datetime.datetime.utcnow().isoformat() + 'Z',
'Severity': {'Label': 'HIGH'},
'Title': 'CodeBuild Project has Privileged Mode Enabled',
'Description': f"The CodeBuild project {project['name']} has privilegedMode enabled, which poses a security risk.",
'Resources': [{'Type': 'AwsCodeBuildProject', 'Id': project['arn'], 'Region': aws_region}]
}
findings.append(finding)
if findings:
securityhub.batch_import_findings(Findings=findings)
print(f"Imported {len(findings)} findings into Security Hub.")
def lambda_handler(event, context):
audit_codebuild_projects()
return {'statusCode': 200, 'body': json.dumps('Audit complete!')}
Troubleshooting Complex Pipeline Security Issues
Securing a pipeline introduces new potential points of failure. Here’s how to debug them.
❌ CodeBuild Fails with `AccessDenied` to another AWS Service
- Symptom: The build fails with an access denied error when trying to interact with S3, ECR, KMS, or another service.
- Root Cause Checklist:
- **CodeBuild Service Role:** The most common cause. The IAM role assumed by the CodeBuild project lacks the necessary permissions for the target service.
- **Resource Policy:** Does the target resource (e.g., the S3 bucket) have a resource-based policy that explicitly denies the CodeBuild role's ARN?
- **VPC Endpoint Policy:** If CodeBuild is in a VPC, does the VPC Endpoint for the target service have a policy that denies the action?
- **Permissions Boundary:** Is there a permissions boundary attached to the CodeBuild role that prevents it from having the required permissions?
🔑 Secrets Manager `AccessDeniedException` from within VPC
- Symptom: A CodeBuild project in a VPC fails to retrieve a secret, even though its IAM role has `secretsmanager:GetSecretValue`.
- Cause: The CodeBuild project is in a private subnet with no route to the public internet, and a VPC endpoint for Secrets Manager is either missing or has a restrictive policy.
- **Solution:** Create an Interface VPC Endpoint for `com.amazonaws.REGION.secretsmanager`. Ensure its security group allows inbound HTTPS from the CodeBuild security group, and verify the endpoint policy allows the action from the CodeBuild role's ARN.
🔑 Expert-Level DevSecOps Best Practices
- Separate Roles for Each Stage: Use distinct, least-privilege IAM roles for your build, and deployment stages. Never use a single, overly permissive role for the entire pipeline.
- **Treat `buildspec.yml` as a Security Control:** Your buildspec is a powerful policy enforcement point. Codify SAST, SCA, IaC, and container scanning steps and fail the build on critical findings.
- **Never Use Long-Lived Credentials:** There should be no IAM user access keys in your pipeline. All authentication should be done via IAM roles for AWS services.
- **Fetch Secrets at Runtime:** Avoid using CodeBuild environment variables for secrets. Fetch them dynamically from AWS Secrets Manager or Parameter Store during the build process.