Kubernetes Cost Optimization: AWS EKS vs GKE vs AKS (2025)

The Problem

Most teams cargo-cult resource requests from Stack Overflow without measuring actual usage. A pod requesting 2 CPU might only use 200m (10% utilization).

The Solution

# Before: Overprovisioned
apiVersion: v1
kind: Pod
metadata:
  name: webapp
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        cpu: "2000m"      # Requesting 2 CPUs
        memory: "4Gi"     # Requesting 4GB
      limits:
        cpu: "4000m"      # Limit at 4 CPUs
        memory: "8Gi"     # Limit at 8GB

# After: Right-sized based on metrics
apiVersion: v1
kind: Pod
metadata:
  name: webapp
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        cpu: "500m"       # Actually using 300-400m
        memory: "1Gi"     # Actually using 600-800Mi
      limits:
        cpu: "1000m"      # 2x headroom for spikes
        memory: "2Gi"     # 2x headroom for spikes

Impact: Reduced CPU request by 75%, memory by 75%. This allows 4x more pods on the same nodes.

How to Right-Size

Use Vertical Pod Autoscaler (VPA) in recommendation mode:

kubectl apply -f https://github.com/kubernetes/autoscaler/releases/latest/download/vertical-pod-autoscaler.yaml

# Create VPA in recommendation mode
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: webapp-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: webapp
  updateMode: "Off"  # Recommendation only

# Get recommendations
kubectl describe vpa webapp-vpa

Horizontal Pod Autoscaler (HPA)

Scale pods based on CPU, memory, or custom metrics:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: webapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: webapp
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Cluster Autoscaler

Scale nodes based on pending pods. Configuration varies by cloud (see EKS/GKE/AKS sections).

Impact: Automatically add nodes during traffic spikes, remove them during low usage. Can reduce costs by 30-50% for variable workloads.

Cost Savings

• AWS Spot: Up to 90% cheaper than On-Demand
• GCP Preemptible: Up to 80% cheaper
• Azure Spot: Up to 90% cheaper

Best Practices

• Use for stateless workloads: Web servers, batch jobs, CI/CD workers
• Mix with on-demand: Run critical services on on-demand, everything else on spot
• Use node taints/tolerations: Prevent critical pods from landing on spot nodes
• Diversify instance types: Use multiple spot pools to reduce interruption risk

# Taint spot nodes
kubectl taint nodes spot-node-1 spot=true:NoSchedule

# Tolerate spot in deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: batch-worker
spec:
  template:
    spec:
      tolerations:
      - key: "spot"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
      nodeSelector:
        node-type: spot

Common Storage Waste

• Orphaned volumes: PVCs deleted but underlying volumes remain (costs $$$)
• Wrong storage class: Using high-performance SSD for logs/backups
• Oversized volumes: 500GB volumes for 10GB of actual data

Storage Class Best Practices

# Use cheaper storage tiers for appropriate workloads
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: logs-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: standard  # Use standard/HDD for logs
  resources:
    requests:
      storage: 20Gi

# Use premium only for databases
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: premium-ssd  # Fast storage for DB
  resources:
    requests:
      storage: 100Gi

Impact: Standard storage is 3-5x cheaper than premium SSD. Use it for logs, backups, and non-critical data.

Prevent Runaway Costs

# Namespace-level quota
apiVersion: v1
kind: ResourceQuota
metadata:
  name: dev-quota
  namespace: development
spec:
  hard:
    requests.cpu: "50"
    requests.memory: "100Gi"
    limits.cpu: "100"
    limits.memory: "200Gi"
    persistentvolumeclaims: "10"

# Default limits for pods without resources
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: development
spec:
  limits:
  - default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "100m"
      memory: "128Mi"
    type: Container

Compute Savings Plans

Commit to consistent compute usage for 1-3 years for up to 72% discount:

• 1-year no upfront: ~40% savings
• 3-year all upfront: ~72% savings

Best practice: Cover your baseline 24/7 workloads with Savings Plans, use spot for burst capacity.

Fargate Spot Pricing

Up to 70% cheaper than regular Fargate. Perfect for:

• CI/CD pipelines
• Data processing jobs
• Machine learning training

# Fargate profile with spot
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-east-1
fargateProfiles:
  - name: batch-spot
    selectors:
      - namespace: batch
        labels:
          workload-type: spot-ok
    capacityType: SPOT

Auto Scaling Group Configuration

resource "aws_autoscaling_group" "eks_nodes" {
  name = "eks-mixed-nodes"
  
  mixed_instances_policy {
    instances_distribution {
      on_demand_base_capacity                  = 2
      on_demand_percentage_above_base_capacity = 20
      spot_instance_pools                      = 4
      spot_allocation_strategy                 = "capacity-optimized"
    }
    
    launch_template {
      launch_template_specification {
        launch_template_id = aws_launch_template.eks.id
        version           = "$Latest"
      }
      
      override {
        instance_type = "m5.large"
      }
      override {
        instance_type = "m5a.large"
      }
      override {
        instance_type = "m6i.large"
      }
      override {
        instance_type = "m6a.large"
      }
    }
  }
}

Impact: 2 on-demand nodes for stability, 80% spot for cost savings. Diversify across 4 instance types to reduce interruption risk.

Cost and Performance Benefits

• 20% cheaper than comparable x86 instances
• 40% better price-performance for many workloads

Supported: Most EKS versions support Graviton (ARM64). Just ensure your container images support linux/arm64.

# Build multi-arch images
docker buildx build --platform linux/amd64,linux/arm64 -t myapp:latest --push .

Control Plane Pricing

AWS charges $0.10/hour ($73/month) per EKS cluster. Strategies:

• Multi-tenancy: Run multiple apps in one cluster using namespaces
• Consolidate dev/staging: Use one cluster with different namespaces instead of separate clusters
• Delete idle clusters: Shut down dev/test clusters overnight

Impact: Reducing from 10 clusters to 3 saves $510/month in control plane costs alone.

What is Autopilot?

GKE manages nodes, autoscaling, and security. You only pay for pod resource requests (not idle capacity).

Cost Model

Pay only for CPU, memory, and ephemeral storage requested by pods. No wasted capacity on overprovisioned nodes.

When to Use

• Variable workloads with unpredictable scaling
• Teams without dedicated K8s expertise
• Workloads that don't need node-level customization

# Create Autopilot cluster
gcloud container clusters create-auto my-cluster \
    --region=us-central1 \
    --release-channel=regular

Impact: Can reduce costs by 30-50% for variable workloads by eliminating idle node capacity.

GKE Spot Configuration

# Create node pool with spot VMs
gcloud container node-pools create spot-pool \
    --cluster=my-cluster \
    --spot \
    --machine-type=n2-standard-4 \
    --num-nodes=3 \
    --enable-autoscaling \
    --min-nodes=1 \
    --max-nodes=10 \
    --zone=us-central1-a

Pricing: Up to 80% discount vs regular VMs. Max runtime: 24 hours (then VM is terminated).

Enable Usage Metering

gcloud container clusters update my-cluster \
    --enable-resource-consumption-metering \
    --resource-consumption-bigquery-dataset=gke_usage

Exports detailed resource consumption to BigQuery for chargeback and cost analysis by namespace, label, or pod.

GCP Committed Use Pricing

• 1-year commitment: ~37% discount
• 3-year commitment: ~55% discount

Applied automatically to matching VM usage. No need to configure per-cluster.

E2 Cost Advantage

E2 instances are 30-40% cheaper than N1/N2 for similar performance. Great for:

• Development environments
• Staging clusters
• Batch processing
• Web servers with predictable load

Reserved VM Pricing

• 1-year reservation: ~40% savings
• 3-year reservation: ~62% savings

Reserve capacity for your baseline AKS node pools. Use spot for burst capacity.

Create Spot Node Pool

# Create spot node pool
az aks nodepool add \
    --cluster-name myAKSCluster \
    --resource-group myResourceGroup \
    --name spotnodepool \
    --priority Spot \
    --eviction-policy Delete \
    --spot-max-price -1 \
    --enable-cluster-autoscaler \
    --min-count 1 \
    --max-count 10 \
    --node-count 3

Pricing: Up to 90% discount. Set spot-max-price to -1 to pay current spot rate (never more than on-demand).

What are Virtual Nodes?

Serverless burst capacity using Azure Container Instances (ACI). Pay per second for pods that run on virtual nodes.

Best Use Cases

• Burst workloads (CI/CD jobs)
• Event-driven processing
• Batch jobs

# Enable virtual nodes
az aks enable-addons \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --addons virtual-node \
    --subnet-name VirtualNodeSubnet

Windows Node Savings

If you have Windows Server licenses with Software Assurance, you can use them for AKS Windows node pools at no additional licensing cost.

Savings: Up to 40% on Windows node compute costs.

B-Series Benefits

B-series VMs are 40-60% cheaper than general-purpose VMs. Perfect for:

• Development clusters
• Test environments
• Low-traffic applications

Note: Not suitable for production workloads with sustained CPU usage.

Assumptions

• Instance type: 4 vCPU, 16GB RAM (m5.xlarge / n2-standard-4 / D4s_v3)
• 100 nodes total
• 70% baseline (reserved), 30% burst (spot)
• 24/7 operation

Monthly Cost Breakdown

Winner for this scenario: AWS EKS by a small margin, but all three are within 5% of each other when optimized properly.