`

8 Things That Broke When I Deployed Kubernetes on AWS EKS Fargate (And How I Fixed Them)

Building a production-shaped microservices platform from scratch — what the tutorials don't cover.

I spent 8 weeks building ShopSphere: a cloud-native e-commerce backend with 3 FastAPI microservices, deployed on Amazon EKS, monitored with Prometheus and Grafana, secured with GuardDuty and AWS Secrets Manager, and delivered by a GitHub Actions CI/CD pipeline.

The stack: Python 3.13 + FastAPI → Docker → ECR → EKS Fargate → RDS PostgreSQL → ALB → Terraform IaC → GitHub Actions.

Everything I've written below is real. I didn't learn it from a tutorial — I learned it because it broke at 11pm and I had to figure out why.

Why EKS Fargate (and the EC2 quota wall)

The original plan was a standard EKS cluster with managed EC2 node groups. That plan died immediately.

New AWS accounts start with an EC2 vCPU quota of 0 for several instance families. Both On-Demand and Spot were blocked. Requesting quota increases takes days and isn't guaranteed. The project couldn't wait.

Fargate is the alternative: AWS runs each pod on its own dedicated microVM. No EC2 Auto Scaling Groups, no node management, no quota to hit. You pay per pod-second of CPU and memory rather than per instance.

I migrated the cluster to Fargate. This was the right call. But Fargate has its own set of constraints that are scattered across AWS documentation, GitHub issues, and Stack Overflow threads — never in one place. Here's what I hit.

Problem 1: CoreDNS silently failing to schedule

Symptom: After applying the Fargate profile and deploying, service name resolution failed cluster-wide. The ALB controller couldn't start. External Secrets Operator couldn't start. Prometheus couldn't scrape anything. Everything that needed to reach some-service.namespace.svc.cluster.local just timed out.

Root cause: Fargate uses a mutating admission webhook to intercept pod creation and inject Fargate-specific configuration. For a pod to be scheduled on Fargate, the webhook needs to process it. For the webhook to process it, the pod needs the annotation eks.amazonaws.com/compute-type: fargate.

CoreDNS's default Kubernetes deployment doesn't have this annotation. So Fargate's webhook ignores CoreDNS pods, they never get scheduled, and they sit Pending indefinitely. Since CoreDNS is the cluster's DNS resolver, everything else fails.

Fix:
bash
kubectl patch deployment coredns -n kube-system \
--type=json \
-p='[{"op":"add","path":"/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type","value":"fargate"}]'
kubectl rollout restart deployment/coredns -n kube-system


This is documented in AWS's EKS + Fargate guide but easy to miss when you're following a general EKS tutorial.

Problem 2: Prometheus, AlertManager, Grafana all refuse to start

Symptom:helm install monitoring prometheus-community/kube-prometheus-stack completes without error. But all pods are stuck with Pod not supported on Fargate: volumes not supported.

Root cause:kube-prometheus-stack's default Helm values request PersistentVolumeClaims backed by EBS storage for Prometheus (metrics storage), AlertManager (alert state), and Grafana (dashboard state). Fargate pods cannot mount EBS volumes. At all. It's not a configuration issue — it's a fundamental architectural constraint.

Fix: Switch to ephemeral in-pod storage for the entire monitoring stack:

`yaml

prometheus-values.yaml

prometheus:
prometheusSpec:
storageSpec: {} # no PVC — ephemeral storage
retention: 6h # short retention; this is dev, not production

alertmanager:
alertmanagerSpec:
storage: {} # no PVC

grafana:
persistence:
enabled: false # no PVC
sidecar:
dashboards:
enabled: true # load dashboards from ConfigMaps instead
`

The monitoring stack becomes stateless by design. Prometheus re-scrapes from pod startup on restart. Grafana dashboards live in ConfigMaps — code-defined, version-controlled, no state to lose.

Problem 3: External Secrets Operator webhook port collision

Symptom: ESO installs successfully. The SecretStore applies without error. But ExternalSecret objects never sync — they stay in a permanent pending state. ESO pods show TLS errors in their logs: x509: certificate is valid for [...], not for [fargate-node-ip].

Root cause: ESO's admission webhook runs on port 10250 by default. On Fargate, every pod runs in its own microVM that has its own kubelet — also on port 10250. When ESO registers its webhook with the Kubernetes API server, and the API server tries to call the webhook to validate ExternalSecret objects, it connects to what it thinks is the ESO webhook address but is actually the Fargate node's kubelet on that port. The TLS certificate ESO presents doesn't include the Fargate node's internal address in its SANs — hence the mismatch.

This affects several Kubernetes webhook-based operators on Fargate: cert-manager, ADOT, and ESO all have open issues for this.

Fix:
`yaml

In the ESO Helm values

webhook:
port: 9443 # anything other than 10250
`

Problem 4: Fargate profile updates stranding pods permanently Pending

Symptom: After adding a new namespace to the Fargate profile, some pods in existing namespaces get stuck Pending and never schedule, despite the profile update completing successfully.

Root cause: Fargate profiles are immutable — adding a namespace selector requires destroying and recreating the profile. AWS does this automatically during an update, but there's a brief window (a few seconds to a minute) where no Fargate profile is active. Any pod that gets created during this window goes through normal Kubernetes scheduling. The default scheduler tries to find an EC2 node — there are none. The pod sits Pending. When the new profile comes back, it only evaluates pods at creation time, not retroactively. Pods already stuck Pending with the wrong scheduler state never get reconsidered for Fargate.

Fix: After any Fargate profile update, find and delete all Pending pods so they get recreated and scheduled correctly:

`bash
kubectl get pods --all-namespaces | grep Pending

For each stuck pod:

kubectl delete pod -n
`

Automate this if you're doing frequent profile updates during setup.

Problem 5: IRSA roles referenced but not created

Symptom: Pods start successfully. Health checks pass. But any AWS API call — reading from Secrets Manager, listing ECR images — fails with AccessDenied. The pod appears healthy but is silently broken.

Root cause: I added IRSA annotations to the Kubernetes ServiceAccounts:

yaml
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/shopsphere-user-service-role"


But I hadn't actually created that IAM role in Terraform yet. The annotation references a role that doesn't exist. Kubernetes applies the ServiceAccount fine. The pod starts fine. The EKS pod identity webhook injects the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables correctly. The AWS SDK tries to assume the role — and gets AccessDenied because the role doesn't exist.

This is completely invisible until you actually make an AWS API call from inside the pod.

Fix: For each service, add to Terraform:

hcl
resource "aws_iam_role" "user_service" {
name = "shopsphere-user-service-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = {
Federated = "arn:aws:iam::${var.aws_account_id}:oidc-provider/${local.oidc_provider}"
}
Action = "sts:AssumeRoleWithWebIdentity"
Condition = {
StringEquals = {
"${local.oidc_provider}:sub" = "system:serviceaccount:shopsphere:user-service-sa"
}
}
}]
})
}


Every Kubernetes ServiceAccount that needs AWS permissions needs a corresponding IAM role and OIDC trust policy in Terraform. No exceptions.

Problem 6: read-only root filesystem breaking container startup

Symptom: After adding readOnlyRootFilesystem: true and capabilities: drop: [ALL] to the pod security context, pods fail to start with permission errors.

Root cause: The original Dockerfile used a CMD pattern that:

1. chown-ed the application directory at container start
2. Used su or gosu to drop from root to the app user

Both require either write access to the filesystem (for chown) or Linux capabilities that we just dropped (for su/gosu). With readOnlyRootFilesystem: true and no capabilities, the container can't start.

Fix: Move all ownership-setting to Dockerfile build time:

`dockerfile

In the runtime stage, before switching to appuser:

RUN adduser --disabled-password --no-create-home appuser && \
chown -R appuser:appuser /app

USER appuser

CMD just starts the app — no chown, no su needed

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8001"]
`

For volumes that need write access at runtime (temp files, SQLite), use emptyDir volumes in the pod spec and configure fsGroup in the security context — Kubernetes handles the ownership:

`yaml
securityContext:
fsGroup: 1000 # Kubernetes chowns volume mounts to this GID at pod start
volumes:

• name: tmp emptyDir: {} `

Problem 7: Diagnosing an account-level ELB restriction

Symptom: Creating a LoadBalancer-type Service fails. The AWS Load Balancer Controller logs show OperationNotPermitted. IAM permissions look correct. Quotas look fine. No relevant errors in CloudTrail except the failure itself.

Root cause: Some AWS accounts have account-level API restrictions applied that are distinct from IAM permissions and service quotas. These appear in the raw API response but not in the kubectl error summary. The pattern: OperationNotPermitted rather than AccessDenied or LimitExceeded.

This required opening an AWS Support case. It's not a configuration mistake. Attempting to work around it through configuration changes wastes time.

Workaround while the Support case resolves:kubectl port-forward for Grafana and AlertManager access. This is actually more secure — no public LoadBalancer for the monitoring stack.

Lesson: Read the full raw API error response, not just the kubectl summary. OperationNotPermitted and AccessDenied have different root causes and require different responses. Knowing which one you're looking at tells you whether to keep debugging configuration or open a Support case.

Problem 8: Fargate profile namespace selectors and webhook timing

Symptom: After applying a new Fargate profile that should cover a new namespace, pods in that namespace still don't schedule. They show 0/0 nodes available.

Root cause: A Fargate profile only schedules pods that match its namespace + label selectors, but the profile has to exist before the pod is created. If you apply the profile and then immediately apply the namespace and deployment in the same kubectl apply -f, there's a race — the pods may be created before the profile is fully active.

Fix: Add a sleep or confirmation step:

`bash

Wait for the Fargate profile to be ACTIVE before deploying

aws eks wait fargate-profile-active \
--cluster-name shopsphere-cluster \
--fargate-profile-name shopsphere-fp

Then deploy

kubectl apply -f k8s/base/
`

What the finished system looks like

After solving all of the above:

• git push origin main → 4 GitHub Actions jobs (test → scan → approve → deploy) → new version live in EKS with zero manual steps
• curl http://ALB_URL/health → 200 from all 3 services
• Grafana shows p95 latency, request rate, error rate, and pod health in real time
• AlertManager sent a real email when I deliberately triggered an error spike, and a resolution email when I fixed it
• kubectl get externalsecret -n shopsphere → SecretSynced — DB credentials come from Secrets Manager, not a YAML file
• kubectl get networkpolicies -n shopsphere → 5 policies, default-deny enforced
• GuardDuty and CloudTrail monitoring the account

The project is at github.com/saurabhg4356/shopsphere with full source, architecture diagram, and setup instructions.

What I'd tell someone starting this today

1. Set up a AWS budget alert before your first terraform apply. The NAT Gateway is always running and costs money even when nothing is happening.

2. Read the Fargate-specific documentation, not just the EKS documentation. The two have meaningfully different constraints and the Fargate docs are more scattered.

3. When something fails, read the full raw API error.kubectl describe gives summaries. The raw API response — in CloudTrail, in pod events, in controller logs — tells you the actual error code, which often tells you exactly what category of problem you're dealing with.

4. Terraform modules from the start. I built a flat structure first and refactored it into modules. It's much easier to start modular than to extract modules from a flat structure later.

5. The hard problems aren't the code. FastAPI is simple. Docker is simple. Kubernetes YAML is tedious but learnable. The genuinely hard part is the system-level debugging — when six different components interact and only one of them is wrong, and that one has a misleading error message.
   `