10. Chapter review

The manual sequence works the first few times and then quietly stops working: someone forgets to run tests, someone builds with the wrong env var, someone deploys to staging when they meant prod. Every manual step is a step that can be skipped. The CI/CD pipeline replaces all of that with a fixed sequence (test, build, push, register task definition, update service, wait for stability) that runs identically every time and is gated on the test result, not on whoever is at a terminal. The workflow file lives in the repo, so the deploy process evolves under the same review discipline as the code it ships.

The four signals (latency, traffic, errors, saturation) each correspond to a different failure mode. Latency surfaces slow downstream calls (slow queries, slow third-party APIs, contention on the host). Traffic surfaces usage shape -- both growth and unexpected drops, which often mean clients are giving up. Errors catch bugs, misconfigurations, and dependency failures. Saturation is the leading indicator: connection-pool full at 95% usually surfaces a couple of minutes before the latency and error signals do. The framework's value isn't comprehensiveness; it's that four metrics is few enough that the on-call can keep them in their head, and they cover most of what actually breaks in production.

The policy holds the named metric at the target (70% CPU here): when the average crosses above, ECS adds tasks until it falls back; when it crosses below, ECS removes them. Cooldowns prevent the policy from acting on its own previous action. After scaling out, the 60-second cooldown lets new tasks start and pass health checks before another scaling decision is made; without it, the policy launches too many. After scaling in, the longer 300-second cooldown waits until the lull is real, because if traffic resumes and tasks were just terminated, the policy has put the service in a capacity hole. The asymmetry maps to the asymmetric cost of the two failures: scaling out too much is money; scaling in too soon is downtime.

Rolling: stop one old task, start one new one, wait for the health check, repeat. Old and new versions overlap during the rollout, which is fine for backwards-compatible changes and uncomfortable for ones that aren't. Rollback means redeploying the previous task definition, which is also a rolling update -- slow if things are really broken. Blue-green: a parallel green stack is brought up alongside the running blue stack, validated against real load on a separate target group, then traffic cuts over at the ALB listener level. Rollback is one listener-rule update back to blue, in seconds. The cost is running two stacks for the duration of the deploy. The rule of thumb: rolling-with-circuit-breaker for everything by default; blue-green for the deploys where a brief capacity dip or a slow rollback would actually hurt.

Open Cost Explorer, filter to the relevant window, group by service. That tells you which service line moved. If it's ECS, check task-count history and any CPU/memory allocation changes. If it's RDS, look for instance-class changes and storage growth. If it's data transfer, look at ALB traffic and any S3 movement. Cost allocation tags let you split the spike across environments (was it dev or prod?) or projects. Common causes: auto-scaling stayed scaled out longer than expected, a test resource was never deleted, log retention quietly accumulated months of data, or Free Tier expired. Once you've named the driver, the decision is whether it's growth (fine) or waste (delete it).

Confirm first: open the dashboard, compare error rate, latency, and traffic against baseline. Then go to Logs Insights and run fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 50 against the window around 14:30. Look for the dominant error message; the usual suspects are connection-pool exhaustion, external API timeouts, and OOM. Cross-reference Container Insights for CPU, memory, and task count to rule out saturation. Check ECS service events for task failures. Once the root cause is named (say, the connection pool maxed out at 10), apply the immediate fix (raise the pool size), push it through the CI/CD pipeline from section 3, and confirm the error graph drops. Postmortem captures the timeline, the cause, the fix, and the prevention work (in this example: pool-utilisation metric and alarm).

Anything in the task definition's environment field shows up in plaintext in describe-task-definition output, which is visible to anyone with ECS read permission. Secrets Manager (or Parameter Store) keeps the value encrypted at rest with KMS, gates access through IAM, and logs every retrieval in CloudTrail. When the task definition references the secret as an ARN under secrets[] instead, ECS fetches the value at task start via the execution role and injects it as an env var inside the container -- the container code sees the same env var it always did, but describe-task-definition only reveals the ARN. The other operational win is that rotating the secret doesn't require a redeploy; the next task that starts picks up the new value automatically.

The difference is whether the operational layer exists at all. A one-shot deploy gets the code running and then nothing watches it; six months later, drift, secrets leakage, untracked spend, and silent regressions accumulate. The operational layer the chapter added has seven pieces, and each one is doing a specific job: CI/CD removes human error from the deploy step; the dashboard plus three alarms surface problems before users notice; auto-scaling sizes capacity to load; cost guardrails keep the bill explainable; deployment circuit breakers roll back automatically when a deploy goes bad; runbooks make incidents resolvable by anyone on call; secrets management plus security-group audits keep the credential and network surface tight. None of those is heroic on its own; together they're what makes the deployment maintainable.

Low CPU/memory with high latency points away from the compute tier and toward something downstream. Start with database query time: fields @timestamp, @message | filter @message like /query took/ | parse @message /query took (?<duration>\d+)ms/ | filter duration > 1000 | sort duration desc. If query times look normal, check the external APIs the news endpoint calls (News API, Guardian) for degradation. Then look at the Redis hit rate; a sudden drop in hit rate means every request is taking the slow uncached path, which is the same shape as a latency spike with healthy CPU. The general rule: latency without saturation is almost always a dependency problem, not a capacity problem.

Several configuration issues could limit scaling: (1) Task placement constraints, if you don't have enough availability zones or subnets configured, ECS can't place additional tasks even though the service allows 10. (2) Insufficient ENI capacity, each Fargate task requires an elastic network interface. VPC subnet CIDR blocks might be exhausted. (3) Service quotas, AWS accounts have default limits on concurrent Fargate tasks per region (often 100-500). Check service quotas in AWS Console. (4) Target tracking calculation lag, auto-scaling evaluates metrics every 60 seconds and acts conservatively. Five tasks might be technically sufficient to bring CPU below 70% target, so scaling stops there despite temporary 95% spikes. To diagnose: Check ECS service events for task placement failures. Review CloudWatch metrics to see if CPU actually stays above threshold long enough to trigger additional scaling. Verify VPC has sufficient IP addresses available. This troubleshooting process demonstrates systematic debugging: check configuration limits, check capacity constraints, check metric interpretation.