1:17:34
Upgrade Amazon EKS Cluster Without Downtime | Full Guide 🔥
Code with Gauri
Overview
This video provides a comprehensive guide on upgrading an Amazon EKS cluster without downtime. It details the necessary prerequisites, the step-by-step process for upgrading the control plane, add-ons, and worker nodes, and emphasizes the importance of thorough preparation and testing. The guide covers checking release notes for deprecated APIs, ensuring version compatibility between control plane and worker nodes, verifying IP address availability in subnets, and confirming add-on compatibility. It also touches upon backup strategies, stakeholder notification, and the practical execution using AWS CLI and EKSCTL, demonstrating a zero-downtime upgrade for running applications.
How was this?
Save this permanently with flashcards, quizzes, and AI chat
Chapters
- DevOps engineers are responsible for maintaining EKS clusters, including regular version upgrades.
- Kubernetes releases new versions approximately every 3 months, necessitating upgrades to stay current.
- Upgrades must be performed without impacting running applications (zero downtime).
- Cluster upgrades involve updating components like the control plane, add-ons, and worker nodes separately.
Understanding the necessity and the component-based nature of EKS upgrades is crucial for maintaining a secure and up-to-date Kubernetes environment.
You cannot directly upgrade from Kubernetes version 1.30 to 1.34; you must upgrade sequentially (1.30 -> 1.31 -> 1.32 -> 1.33 -> 1.34).
- The recommended upgrade order is control plane first, then add-ons, and finally worker nodes.
- Worker node versions must be the same as or up to two minor versions lower than the control plane version.
- Control plane versions cannot be downgraded, only upgraded.
- Add-ons can potentially be downgraded, but control plane versions cannot.
Adhering to the correct upgrade order and understanding version compatibility rules prevents potential conflicts and ensures a stable cluster.
If the control plane is at version 1.34, worker nodes can be 1.34, 1.33, or 1.32, but not 1.35.
- Thoroughly review Kubernetes release notes to identify deprecated APIs and breaking changes.
- Update application manifests (YAML files) to comply with API changes (e.g., updating Ingress API versions).
- Ensure the control plane and worker node versions are aligned before starting the upgrade.
- Test upgrades in lower environments (dev, UAT, pre-prod) before attempting in production due to the inability to downgrade the control plane.
- Have at least five available IP addresses in the cluster's subnet for rolling updates of nodes.
- Verify the compatibility of all add-ons with the target Kubernetes version.
- Take backups of applications and configurations as a safety measure.
- Notify all stakeholders about the planned upgrade window and avoid scheduling new deployments during this time.
These prerequisites are essential for mitigating risks, preventing unexpected issues, and ensuring a smooth and successful cluster upgrade.
Checking release notes for Kubernetes 1.35 might reveal that 'fail cgroups v1' is deprecated and will be set to true by default, impacting how nodes start.
- Use `eksctl` or AWS CLI commands to initiate the control plane upgrade.
- Monitor the upgrade process, which can take several hours.
- After the control plane is upgraded, update EKS add-ons (like VPC CNI, CoreDNS, kube-proxy) to compatible versions.
- Add-on upgrades can be managed via the AWS console or AWS CLI/EKSCTL commands.
- Verify that the client `kubectl` version is also updated or compatible with the new server version.
Successfully upgrading the control plane and add-ons forms the foundation for a stable, upgraded cluster environment.
Upgrading the VPC CNI add-on from version v1.20.4 to v1.21.1 using the AWS CLI command `aws eks update-addon`.
- For managed node groups, the AWS console or `eksctl` can initiate node group version updates.
- This process involves creating new nodes with the target version and gradually replacing the old ones.
- During node replacement, existing applications remain accessible, demonstrating zero downtime.
- For self-managed node groups, cordon nodes (make unschedulable), drain them (evict pods), update the launch template (AMI), and then terminate old nodes or use instance refresh.
- Monitor the node replacement process to ensure all old nodes are terminated and new nodes are healthy.
- Verify that all workloads have successfully migrated to the new nodes.
Upgrading worker nodes is the final step to ensure all cluster components are running the desired Kubernetes version, completing the upgrade.
Observing `kubectl get nodes` showing a mix of old (1.34) and new (1.35) nodes during the upgrade, while the Nginx application remains accessible.
- After the upgrade, verify the cluster version (`kubectl version`), node status (`kubectl get nodes`), and the health of all pods and add-ons.
- Perform smoke tests to ensure applications are functioning correctly.
- EKS does not support control plane downgrades; however, managed node groups and add-ons can be rolled back if necessary.
- Workloads themselves can also be rolled back if they were upgraded as part of the process.
Thorough post-upgrade verification confirms the success of the upgrade and ensures the cluster is stable, while understanding rollback options provides a safety net.
Confirming the final cluster version is 1.35 using `eksctl cluster-info` and verifying all pods are running on nodes with the new version.
Key takeaways
- EKS cluster upgrades are a continuous process due to frequent Kubernetes releases and must be performed without disrupting running applications.
- A successful zero-downtime upgrade requires meticulous planning, including reviewing release notes, ensuring version compatibility, and testing in non-production environments.
- The upgrade process involves distinct steps for the control plane, add-ons, and worker nodes, each with specific considerations.
- Maintaining sufficient IP addresses in subnets is critical for the rolling update mechanism of worker nodes.
- While control plane downgrades are not supported, managed node groups and add-ons offer rollback capabilities.
- Thorough verification post-upgrade, including checking node and pod health, is essential to confirm a successful transition.
- For self-managed node groups, manual steps like cordoning and draining are required before updating launch templates and terminating old nodes.
Key terms
EKS Cluster UpgradeControl PlaneWorker NodesAdd-onsZero DowntimeRelease NotesDeprecated APIsVersion CompatibilityRolling UpdateCordonDrainManaged Node GroupsSelf-Managed Node GroupsEKSCTLAWS CLI
Test your understanding
- Why is it important for DevOps engineers to regularly upgrade EKS clusters, and what is the primary constraint during this process?
- What are the recommended steps and version compatibility rules when upgrading the control plane and worker nodes in an EKS cluster?
- Describe the essential prerequisites that must be met before attempting an EKS cluster upgrade in a production environment.
- How does the video demonstrate achieving zero downtime for applications during the worker node upgrade process?
- What are the key differences in the upgrade process between managed node groups and self-managed node groups in EKS?