Making sense of kubernetes Resource reservations in Azure AKS

I was running through kubernetes core concepts w.r.t Azure AKS and came across this Microsoft documentation here. Naturally it took me some time(actually a lot of time) to understand and grasp the internals of how the compute and memory resources are allocated/reserved for pods in kubernetes. Hence writing this post share some some insights of my learning on this.

Compute

Limits and requests for CPU resources are measured in cpu units. One cpu, in Kubernetes, is equivalent to 1 vCPU/Core for cloud providers and 1 hyperthread on bare-metal Intel processors. 1 Core is effectively expressed as 1000 milli cores for notations in yaml files in k8s context.

Memory

Limits and requests for memory are measured in bytes and expressed in MiB(Mebibyte’s) for pod specifications in k8s context. So a 8GiB Node would be an equivalent of 8192 Mib for notations in yaml files in k8s context.

Pods deployed in Kubernetes cluster consume resources such as CPU, memory and storage. However, not all resources in a Node can be used to run Pods.

The operating system and the kubelet require memory and CPU too, and you should cater for those extra resources. If you look closely at a single Node, you can divide the available resources in:

  1. Resources needed to run the operating system
  2. Resources necessary to run Kubernetes agents such as the Kubelet and the container runtime etc..
  3. Resources available to Pods
  4. Resources reserved to the eviction threshold
Kubernetes Resource Allocation snapshot

w.r.t to Azure AKS, how the distribution of the resources is calulated is described here.

  1. Specific CPU cores allocated for Kubelet and OS
  2. The following memory eviction rule on Azure AKS : memory.available<750Mi, which means a node must always have at least 750 Mi allocatable at all times.
  3. Memory reservations for the kubelet
  • 25% of the first 4 GB of memory
  • 20% of the next 4 GB of memory (up to 8 GB)
  • 10% of the next 8 GB of memory (up to 16 GB)
  • 6% of the next 112 GB of memory (up to 128 GB)
  • 2% of any memory above 128 GB

In trying to make sense of the calculations, i have come up with this table to show some examples CPU/Memory resource allocations for certain CPU/Memory combinations below

Total CPU CORESCPU Reserved for Kubelet (in milli cores)CPU Reserved for OS (in milli cores)CPU Reserved for Pods (in milli cores)
160100840
21001001800
41401003760
81801007720
1626010015640
3242010031480
6474010063160

Total Memory (GiB)Memory Reserved for Kubelet (in GiB)Memory Reserved for OS (in GiB)Hard Eviction Threshold (in GiB)Memory Reserved for Pods (in GiB)
10.250.10.750.00
20.50.10.750.65
3.50.8750.10.751.78
410.10.752.15
81.80.10.755.35

For an example, Considering a 2vCPU, 8GB RAM Azure Node for a Kubernetes cluster, this is how it would

Understanding these concepts clearly helps us to allocate sufficient resources to pods in our pod specification during Cluster deployment process and also helps to set Resources Quotas correctly depending on the type of application run and the expected usage of compute and memory resources.

References:

  1. Azure AKS Core concepts: https://docs.microsoft.com/en-us/azure/aks/concepts-clusters-workloads
  2. Kubernetes documentation: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers and https://learnk8s.io/blog

Hope this is helpful. Happy Learning !!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s