I was running through kubernetes core concepts w.r.t Azure AKS and came across this Microsoft documentation here. Naturally it took me some time(actually a lot of time) to understand and grasp the internals of how the compute and memory resources are allocated/reserved for pods in kubernetes. Hence writing this post share some some insights of my learning on this.
Limits and requests for CPU resources are measured in cpu units. One cpu, in Kubernetes, is equivalent to 1 vCPU/Core for cloud providers and 1 hyperthread on bare-metal Intel processors. 1 Core is effectively expressed as 1000 milli cores for notations in yaml files in k8s context.
Limits and requests for
memory are measured in bytes and expressed in MiB(Mebibyte’s) for pod specifications in k8s context. So a 8GiB Node would be an equivalent of 8192 Mib for notations in yaml files in k8s context.
Pods deployed in Kubernetes cluster consume resources such as CPU, memory and storage. However, not all resources in a Node can be used to run Pods.
The operating system and the kubelet require memory and CPU too, and you should cater for those extra resources. If you look closely at a single Node, you can divide the available resources in:
- Resources needed to run the operating system
- Resources necessary to run Kubernetes agents such as the Kubelet and the container runtime etc..
- Resources available to Pods
- Resources reserved to the eviction threshold
w.r.t to Azure AKS, how the distribution of the resources is calulated is described here.
- Specific CPU cores allocated for Kubelet and OS
- The following memory eviction rule on Azure AKS : memory.available<750Mi, which means a node must always have at least 750 Mi allocatable at all times.
- Memory reservations for the kubelet
- 25% of the first 4 GB of memory
- 20% of the next 4 GB of memory (up to 8 GB)
- 10% of the next 8 GB of memory (up to 16 GB)
- 6% of the next 112 GB of memory (up to 128 GB)
- 2% of any memory above 128 GB
In trying to make sense of the calculations, i have come up with this table to show some examples CPU/Memory resource allocations for certain CPU/Memory combinations below
|Total CPU CORES||CPU Reserved for Kubelet (in milli cores)||CPU Reserved for OS (in milli cores)||CPU Reserved for Pods (in milli cores)|
|Total Memory (GiB)||Memory Reserved for Kubelet (in GiB)||Memory Reserved for OS (in GiB)||Hard Eviction Threshold (in GiB)||Memory Reserved for Pods (in GiB)|
For an example, Considering a 2vCPU, 8GB RAM Azure Node for a Kubernetes cluster, this is how it would
Understanding these concepts clearly helps us to allocate sufficient resources to pods in our pod specification during Cluster deployment process and also helps to set Resources Quotas correctly depending on the type of application run and the expected usage of compute and memory resources.
- Azure AKS Core concepts: https://docs.microsoft.com/en-us/azure/aks/concepts-clusters-workloads
- Kubernetes documentation: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers and https://learnk8s.io/blog
Hope this is helpful. Happy Learning !!