How to write a custom Kubernetes Controller:
In this KB sized post we will understand internals of Kubernetes Controller with the help of a tiny custom controller - Chronos that I wrote to watch changes to Pods deployed in all the namespaces.
What is a Controller?
In simple terms, a controller is a endless loop that continuously checks the state of every object inside a system . Whenever the state of an object changes, a dedicated handler is invoked that is responsible for performing an action.
When you create an object in Kubernetes you define the state of the object which is then persisted to the data store. Kubernetes runs many controllers in the background to continuously track the state of these objects. If the controller detects any change to that object, Kubernetes tries to bring it back to the desired state.
A simple exercise will be sufficient to understand this. Schedule a deployment containing 2 replicas which will then create 2 pods. Now the desired state of the deployment is to run 2 pods at all times. If you delete one of the pods, Kubernetes will automatically spin up a new pod to match the desired state (which is 2 pods). In background, one of the inbuilt Kubernetes controller called as Replication Controller which is responsible for tracking changes to the replica set of deployments detected the change in the state of the pod and took necessary action.
The simplest form of controllers logic would look like below:
for {
desiredState := getDesiredState()
currentState := getCurrentState() if currentState != desiredState {
makeChanges(desired, current)
}
}
As you can see, its an endless loop that runs some logic to maintain the desired state of Kubernetes.
Components of a Controller
Kubernetes controller has many components but the most important ones are listed below:
- Informer
If a controller wants to check the state of an object, it has to make a http request to the API server. However performing such API calls frequently would bring down the performance of the entire system. Informers solve this problem by querying the data store for a specific object only for the first time. This information is stored in local cache of the controller. After that it starts watching that resource continuously and informs the controller only when it detects a change to an object state.
Now, the local cache created by the informer will be used by the informer alone. However in Kubernetes, multiple controllers might be watching a single resource and in such cases each controller will populate its own cache with information and there are chances that this information will be different for each controller since the caches are not in sync. Therefore another kind of informer called as a Shared Informer is used so that the cache is shared among all the controllers.
To create a shared informer you can use NewSharedInformerFactory function available in k8s.io/client-go/informers package which returns a factory of all the informers that can be created. You can now create an informer by selecting the kind of object you want to watch. In below example, a pod informer is created.
kc, err := utils.GetClient(config)if err != nil {
logrus.Fatal(err)
}factory := informers.NewSharedInformerFactory(kc, 0)
informer := factory.Core().V1().Pods().Informer()
For information on types of informers provided by NewSharedInformerFactory check the SharedInformerFactory interface available under k8s.io/client-go/informers:
type SharedInformerFactory interface {
internalinterfaces.SharedInformerFactory
ForResource(resource schema.GroupVersionResource) (GenericInformer, error)
WaitForCacheSync(stopCh <-chan struct{}) map[reflect.Type]bool
Admissionregistration() admissionregistration.Interface
Internal() apiserverinternal.Interface
Apps() apps.Interface
Autoscaling() autoscaling.Interface
Batch() batch.Interface
Certificates() certificates.Interface
Coordination() coordination.Interface
Core() core.Interface
Discovery() discovery.Interface
Events() events.Interface
Extensions() extensions.Interface
Flowcontrol() flowcontrol.Interface
Networking() networking.Interface
Node() node.Interface
Policy() policy.Interface
Rbac() rbac.Interface
Scheduling() scheduling.Interface
Storage() storage.Interface
}2. Workqueue
Based on the name one can easily figure out that it is some kind of queue that is maintained by the controller with all the items that have to be processed. But why is it required?
A shared informer cannot track the activity of each controller and depends on an external queuing mechanism which is provided by a work queue.
Whenever an informers event handler receives an event, it places a unique key resource_namespace>/<resource_name> in the workqueue which will be later used by workers to process this event. If the name space portion is empty then the key will be just the resource name. We can write functions to process the events placed on the queue. If we fail to process an event we can choose to re queue the item or discard it. This logic will also depend on the type of workqueue that is being used. There are many types of work queues available and the one I am using is called as Rate Limiting Queue. Check the k8s.io/client-go/util/workqueue library for more information on workqueue.
q := workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter())Chronos
Chronos is a fancy name that I gave to a simple controller that I wrote for watching changes to Pods in all namespaces. You can find the entire code under this repository.
The controller.go file under pkg directory holds the main controller logic. The code does the following:
- Creates a new shared informer
- The informer handles Add, Update and Delete events of Pods in all namespaces
- Creates a new work queue
- Creates a worker function to process the events placed onto the queue
Custom Types:
Next, we will create a new client that we will pass to the sharedinformer factory function to create an instance of Pod Informer.
The newController function receives the client and informer parameters created in Start func and returns a controller. A new work queue is created and event handlers are attached to the informer.
The run func starts the informer.We will also synchronize the local cache before we start processing the items in queue that were placed by the event handlers:
The runWorker func will start a infinite loop. processNextItem func will get the next item in from the queue that has to be processed and will pass it to the processItem func.
processItem func uses GetIndexer().GetByKey() func to get the key of the item from the queue. It then logs the type of event and the object to stdout:
See it in action
Pass the path of kubeconfig file using -k flag to the command and it will start watching Pods.
It will watch the pods in all namespaces and process create, update and delete events.
Wrap Up!
If you are planning to write a custom controller, you might have to understand the concepts well before you can start writing one. It might seem confusing in the beginning but it will make sense once you have understood the fundamentals. Most of the documentation that I came across had more or less similar kind of pattern to implement custom controllers.
Below are the docs that I referred:
- https://engineering.bitnami.com/articles/kubewatch-an-example-of-kubernetes-custom-controller.html
- https://github.com/bitnami-labs/kubewatch/blob/master/pkg/controller/controller.go
An additional tip would be to read the comments of all the functions and interfaces that you are using for writing the controller (especially from k8s.io/client-go/informers, k8s.io/client-go/tools/cache, k8s.io/client-go/util/workqueue libraries) as these are well documented and easy to understand.
Thanks for taking time to read this post!
Note: CloudLego is providing free training’s on Kubernetes, Terraform, Azure Devops and Azure Cloud. If you or any of your acquaintances are interested please drop an email to support@cloudlego.com and mention your interested topic in the subject line. You will then receive an email with training details. We are doing this particularly for people who are affected by current situation of Software industry but anyone who has an interest to learn is also welcome!
