How to implement a custom Kubernetes validation admission controller?
In my earlier post, I have outlined what is an admission controller, types of controllers shipped with Kubernetes and how to enable these controllers.
In this post we will see how to write a basic version of admission controller that will help us to understand the concepts. It does require a fair amount of knowledge of how REST endpoints work and how to create a minimal REST API server (preferably in Golang). You should also be familiar with Golang or at least know how to read the docs of a Kubernetes go packages and understand the code. Also, some hands on experience is required for generating self signed SSL certificates (there are lots of docs available on the internet).
Does that sound too much? Don’t panic. I will try to keep things as simple as possible.
Theory
In my previous post, I had mentioned that admission controllers are started by passing a list of controller names while starting the kube-apiserver. This raises a question that if controllers are registered during the kube-apiserver startup, how do we register custom controllers that are not shipped with Kubernetes and after the API server is already up and running?
The answer lies in the fact that there are 2 special admission controllers shipped with Kubernetes that enable the integration of additional admission controllers with Kubernetes API server. These are:
- ValidatingAdmissionWebhook
- MutatingAdmissionWebhook
Web-hooks? Yes. Kubernetes API server provides 2 types of web-hooks and depending on the type of admission controller that we are interested in, we can leverage any of these web hooks to integrate our custom admission controller with kube-apiserver.
If we could create a flow diagram for admission controller process, it would look like below:
Now, what kind of admission controllers you can create and integrate with Kubernetes? It can be any of the below:
- Validating Controllers: These custom admission controllers perform some kind of validation on the request object that was forwarded by api server and based on a logic, sends back a response to api server that contains information on whether to allow or reject the request. These controllers are registered with Kubernetes using the ValidatingAdmissionWebhook. How ? We will see that in a while.
- Mutating Controllers: These custom admission controllers on receiving a request, change few attribute values of the request ( mutates the request ) and then sends back the response that contains the mutated request object. These controllers are registered with Kubernetes using the MutatingAdmissionWebhook.
What will our custom admission controller do?
By default, if we deploy any object in Kubernetes and missed to specify the “Namespace”, that object will be created in the “default” namespace. We do not want that to happen, since the “default” namespace usually ends up with all sort of junk objects that were deployed only for testing purpose. Also it is good practice to deploy objects in a dedicated Namespace.
Our version of validator admission controller will intercept requests that are related to Pod creation operation and if no namespace is specified its going to reject the request straight away with a custom error message. Request of all other objects will be ignored by our admission controller and no action will be taken. Request for Pod object that is not a Create operation will also be ignored by this controller.
From on wards, we will call our custom admission controller as “Validator”.
Pre-flight Checklist
- A self managed Kubernetes cluster. During the development phase you will have to access kube-apiserver logs frequently to check entries created by the admission controller. A managed Kubernetes service (like AKS or EKS) will not provide easy access to API server logs.
- Generate Self signed SSL certificates. This is required since the communication between the controller and API server is SSL encrypted. (refer the Certificate Generation section of README.md file)
- Understand important Kubernetes go packages. Why? The admission controller has to parse the API request, extract required information, perform some logic and send back a response to API server. To do all this we need to know the structure of API request and how to use different interfaces and custom types to parse the request. Also we need to send the response in a structure that is understood by API server. The admission controller that we will create will make use of below Kubernetes packages:
k8s.io/api/admission/v1beta1 : We are interested in two custom types in this package:
- type AdmissionReview : We will use this type to parse the request and extract all the required fields on which we can perform some validation logic.
- type AdmissionResponse : After performing our validation logic, we need to send back a response object to API server. We will use this type to construct a response object that can be understood by API server.
k8s.io/api/core/v1 : This package holds the types and functions of core API objects. The type that we are interested in here is “type Pod” . We have to unmarshal the extracted object (which is Pod in our case) so that we can directly use the structure fields of a Pod object in our validation logic, example its name and namespace fields.
k8s.io/apimachinery/pkg/apis/meta/v1 : This package contains types that are common to all versions. We are interested in “type Status” . We will use this type to set a status in the response object.
3. Finally, loads of snacks, few cups of hot coffee or tea and frequent breaks. Most importantly Patience….we have to go through a lot of trial and error to get this working.
Get, Set…Code
The code is available under this git repository.
First and foremost we need a RESTful API server that exposes an end point (our controller exposes an endpoint at /validate ) and handles POST request. I have used Revel which is a good Golang web framework for building restful API servers.
So, do we have to stick to Golang for writing a custom admission controller? Not really. As far as your controller is able to parse the request object and construct a valid response object, you can use any development language you are comfortable with. However, note that Kubernetes is entirely developed in Golang and extending its functionality becomes easy and maintainable when written using Golang. Also, even if you choose to use a different programming language, you have to understand the structure of Kubernetes objects which is available only in Golang.
I will highlight important files available in the git repository that we would be working with:
- app.go contains the controller logic to parse and validate the request and send back the response.
In the above code block, we are parsing the request that was forwarded to validator by kube-apiserver. It basically binds the received object to a variable ( request ) of type AdmissionReview. We then attempt to unmarshal a portion of request that contains the information about the Pod object ( request.Request.Object.Raw). If we fail to de-serialize the object we respond back with a response of type AdmissionResponse and set Allowed field to true because we do not want the request to be rejected before we could validate it.
This is the code block where we are checking whether the received Pod object contains a Namespace set to “default” ( which would be automatically set by Kubernetes if namespace wasn’t provided during deployment ). If it is set to “default”, we are rejecting the request by setting the field Allowed as false and also a custom error message is set in the field Result.Message.
In the final code block, if the validation succeeds ( which will be true if the Namespace field was set to anything other than the value “default”) we respond back with Allowed field set to true and also perform a basic log entry at the controller end.
2. deployment.yaml file is a kubernetes manifest to deploy our admission controller as a deployment in the Kubernetes cluster. We also have to expose our pod on port 443, since its the port on which our controller will be listening for POST request.
Earlier we saw that admission controllers can be integrated using webhooks. But how does Kubernetes know about our admission controller and on which endpoint it should try to contact the controller (to make the POST request that forwards the request object to our controller)?
To inform Kubernetes about our admission controller, we have to create an object of kind “ValidatingWebhookConfiguration”. The file webhook.yaml contains all the details that is required to integrate the controller with Kubernetes.
Under the webhooks section, for service attribute we are specifying the name of the Kubernetes service attached to our controller pod, the namespace of the service and the path to which the POST request has to be done by API server. Also as explained earlier, communication between controller and API server is SSL encrypted, we have to pass a base64 encoded string of ca cert file generated earlier.( refer the Certificate Generation section of README.md file )
Under the rules, we are explicitly telling Kubernetes that this admission controller should be invoked only if the operation is of type CREATE and if the resource is a pod. Of course we can invoke it for multiple objects and for different kinds of operations too.
3. app.conf file is the configuration file for our Revel application. Unless you have made any additional changes to application configuration, the default setup will work without any issues. (refer the Modification (optional) section of README.md)
4. Use the Dockerfile to build an image and push it to repository. Make sure to edit the image name under deployment.yaml to point it to this image registry.
5. Follow the Certificate Generation section of README.md file and copy the validator-crt.pem and validator-key.pem files under the root directory.
Action
Once you have built the docker image, execute the below steps in sequence:
- Deploy the webhook.yaml manifest in your cluster. Navigate to the directory where you have cloned the repository and switch to k8 directory. Execute kubectl create -f webhook.yaml
- Deploy deployment.yaml manifest. This will create a deployment with name validator and a service validator that listens on port 443 (type as clusterIP). Execute kubectl create -f deployment.yaml
- Once done, you can verify if the objects were created
4. Try to deploy a test pod in “default” namespace. You should see an error as below:
5. API server logs for this CREATE operation:
W1209 18:50:08.657435 1 dispatcher.go:142] rejected by webhook “validator.default.svc”: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:””, APIVersion:””}, ListMeta:v1.ListMeta{SelfLink:””, ResourceVersion:””, Continue:””, RemainingItemCount:(*int64)(nil)}, Status:”Failure”, Message:”admission webhook \”validator.default.svc\” denied the request: Deployments in ‘default’ namespace is restricted by cluster admin!”, Reason:””, Details:(*v1.StatusDetails)(nil), Code:400}}
6. But you will be able to deploy a Pod in other namespace as shown below:
Time to Relax
If you got the about output then, Congrats! you have just developed a Kubernetes admission controller. If not then let me know where you got stuck and probably I can help you out.
Most of the Kubernetes tools that perform some kind of validation on the API request uses the above technique, example OPA gatekeeper for policy validation.
But do we really need to build one? As usual it depends on what kind of problem the admission controller is going to solve. Probably the best thing might be to check the CNCF projects to see if there is a tool that solves your problem. If not then feel free to implement one for you! Beware that custom admission controller might have side effects if the logic is not implemented properly so be sure to do lot of testing.
So, was it easy? I don’t think so, at least not for me. However I found it very interesting, challenging and satisfying to see that it worked at last.
Let me know if I missed out to mention something. Your feed backs are always welcome.
Note: CloudLego is providing free training’s on Kubernetes, Terraform, Azure Devops and Azure Cloud. If you or any of your acquaintances are interested please drop an email to support@cloudlego.com and mention your interested topic in the subject line. You will then receive an email with training details. We are doing this particularly for people who are affected by current situation of Software industry but anyone who has an interest to learn is also welcome!
