A bit of context
I recently had the opportunity to work on the elaboration of a CI/CD pipeline based on the GitOps methodology. In this pipeline, code pushed by developers triggers the creation of namespaces for each development branch. Since several teams share the same Kubernetes cluster, RBAC rules are applied to each namespace. If the namespace is created by the front team, the back team should not have access to it. And of course, developers do not have the rights to create RBAC resources (otherwise it wouldn’t be fun :D).
I then took advantage of these specific needs to learn more about the Kubernetes controller pattern in order to automate the deployment of RBAC resources when namespaces are created. nsinjector is the result of my readings.
In this article, I explain how to use nsinjector to automatically deploy Kubernetes resources (in this case a
ClusterRole and a
ClusterRoleBinding) when a namespace with a name beginning by
dev- is created.
The rest of this post assumes a base knowledge of Kubernetes. If you want to know more about Kubernetes basics take a look at this doc.
We will first need a Kubernetes cluster for our tests. If you are here you probably have access to a cluster in one way or another, but if you need one, you can create it locally very quickly with minikube. And if you’re on MacOS it’s even simpler since Docker Desktop embeds a local Kubernetes cluster.
In order to interact with the cluster, we will use the command line tool
kubectl. Follow the official documentation for installation process.
Custom Resource Definition (CRD)
I won’t go into details here, but a controller is generally based on the definition of custom resources (CRD) which allow to extend the functionalities of Kubernetes beyond the native objects (Deployment, Service, Job, etc.). nsinjector therefore uses the CRD
NamespaceResourcesInjector to selectively deploy resource when a namespace is created.
To install this CRD, run the following command:
> kubectl apply -f https://raw.githubusercontent.com/blakelead/nsinjector/master/deploy/k8s/crd/namespaceresourcesinjector-crd-1.16.yaml
The second step is to deploy the controller. Since the controller needs to interact with a large number of resources, we first apply the following RBAC objects:
# rbac.yaml apiVersion: v1 kind: Namespace metadata: name: nsinjector-controller --- apiVersion: v1 kind: ServiceAccount metadata: name: nsinjector-controller namespace: nsinjector-controller --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: nsinjector-controller rules: - apiGroups: [""] resources: ["namespaces"] verbs: ["list", "get", "watch"] - apiGroups: ["blakelead.com"] resources: ["namespaceresourcesinjectors"] verbs: ["list", "get", "watch", "update"] - apiGroups: ["rbac"] resources: ["*"] verbs: ["list", "get", "watch", "update"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: nsinjector-controller subjects: - kind: ServiceAccount name: nsinjector-controller namespace: nsinjector-controller roleRef: kind: ClusterRole name: nsinjector-controller apiGroup: rbac.authorization.k8s.io
Then apply them:
> kubectl apply -f rbac.yaml
We can then deploy the controller itself:
# nsinjector-controller.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nsinjector-controller namespace: nsinjector-controller labels: app: nsinjector-controller spec: replicas: 1 selector: matchLabels: app: nsinjector-controller template: metadata: labels: app: nsinjector-controller spec: serviceAccountName: nsinjector-controller containers: - name: nsinjector-controller image: blakelead/nsinjector-controller:v0.1.0
Apply the file:
> kubectl apply -f nsinjector-controller.yaml
The controller is responsible for monitoring the creation of namespaces, and applying matching
NamespaceResourcesInjector objects. Here is an example of a `NamespaceResourcesInjector’ that can be applied:
# rbac-dev-inject.yaml kind: NamespaceResourcesInjector apiVersion: blakelead.com/v1alpha1 metadata: name: nri-test spec: namespaces: - dev-.* resources: - | apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: dev-role rules: - apiGroups: [""] resources: ["pods", "pods/portforward", "services", "deployments", "ingresses"] verbs: ["list", "get"] - apiGroups: [""] resources: ["pods/portforward"] verbs: ["create"] - apiGroups: [""] resources: ["namespaces"] verbs: ["list", "get"] - | apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: dev-rolebinding subjects: - kind: User name: dev roleRef: kind: Role name: dev-role apiGroup: rbac.authorization.k8s.io
Apply the file :
> kubectl apply -f rbac-dev-inject.yaml
This will instruct the controller to deploy the
dev-rolebinding objects when a namespace whose name matches the
dev-.* regex is created.
Create a namespace
Now if you create a namespace, you’ll see that the two resources will automatically be created:
> kubectl create namespace dev-namespace-1 namespace "dev-namespace-1" created > kubectl get clusterroles NAME CREATED AT dev-role 2020-10-12T16:50:53Z ... > kubectl get clusterrolebindings NAME ROLE AGE dev-rolebinding ClusterRole/dev-role 1m ...
This is the first article of my blog, and as such I’m sure there are some rough edges here and there. Please let me know what I did right or wrong in the comments below.
I would like to write another article following this one explaining in detail how I wrote the controller. It could serve as a Develop yout own Kubernetes controller tutorial. If it’s something you are interested in, let me know and I’ll put it in my backlog ;).