Certs

Kubernetes uses SSL certificates to encrypt and authenticate calls. In this blog, I demonstrate how a major part of it works and create a general idea on top of said demonstrations. As the main scenario of the blog, I am handling certificate generation as an engineer who is trying to bootstrap a cluster from zero.

Bootstrapping a Kubernetes cluster requires generating various certificates. This process is typically handled by kubeadm. But because I’m a blockhead and refuse to use it (for this post) — in favor of that good old sweet masochistic pain (and maybe some learning opportunities) — I have to do a lot of manual certificate generation and explanations. After the initial bootstrap, Kubernetes will take over the management of these certificates, including their rotation and the creation of new ones when the cluster expands. But before all of that good stuff, we have to suffer.

Okay, first we need to understand what actually needs to be generated. Usually, we’d check with the vendor for specifications on things like this, but due to what we’re doing and how we’re doing it, that information will mostly have to be gathered through trial and error.

But we have to start somewhere. On the Kubernetes website, there is a guide on providing certificates to kubeadm. This is available as a customization option for entities that want to generate certificates using their internal Certificate Authority services. kubeadm allows this customization in two ways:

1. Partial mode — by providing an intermediate CA certificate that will be used to generate the required certificates.
2. Full manual mode — where each certificate must be provided separately.

This seems to be as close as we get to an official guide on the required certificates. and by this, i do mean reverse enginnering tool that is created to make cluster provisioning esier, so i can then do it in a ridiculusly hard way. : // Source: https://kubernetes.io/docs/setup/best-practices/certificates/#all-certificates

source provides list of items with general descriptions and few tables, while they are not complete by any means, they do provide a basic idea, and with some injection of my personal experience and much more subject specific documentations, i should manage to create some aggregated information.

lets dive down

according to source we will need;

• Server certificate for the API server endpoint
• Server/Client certificate for the etcd server
• Server/Client(Peer) Certificate for the etcd clustering
• Server certificates for each kubelet, exposing metrics and runtime data
• Optional server certificate for the front-proxy
• Client certificates for each kubelet, used to authenticate to the API server as a client of the Kubernetes API
• Client certificate for each API server, used to authenticate aginst etcd
• Client certificate for the controller manager to authenticate with the API server
• Client certificate for the scheduler to authenticate with the API server
• Client certificates, for kubelet to authenticate with API server
• Client certificates for administrators of the cluster

basiclly provides following table:

and bounch of client certificates…

So what does all of this mean? oh hell, i don’t really know, well i do but putting it into easy to understand words makes my head feel like a kettle. so lets draw some stuff.

Starting with etcd: we have two types of connections to illustrate here — the clustering between etcd nodes, and the clients connecting to that cluster.
Each control node runs an etcd instance, so we end up with three instances that communicate with each other bidirectionally.
Then we have the client — in our case, the API server — talking to etcd.
Depending on how the API server is configured, it could connect to an etcd load balancer, all etcd endpoints at once, or a locally hosted etcd endpoint if it’s running on the same control node.
For the sake of simplicity, let’s assume each control node also runs the API server, and it communicates directly with the local etcd instance.

etcd listens on two ports:

• 2379 (via IP/localhost) for client connections
• 2380 (via IP) for peer communication

This means each etcd instance should be configured with a unique certificate that validates the following:

• 127.0.0.1
• the node’s IP address
• the node’s given name

and we get somthing like this;

this looks easy and simple enough, right?
No. Of course, it’s never that simple : /

Closely looking at the drawing, it should be relatively easy to spot a few problems.
Like — why do we have the exact same certificate used for both peer and server roles?
Well, technically they are separate, but considering their content, they might as well be the same.
They share:

• Certificate extensions: both server and client
• CN and IP fields
• The signer!

It shouldn’t be like this.
Assuming the drawing is correct with the presented connections (which it should be, based on standard etcd/Kubernetes cluster layouts), the certificates should be more along the lines of:

• 3 peer certificates, with:

  • Certificate extensions set to both client and server
  • CN set to the FQDN
  • IP set to the node’s IP

• 3 server certificates, with:

  • Certificate extension set to server
  • CN set to the FQDN
  • IPs set to the node’s IP and 127.0.0.1

Now this looks more like what I had in mind, and it aligns with etcd docs/requirements: etcd doc

But this is different from kubeadm…
Why does Kubernetes overreach with its requirements? No clue.
Could be a simplification to cover a broader audience — like allowing a server cert to also be used by the CLI (e.g., etcdctl).
Maybe it’s just an oversight, either in the documentation or in kubeadm itself.
I really don’t know nor checked enough to give a concrete answer.

Blockhead’s NOTE:
The client extension in the server certificate is actually a requirement of etcd.
I’m not entirely sure why, as it really shouldn’t be needed — but it does require it.
Otherwise, it simply refuses to use the certificate, due to some kind of pre-check in the underlying library.

What will I do?
Well, I’m a blockhead — so it should be obvious.
I’m going to follow my approach, since it has fewer requirements — even if, in practice, it doesn’t change much.

But we’re still not ready to generate any certificates yet.
The drawing I did was only for etcd — we need to do the same for other components.

Using the documentation as a guideline, we first have to write up the connections for each component. We get something like this:

• API server to Kubelet — used to interact with the runtime, for example: log inspection, pod attachment, port forwarding, etc.
  There are two certificates present here: one for the Kubelet (which exposes an HTTPS endpoint), and a client certificate for the API server.

• Kube clients — this means any Kubernetes consumer, including internal services like the Scheduler and Controller.
  This is really simple: they are regular consumers, and as such, are served by the same server certificate that the API server exposes.
  Their authentication depends on enabled role management, like RBAC.

Well, this seems pretty simple.
Basically, by the nature of Kubernetes architecture — which they describe as a “hub and spoke” — the majority of connections terminate at the API server.
So, other than a few scenarios (in a standard deployment), we mostly just have a bunch of normal Kubernetes API consumers, like kubectl.

But I want to discuss this further with some examples along the way.

First of all, what does the “hub and spoke” approach really mean?
Basically, all of the components call the API server to provide their respective services.
We can demonstrate this with a few examples:

Let’s do the Scheduler and kubectl.

kubectl should be the easiest to understand.
When we run a kubectl command, it’s actually sending an HTTP call to the API server.
This could be a direct call to an IP or a call routed through a load balancer.
This applies to pretty much every interaction:

You want to get a list of pods? → HTTP call to the API server.
You want to create a deployment? → Again, HTTP call to the API server.

Even if the API server itself doesn’t actually start the pod, the action still begins there.

The Scheduler is similar to kubectl — it’s a client to the API server, just a regular client, nothing more.
It’s simply a program sitting and watching to see if there’s a need to schedule a new pod.
It does this by watching a specific resource, similar to how we can watch a list of pods with kubectl get pods -w.

When a new update is received — like a Deployment being created — it fetches that Deployment and starts analyzing how it should be scheduled.
Once it makes a decision, it calls the API server again with scheduling directives.

Its authentication is similar to kubectl as well: a kubeconfig file containing a user identity and certificate.

see how there’s no connection from user to scheduler even though it definitely contributes to the user’s request?

We should also discuss how configuration is provided, and how it affects a component’s choice of preferred certificate — both client and trust store.

Kubernetes components handle configuration via environment variables, command-line parameters, or configuration files.
For example, I’m running them as static pods and specifying parameters directly in their respective YAML files.
Here’s an example using the kube-apiserver manifest:

  - command:
    - kube-apiserver
    - --advertise-address=10.10.16.12
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=Hostname,InternalDNS,InternalIP
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/16
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

We have quite a few parameters here. I’ll provide some descriptions for the parameters of interest, but please check out the Kubernetes documentation for more accurate and broader information:

• --etcd-cafile: This parameter defines the whitelist of the trust chain for the etcd endpoint. This file is used to verify etcd as a trusted endpoint and helps ensure we’re not being misled by a man-in-the-middle actor when interacting with etcd.
• --etcd-certfile, --etcd-keyfile: These parameters control the client certificate sent to etcd for authentication.
• --kubelet-client-certificate, --kubelet-client-key: These are used to authenticate the API server’s requests sent to the kubelet endpoint.
• --kubelet-preferred-address-types: This sets the order in which the API server will try to connect to the endpoint until a successful connection is made, e.g., Hostname > InternalDNS > InternalIP.
• --service-account-key-file, --service-account-issuer, --service-account-signing-key-file: These specify information for service account token signing and verification.
• --tls-private-key-file, --tls-cert-file: These specify the certificate used for the API server’s HTTPS endpoint.

So basically, when the API server connects to the etcd service, it verifies it against the file provided via --etcd-cafile and presents the client certificate provided via --etcd-certfile. Pretty simple, right?

This also means we can fully use a separate CA for etcd — or for pretty much any other connection — as long as that connection allows specifying separate trust/certificate files.

This also explains where requirements for CN, SAN, and IP fields come from. For example, if we connect to etcd via 127.0.0.1 (https://127.0.0.1:2379), then we must include 127.0.0.1 in the IP field of the certificate. If we used the internal IP instead, we could skip that addition — assuming there’s no other service using the same IP for communication. And so on…

All of this should raise one major question. Everything so far explains how a call is authenticated, meaning how Kubernetes determines we are legitimate — but what about authorization?
In other words: what permissions should the authenticated identity actually have?

Well, this depends on our configuration — specifically, the value of --authorization-mode.
Currently, we have two values set:

• Node: Allows kubelet the necessary permissions to do its job as a Kubernetes node.
• RBAC: Enables roles, cluster roles, bindings, etc.

The interesting part here is RBAC.

Client certificates, after authentication, are authorized via RBAC in the same way a user would be.
The catch is how identity is assigned to the certificate, which is determined via the certificate’s subject fields — specifically the CN and O fields.
This means that if a presented client certificate contains CN=bob.smith,O=developers, the API server will interpret the username as bob.smith and the group as developers.
From there, RBAC will grant the appropriate permissions.

I also have to highlight some quirks with trust stores. TLS is a standard, but how it is implemented can differ between services. For example, let’s present a scenario:

We have a root-ca and two intermediates: etcd-ca and kube-ca.

When we start etcd and pass both root-ca and etcd-ca as trusted certificates, it will authenticate certificates signed by etcd-ca, which is what we want.
However, it will also authenticate certificates signed by kube-ca.

Why? Because we trust root-ca, and kube-ca is signed by root-ca — so the trust chain is valid.

This defeats the purpose of splitting up intermediates.

Now, we do have the option of trusting only the intermediate (etcd-ca).
This approach will allow us to specifically trust only certificates signed by etcd-ca and reject everything else — including those signed by root-ca.
But that defeats the purpose of having a root-ca that signs etcd-ca.

Why is this the case?
Well, that’s just how TLS works.
There are other methods a service might use to implement TLS — for example, some services allow you to specify an issuer during validation.
This lets you trust both the intermediate and the root, with client trust set to specific issuer(intermediate). The intermediate will have the exclusive ability to sign client certificates, while the root still retains the ability to regenerate a new intermediate — as long as it matches the whitelisted issuer.

But again, this all depends on how TLS is implemented for that specific service.

Note I actually haven’t tested etcd specifically with this scenario, so I can’t claim it works exactly like this.
Please treat this as a general example of how a service might implement TLS — not how etcd does.

I hope this sheds some light. now lets create new certificate table

Okay, we can start generating now. I’ve chosen Terraform as my poison for provisioning.
First step is integrating TLS into it. Hopefully, Terraform provides TLS management…
Lucky me — it does: https://registry.terraform.io/providers/hashicorp/tls/latest

First, we have to create CAs.
The number and type (root/intermediate) will depend on how many clusters you have and how you plan to manage them later on.
I won’t be able to cover all possible scenarios, but it should be pretty straightforward to understand how this should be architected to suit a specific setup.
Really — if you’ve made it this far and understood everything, then you should be more than fine.

For this blog, I’ll keep it simple while still trying to cover a wide range of scenarios.
I’ll generate one root CA and two intermediate CAs — one for etcd and one for kube.

Note All of this will be stored in the tfstate file.
If the state is distributed/shared, it will be easy to extract your CA keys.

first root, self-signed since it’s the first in the chain.

resource "tls_private_key" "root-ca" {
  algorithm = "RSA"
  rsa_bits  = 4096
}


resource "tls_self_signed_cert" "root-ca" {
   private_key_pem = tls_private_key.root-ca.private_key_pem

   is_ca_certificate = true

   subject {
     common_name  = "root-ca"
   }
    
  validity_period_hours = 43830 #5 Years.

  allowed_uses = [
    "cert_signing",
    "crl_signing",
    "digital_signature",
  ]
}

now we can sign intermediates.

resource "tls_private_key" "etcd-ca" {
  algorithm = "RSA"
  rsa_bits  = 4096
}
  
resource "tls_cert_request" "etcd-ca" {
  private_key_pem = tls_private_key.etcd-ca.private_key_pem

  subject {
    common_name  = "etcd-ca"
  }
}

resource "tls_locally_signed_cert" "etcd-ca" {
  cert_request_pem   = tls_cert_request.etcd-ca.cert_request_pem
  ca_private_key_pem = tls_private_key.root-ca.private_key_pem
  ca_cert_pem        = tls_self_signed_cert.root-ca.cert_pem

  is_ca_certificate = true

  validity_period_hours = 26298

  allowed_uses = [
    "cert_signing",
    "crl_signing",
    "digital_signature",
  ]
}

resource "tls_private_key" "kubernetes-ca" {
  algorithm = "RSA"
  rsa_bits  = 4096
}
  
resource "tls_cert_request" "kubernetes-ca" {
  private_key_pem = tls_private_key.kubernetes-ca.private_key_pem

  subject {
    common_name  = "kubernetes-ca"
  }
}

resource "tls_locally_signed_cert" "kubernetes-ca" {
  cert_request_pem   = tls_cert_request.kubernetes-ca.cert_request_pem
  ca_private_key_pem = tls_private_key.root-ca.private_key_pem
  ca_cert_pem        = tls_self_signed_cert.root-ca.cert_pem

  is_ca_certificate = true

  validity_period_hours = 26298

  allowed_uses = [
    "cert_signing",
    "crl_signing",
    "digital_signature",
  ]
}

lets do actual certificates now; starting with etcd..

resource "tls_private_key" "etcd-server" {
  for_each = var.control-instances
  algorithm = "RSA"
  rsa_bits  = 2048
}

resource "tls_private_key" "etcd-peer" {
  for_each = var.control-instances
  algorithm = "RSA"
  rsa_bits  = 2048
}



resource "tls_cert_request" "etcd-server" {
  for_each = var.control-instances
  private_key_pem = tls_private_key.etcd-server[each.key].private_key_pem 

  subject {
    common_name  = each.key
    organization = "Kubelius totallyacorp"
  }
  ip_addresses = [each.value["ip"], "127.0.0.1"]
  dns_names = [each.key]
}

resource "tls_cert_request" "etcd-peer" {
  for_each = var.control-instances
  private_key_pem = tls_private_key.etcd-peer[each.key].private_key_pem 

  subject {
    common_name  = each.key
  }
  ip_addresses = [each.value["ip"]]
  dns_names = [each.key]
}


resource "tls_locally_signed_cert" "etcd-server" {
  for_each = var.control-instances
  cert_request_pem   = tls_cert_request.etcd-server[each.key].cert_request_pem
  ca_private_key_pem = tls_private_key.etcd-ca.private_key_pem
  ca_cert_pem        = tls_locally_signed_cert.etcd-ca.cert_pem

  validity_period_hours = 8766

  allowed_uses = [
    "digital_signature",
    "server_auth",
    "client_auth",
    "key_encipherment",
  ]
}

resource "tls_locally_signed_cert" "etcd-peer" {
  for_each = var.control-instances
  cert_request_pem   = tls_cert_request.etcd-peer[each.key].cert_request_pem
  ca_private_key_pem = tls_private_key.etcd-ca.private_key_pem
  ca_cert_pem        = tls_locally_signed_cert.etcd-ca.cert_pem

  validity_period_hours = 8766

  allowed_uses = [
    "digital_signature",
    "server_auth",
    "client_auth",
    "key_encipherment",
  ]
}

resource "tls_private_key" "kube-apiserver-etcd-client" {
  algorithm = "RSA"
  rsa_bits  = 2048
}

resource "tls_cert_request" "kube-apiserver-etcd-client" {
  private_key_pem = tls_private_key.kube-apiserver-etcd-client.private_key_pem 

  subject {
    common_name = "kube-apiserver-etcd-client"
  }

}

resource "tls_locally_signed_cert" "kube-apiserver-etcd-client" {
  cert_request_pem   = tls_cert_request.kube-apiserver-etcd-client.cert_request_pem
  ca_private_key_pem = tls_private_key.etcd-ca.private_key_pem
  ca_cert_pem        = tls_locally_signed_cert.etcd-ca.cert_pem

  validity_period_hours = 8766

  allowed_uses = [
    "digital_signature",
    "client_auth",
    "key_encipherment",
  ]
}

API server and Kubelet access

resource "tls_private_key" "kube-apiserver" {
  for_each = var.control-instances
  algorithm = "RSA"
  rsa_bits  = 2048
}

resource "tls_private_key" "kube-apiserver-kubelet" {
  algorithm = "RSA"
  rsa_bits  = 2048
}

resource "tls_cert_request" "kube-apiserver" {
  for_each = var.control-instances
  private_key_pem = tls_private_key.kube-apiserver[each.key].private_key_pem 

  subject {
    common_name  = each.key
  }
  ip_addresses = [each.value["ip"], "10.10.16.10"]
  dns_names = [each.key, "api.kubelius", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster.local"]
}
resource "tls_cert_request" "kube-apiserver-kubelet" {
  private_key_pem = tls_private_key.kube-apiserver-kubelet.private_key_pem 

  subject {
    common_name  = "kube-apiserver-kubelet"
    organization = "system:masters" #this is overkill far as permissions go
  }
}
resource "tls_locally_signed_cert" "kube-apiserver" {
  for_each = var.control-instances
  cert_request_pem   = tls_cert_request.kube-apiserver[each.key].cert_request_pem
  ca_private_key_pem = tls_private_key.kubernetes-ca.private_key_pem
  ca_cert_pem        = tls_locally_signed_cert.kubernetes-ca.cert_pem

  validity_period_hours = 8766

  allowed_uses = [
    "digital_signature",
    "server_auth",
    "key_encipherment",
  ]
}
resource "tls_locally_signed_cert" "kube-apiserver-kubelet" {
  cert_request_pem   = tls_cert_request.kube-apiserver-kubelet.cert_request_pem
  ca_private_key_pem = tls_private_key.kubernetes-ca.private_key_pem
  ca_cert_pem        = tls_locally_signed_cert.kubernetes-ca.cert_pem

  validity_period_hours = 8766

  allowed_uses = [
    "digital_signature",
    "client_auth",
    "key_encipherment",
  ]
}

Client Certificates for Admin, scheduler, controller, kubelet ..

resource "tls_private_key" "kube-super-admin" {
  algorithm = "RSA"                                                                                                             
  rsa_bits  = 2048                                              
}

resource "tls_cert_request" "kube-super-admin" {
  private_key_pem = tls_private_key.kube-super-admin.private_key_pem

  subject {
    common_name  = "kubernetes-super-admin"
    organization = "system:masters"
  }
}


resource "tls_private_key" "kubelet" {
  for_each = var.control-instances
  algorithm = "RSA"                                                                                                             
  rsa_bits  = 2048                                              
}

resource "tls_private_key" "scheduler" {
  algorithm = "RSA"                                                                                                             
  rsa_bits  = 2048                                              
}

resource "tls_private_key" "controller" {
  algorithm = "RSA"                                                                                                             
  rsa_bits  = 2048                                              
}

resource "tls_cert_request" "kubelet" {
  for_each = var.control-instances
  private_key_pem = tls_private_key.kubelet[each.key].private_key_pem

  subject {
    common_name  = "system:node:${each.key}"
    organization = "system:nodes"
  }
}

resource "tls_cert_request" "scheduler" {
  private_key_pem = tls_private_key.scheduler.private_key_pem

  subject {
    common_name  = "system:kube-scheduler"
  }
}

resource "tls_cert_request" "controller" {
  private_key_pem = tls_private_key.controller.private_key_pem

  subject {
    common_name  = "system:kube-controller-manager"
  }
}

resource "tls_locally_signed_cert" "kubelet" {
  for_each = var.control-instances
  cert_request_pem   = tls_cert_request.kubelet[each.key].cert_request_pem
  ca_private_key_pem = tls_private_key.kubernetes-ca.private_key_pem
  ca_cert_pem        = tls_locally_signed_cert.kubernetes-ca.cert_pem

  validity_period_hours = 87600

  allowed_uses = [
    "digital_signature",
    "client_auth",
    "key_encipherment",
  ]
}

resource "tls_locally_signed_cert" "scheduler" {
  cert_request_pem   = tls_cert_request.scheduler.cert_request_pem
  ca_private_key_pem = tls_private_key.kubernetes-ca.private_key_pem
  ca_cert_pem        = tls_locally_signed_cert.kubernetes-ca.cert_pem

  validity_period_hours = 87600

  allowed_uses = [
    "digital_signature",
    "client_auth",
    "key_encipherment",
  ]
}

resource "tls_locally_signed_cert" "controller" {
  cert_request_pem   = tls_cert_request.controller.cert_request_pem
  ca_private_key_pem = tls_private_key.kubernetes-ca.private_key_pem
  ca_cert_pem        = tls_locally_signed_cert.kubernetes-ca.cert_pem

  validity_period_hours = 87600

  allowed_uses = [
    "digital_signature",
    "client_auth",
    "key_encipherment",
  ]
}


resource "tls_locally_signed_cert" "kube-super-admin" {
  cert_request_pem   = tls_cert_request.kube-super-admin.cert_request_pem
  ca_private_key_pem = tls_private_key.kubernetes-ca.private_key_pem
  ca_cert_pem        = tls_locally_signed_cert.kubernetes-ca.cert_pem

  validity_period_hours = 87600

  allowed_uses = [
    "digital_signature",
    "client_auth",
    "key_encipherment",
  ]
}

At this stage, we should be able to provision the cluster. Of course, I have not covered specific deployment scenarios, and my client certificates definitely overreach with their RBAC requirements, but it is totally possible to bootstrap the cluster with this.

Okay, now that we understand the general idea of how certificates are used and how to generate them, we have to start thinking about managing them after the initial generation/usage.

At the start of this post, I mentioned that Kubernetes will manage certificates for us after it is up and running. Well, the controller certainly has the ability to handle certificate requests, but it is never that easy, is it?

The controller implements Kubernetes resources to request and sign certificates. More on that can be found at https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/

How does this work? Well, we basically provide our CA key and certificate to the controller, which has built-in configuration for specific “signers.” Each so-called signer handles requests by predefined logic that falls under characteristics such as:

• Who needs to trust this?
• What is the permitted scope?
• What are the permitted extensions and key usage?
• Duration of the signature
• Can it be auto-approved?

Most of this is not actually important for the consumer of the service. Much of it is exposed for the future to allow third-party providers to handle signatures in a standardized manner. So, let’s focus on the important part, which is the source of the request and the approval procedure.

Let’s take a look at well-known signers:

kubernetes.io/kube-apiserver-client: Handles client certificates that authenticate calls against the API server. This signer is set to manual approval only.

kubernetes.io/kube-apiserver-client-kubelet: Handles client certificates for the kubelet service that authenticate calls against the API server. These can be, and usually are, auto-approved.

kubernetes.io/kubelet-serving: Handles server certificates for the kubelet listener. Manual approvals only.

So, as an example, if I create a CSR request for the API server’s client certificate for user bob with the group set to bobsters, I have to register the following request:

openssl req -newkey rsa:2048 --nodes --subj "/CN=bob/O=bobsters" -keyout bob.key -out bob.csr 

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: csr-bob
spec:
  request: <base64 csr>
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth
  - digital signature

request will appear under CertificateSigningRequestkubectl get csr, and can be manualy approved using kubectl certificate approve

after approval i can retrive certificate form status of signed csr, kubectl get csr csr-bob -o json | jq .status.certificate

rottation of certificates follows same approach, does kubelet requre new serving certificate ? it will register csr, which based on policy will be autoapproved, kubelet will retrive certificate and start using it.

ofcourse there are other querks to it, like how service has to explicitly implement rotation like kubernetes expects it; check if certiifcate is expiring, generate key and csr, register, wait for approval, reload ceritifate etc.. well, i am not going to get into service specifics, core services like kubelet has them and can be simple as adding flag to runtime rotateCertificates: true, and other services sdn and service mesh are too subject specific to cover under this post, this is true for even reletivly sreight forward service like etcd, which does not implement it at all, meaning we have to externaly do it via scripts, operators etc..

Afterwords

This post is part of a bigger project where I implement Kubernetes from scratch, with lots of opinions and explanations of how everything works. As the project is big, with no clearly defined deadline or any real boundaries, I will be posting parts of the main work separately as blog posts.