Th Jenkins Kubernetes Plugin is a great tool to dynamically provision Jenkins slaves as pods on a Kubernetes Cluster(s). All you need to do is add and configure Kubernetes Cloud as part of the Jenkins configuration. While configuring Jenkins hosted on the Kubernetes cloud is very straight-forward in terms of credentials and accessibility, it may require additional steps if you are not running the Jenkins master on Kubernetes or would like to configure it for external Kubernetes cluster(s). This blog post describes the steps to take and pitfalls to avoid while configuring Kubernetes client certificates for the Jenkins Kubernetes Plugin.
Things you will need:
- Access to kube–admin Kubernetes configuration file (typically found in
- Jenkins Kubernetes Plugin 0.12
Installing the Plugin
Jenkins Kubernetes Plugin (at the time of this writing) is at v0.12, and is available via Jenkins update site plugins. Installation is straight-forward and no different from other Jenkins plugins.
Configuring Kubernetes Cloud
Navigate to https://your-jenkins.com/configure and find “Add a new cloud” option
Configure your Kubernets Cluster (cloud).
If your Jenkins Master is hosted on the same Kubernetes Cluster then all you need is to provide the Kubernetes URL for your local cluster as:
- short form: https://kubernetes
- long form: https://kubernetes.default.svc.%5Byour-domain%5D
- your-domain is typically ‘cluster.local’
- you can find additional details on this in Kubernetest Plugin README
Click ‘Test Connection’ to verify the successful connection.
If you are not hosting Jenkins on the same Kubernetes cluster (or not hosting it on Kubernetes at all), then you need to perform a few extra steps to configure the access to your Kubernetes cluster.
If you have access to
kube-admin configuration (typically found under
~/.kube/config) then you can use it to complete the Kubernetes cluster access setup.
Kubernetes server certificate key
Grab the ‘cluster: certificate-authority-data’ value from your
apiVersion: v1 clusters: - cluster: certificate-authority-data: LSuperLongBase64EncodedString==
and decode it to get the
echo LSuperLongBase64EncodedString== | base64 -d > ca.crt
Your output should look something like
-----BEGIN CERTIFICATE----- MIIAnotherSuperLongSeritificateValueString -----END CERTIFICATE-----
Copy and paste the content of the
ca.crt file into the
Kubernetes server certificate. For this specific step we only need the certificate value, but the
ca.crt file will be used in subsequent steps.
Without the server certificate, you may get the following SSL Error
You can disable https certificate check by selecting the check box.
After you either provided the server certificate (or skipped the SSL check altogether), testing the connection may return following access error:
Now we can add the Kubernetes cluster credentials using Kubernetes user certificates (also found in the
First, we need to grab the base64 encoded
user: client-certificate-data: LSuperLongBase64EncodedString== client-key-data: LAnotherSuperLongBase64EncodedString==
Using the same step as with
ca.crt we will decode and create:
Client P12 Certificate File
Using all three files we need to generate client certificate file in PKCS12 format
openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt
NOTE: It is important that you provide a
passphrase (as you will see later)
At this point, you are ready to add a new Kubernetes client certificate to Jenkins.
Click Add -> Jenkins
Kind is set to
Select Upload PKCS#12 certificate and then hit Upload Certificate.
You should see a certificate file selector:
Navigate to the client.pfx file you generated and hit Upload.
Note: You will still see the message which you can ignore:
Enter the password you used for
client.pfx . If you provided the correct password you should see the above error message (‘No certificate uploaded’) changed to a warning (‘Could retrieve key “1”. You may need to provide a password’). You can ignore this warning as well.
Complete the form with an ID and a description. I recommend using (or including) the Kubernetes cluster name as a part of both the ID and the description.
Hit Add and that’s all for creating the Kubernetes client certificate. The Jenkins Credential Provider window will close and you should return to the Configuration view.
Select the newly created certificate in the Credentials drop-down.
Now, hit Test Connection again. This time you should see Connection Successful message
PCKS Certificate Without Passphrase
If you set up the PCKS client certificate without a passphrase, Jenkins will not complain and will accept the certificate. However, using this certificate will result in a somewhat obscure error message:
Other tell-tell signs that your certificate wasn’t “successfully” accepted are:
- You won’t get the warning message ‘Could retrieve key “1”. You may need to provide a password’, and the error message ‘No certificate uploaded’ will remain
- The credentials drop-down box will not include ‘CN=kube-admin’ as a part of the certificate name.
Jenkins Kubernetes Plugin provides additional credentials mechanisms to authenticate against the Kubernetes cluster(s) like a Kubernetes service account
However, at this time I was able to configure Kubernetes Cloud credentials using client certificates only. That is not to say that Kubernetes service accounts don’t work, just that I didn’t figure how to get it going.
I hope you find the steps above helpful in configuring your Jenkins against Kubernetes cluster(s). Let me know if you find any inaccuracies or have any questions, comments or suggestions!