Illya's Blog

TL;DR;

Th Jenkins Kubernetes Plugin is a great tool to dynamically provision Jenkins slaves as pods on a Kubernetes Cluster(s). All you need to do is add and configure Kubernetes Cloud as part of the Jenkins configuration. While configuring Jenkins hosted on the Kubernetes cloud is very straight-forward in terms of credentials and accessibility, it may require additional steps if you are not running the Jenkins master on Kubernetes or would like to configure it for external Kubernetes cluster(s). This blog post describes the steps to take and pitfalls to avoid while configuring Kubernetes client certificates for the Jenkins Kubernetes Plugin.

Requirements

Things you will need:

1. Access to kube–admin Kubernetes configuration file (typically found in ~/.kube/config)
2. Jenkins Kubernetes Plugin 0.12

Installing the Plugin

Jenkins Kubernetes Plugin  (at the time of this writing) is at v0.12, and is available via Jenkins update site plugins. Installation is straight-forward and no different from other Jenkins plugins.

Configuring Kubernetes Cloud

Navigate to https://your-jenkins.com/configure and find “Add a new cloud” option

Configure your Kubernets Cluster (cloud).

Local Cluster

If your Jenkins Master is hosted on the same Kubernetes Cluster then all you need is to provide the Kubernetes URL for your local cluster as:

• short form: https://kubernetes
• long form: https://kubernetes.default.svc.%5Byour-domain%5D
  • your-domain is typically ‘cluster.local’
• you can find additional details on this in Kubernetest Plugin README

Click ‘Test Connection’ to verify the successful connection.

Remote Cluster

If you are not hosting Jenkins on the same Kubernetes cluster (or not hosting it on Kubernetes at all), then you need to perform a few extra steps to configure the access to your Kubernetes cluster.

If you have access to kube-admin configuration (typically found under ~/.kube/config) then you can use it to complete the Kubernetes cluster access setup.

Kubernetes server certificate key

Grab the ‘cluster: certificate-authority-data’ value from your ~/.kube/config file

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LSuperLongBase64EncodedString==

and decode it to get the X509 certificate
echo LSuperLongBase64EncodedString== | base64 -d > ca.crt
Your output should look something like

-----BEGIN CERTIFICATE-----
MIIAnotherSuperLongSeritificateValueString
-----END CERTIFICATE-----

Copy and paste the content of the ca.crt file into the Kubernetes server certificate. For this specific step we only need the certificate value, but the ca.crt file will be used in subsequent steps.

Without the server certificate, you may get the following SSL Error

You can disable https certificate check by selecting the check box.

Credentials

After you either provided the server certificate (or skipped the SSL check altogether), testing the connection may return following access error:

Now we can add the Kubernetes cluster credentials using Kubernetes user certificates (also found in the ~/.kube.config file)

First, we need to grab the base64 encoded client-certificate-data and client-key-data

user:
  client-certificate-data: LSuperLongBase64EncodedString==
  client-key-data: LAnotherSuperLongBase64EncodedString==

Using the same step as with ca.crt we will decode and create:
– client.crt with client-certificate-data
– client.key with client-key-data

Client P12 Certificate File

Using all three files we need to generate client certificate file in PKCS12 format

openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt

NOTE: It is important that you provide a passphrase (as you will see later)

At this point, you are ready to add a new Kubernetes client certificate to Jenkins.

Click Add -> Jenkins

Make sure Kind is set to Certificate

Select Upload PKCS#12 certificate and then hit Upload Certificate.

You should see a certificate file selector:

Navigate to the client.pfx file you generated and hit Upload.
Note: You will still see the message which you can ignore:

Enter the password you used for client.pfx . If you provided the correct password you should see the above error message (‘No certificate uploaded’) changed to a warning (‘Could retrieve key “1”. You may need to provide a password’). You can ignore this warning as well.

Complete the form with an ID and a description. I recommend using (or including) the Kubernetes cluster name as a part of both the ID and the description.

Hit Add and that’s all for creating the Kubernetes client certificate. The Jenkins Credential Provider window will close and you should return to the Configuration view.

Select the newly created certificate in the Credentials drop-down.

Now, hit Test Connection again. This time you should see Connection Successful message

PCKS Certificate Without Passphrase

If you set up the PCKS client certificate without a passphrase, Jenkins will not complain and will accept the certificate. However, using this certificate will result in a somewhat obscure error message:

Other tell-tell signs that your certificate wasn’t “successfully” accepted are:

• You won’t get the warning message ‘Could retrieve key “1”. You may need to provide a password’, and the error message ‘No certificate uploaded’ will remain
• The credentials drop-down box will not include ‘CN=kube-admin’ as a part of the certificate name.

Conclusion

Jenkins Kubernetes Plugin provides additional credentials mechanisms to authenticate against the Kubernetes cluster(s) like a Kubernetes service account

However, at this time I was able to configure Kubernetes Cloud credentials using client certificates only. That is not to say that Kubernetes service accounts don’t work, just that I didn’t figure how to get it going.

I hope you find the steps above helpful in configuring your Jenkins against Kubernetes cluster(s). Let me know if you find any inaccuracies or have any questions, comments or suggestions!