Configuring Certificates for Jenkins Kubernetes Plugin 0.12

TL;DR;

Th Jenkins Kubernetes Plugin is a great tool to dynamically provision Jenkins slaves as pods on a Kubernetes Cluster(s). All you need to do is add and configure Kubernetes Cloud as part of the Jenkins configuration. While configuring Jenkins hosted on the Kubernetes cloud is very straight-forward in terms of credentials and accessibility, it may require additional steps if you are not running the Jenkins master on Kubernetes or would like to configure it for external Kubernetes cluster(s). This blog post describes the steps to take and pitfalls to avoid while configuring Kubernetes client certificates for the Jenkins Kubernetes Plugin.

Requirements

Things you will need:

  1. Access to kubeadmin Kubernetes configuration file (typically found in ~/.kube/config)
  2. Jenkins Kubernetes Plugin 0.12

Installing the Plugin

Jenkins Kubernetes Plugin  (at the time of this writing) is at v0.12, and is available via Jenkins update site plugins. Installation is straight-forward and no different from other Jenkins plugins.

Configuring Kubernetes Cloud

Navigate to https://your-jenkins.com/configure and find “Add a new cloud” option

AddNewCloud

Configure your Kubernets Cluster (cloud).

Local Cluster

If your Jenkins Master is hosted on the same Kubernetes Cluster then all you need is to provide the Kubernetes URL for your local cluster as:

Click ‘Test Connection’ to verify the successful connection.

KubernesCloudConfigurationLocal

Remote Cluster

If you are not hosting Jenkins on the same Kubernetes cluster (or not hosting it on Kubernetes at all), then you need to perform a few extra steps to configure the access to your Kubernetes cluster.

If you have access to kube-admin configuration (typically found under ~/.kube/config) then you can use it to complete the Kubernetes cluster access setup.

Kubernetes server certificate key

Grab the ‘cluster: certificate-authority-data’ value from your ~/.kube/config file

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LSuperLongBase64EncodedString==

and decode it to get the X509 certificate
echo LSuperLongBase64EncodedString== | base64 -d > ca.crt
Your output should look something like

-----BEGIN CERTIFICATE-----
MIIAnotherSuperLongSeritificateValueString
-----END CERTIFICATE-----

Copy and paste the content of the ca.crt file into the Kubernetes server certificate. For this specific step we only need the certificate value, but the ca.crt file will be used in subsequent steps.

Without the server certificate, you may get the following SSL Error
KubernesCloudConfigurationSSLError.png
You can disable https certificate check by selecting the check box.

Credentials

After you either provided the server certificate (or skipped the SSL check altogether), testing the connection may return following access error:

KubernesCloudConfigurationAccessError.png

Now we can add the Kubernetes cluster credentials using Kubernetes user certificates (also found in the ~/.kube.config file)

First, we need to grab the base64 encoded client-certificate-data and client-key-data

user:
  client-certificate-data: LSuperLongBase64EncodedString==
  client-key-data: LAnotherSuperLongBase64EncodedString==

Using the same step as with ca.crt we will decode and create:
client.crt with client-certificate-data
client.key with client-key-data

Client P12 Certificate File

Using all three files we need to generate client certificate file in PKCS12 format

openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt

NOTE: It is important that you provide a passphrase (as you will see later)

At this point, you are ready to add a new Kubernetes client certificate to Jenkins.

KubernesCloudConfigurationAddCredentials

Click Add -> Jenkins

Make sure Kind is set to Certificate

Screen Shot 2017-08-25 at 6.08.36 PM

Select Upload PKCS#12 certificate and then hit Upload Certificate.

You should see a certificate file selector:

Screen Shot 2017-08-25 at 6.09.28 PM.png

Navigate to the client.pfx file you generated and hit Upload.
Note: You will still see the message which you can ignore:
Screen Shot 2017-08-25 at 6.17.19 PM

Enter the password you used for client.pfx . If you provided the correct password you should see the above error message (‘No certificate uploaded’) changed to a warning (‘Could retrieve key “1”. You may need to provide a password’). You can ignore this warning as well.

Complete the form with an ID and a description. I recommend using (or including) the Kubernetes cluster name as a part of both the ID and the description.

Screen Shot 2017-08-25 at 6.23.56 PM.png

Hit Add and that’s all for creating the Kubernetes client certificate. The Jenkins Credential Provider window will close and you should return to the Configuration view.

Select the newly created certificate in the Credentials drop-down.

Screen Shot 2017-08-25 at 6.25.52 PM

Now, hit Test Connection again. This time you should see Connection Successful message

Screen Shot 2017-08-25 at 6.28.21 PM

PCKS Certificate Without Passphrase

If you set up the PCKS client certificate without a passphrase, Jenkins will not complain and will accept the certificate. However, using this certificate will result in a somewhat obscure error message:

Screen Shot 2017-08-25 at 6.32.47 PM

Other tell-tell signs that your certificate wasn’t “successfully” accepted are:

  • You won’t get the warning message ‘Could retrieve key “1”. You may need to provide a password’, and the error message ‘No certificate uploaded’ will remain
  • The credentials drop-down box will not include ‘CN=kube-admin’ as a part of the certificate name.

Conclusion

Jenkins Kubernetes Plugin provides additional credentials mechanisms to authenticate against the Kubernetes cluster(s) like a Kubernetes service account

Screen Shot 2017-08-25 at 7.03.46 PM

However, at this time I was able to configure Kubernetes Cloud credentials using client certificates only. That is not to say that Kubernetes service accounts don’t work, just that I didn’t figure how to get it going.

I hope you find the steps above helpful in configuring your Jenkins against Kubernetes cluster(s). Let me know if you find any inaccuracies or have any questions, comments or suggestions!

 


22 thoughts on “Configuring Certificates for Jenkins Kubernetes Plugin 0.12

  1. Ilya, thank you for this post, you definitely should contribute to jenkinsci/kubernetes-plugin README, as example of external kube cluster config. I lost about 2 days to find out how to config plugin with my externally configured minikube

    Liked by 2 people

  2. Very useful, I also spent many hours trying to understand how to accomplish this. I agree that this should be added to the jenkinsci/kubernetes-plugin README. Thanks!

    Liked by 1 person

  3. i have question.. i am not getting in the kind kubernetes service account since my cluster is outside.. jenkins is not running in side the cluster for this i need to add certificate and all…
    without adding certificate and credentials my pod is running in kubernetes but jenkins master is not able to take slave pod for my job. i am not getting what to do?????

    Like

  4. Hai I am still getting error while trying to connect to Jenkins.. cluster is remote.. and in your steps passphrase you mentioned how to get that passphrase I didn’t understand clearly..
    Last step also credentials showing kubernetes service account you got.. that’s commes only when Jenkins is running inside the cluster right?…

    I have one more doubt
    I have one more Jenkins which connection is successful but after pod is running successfull then Jenkins not able to run the job on that pod?? What would be the issue..
    I am not getting any logs???….

    Like

  5. Awesome tutorial, thank you very much!

    I just got one probleme which made me waste several hours, but I finally solved it: I am behind a corporate proxy, so I needed to add my kubernetes API host to the Jenkins master JVM “no_proxy” option:

    -Dhttp.nonProxyHosts=myk8smaster.domain.tld|someotherhost.tld

    (despite its name, this option is used both for http and https requests)

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s