TL;DR;
Th Jenkins Kubernetes Plugin is a great tool to dynamically provision Jenkins slaves as pods on a Kubernetes Cluster(s). All you need to do is add and configure Kubernetes Cloud as part of the Jenkins configuration. While configuring Jenkins hosted on the Kubernetes cloud is very straight-forward in terms of credentials and accessibility, it may require additional steps if you are not running the Jenkins master on Kubernetes or would like to configure it for external Kubernetes cluster(s). This blog post describes the steps to take and pitfalls to avoid while configuring Kubernetes client certificates for the Jenkins Kubernetes Plugin.
Requirements
Things you will need:
- Access to kube–admin Kubernetes configuration file (typically found in
~/.kube/config
) - Jenkins Kubernetes Plugin 0.12
Installing the Plugin
Jenkins Kubernetes Plugin (at the time of this writing) is at v0.12, and is available via Jenkins update site plugins. Installation is straight-forward and no different from other Jenkins plugins.
Configuring Kubernetes Cloud
Navigate to https://your-jenkins.com/configure and find “Add a new cloud” option
Configure your Kubernets Cluster (cloud).
Local Cluster
If your Jenkins Master is hosted on the same Kubernetes Cluster then all you need is to provide the Kubernetes URL for your local cluster as:
- short form: https://kubernetes
- long form: https://kubernetes.default.svc.%5Byour-domain%5D
- your-domain is typically ‘cluster.local’
- you can find additional details on this in Kubernetest Plugin README
Click ‘Test Connection’ to verify the successful connection.
Remote Cluster
If you are not hosting Jenkins on the same Kubernetes cluster (or not hosting it on Kubernetes at all), then you need to perform a few extra steps to configure the access to your Kubernetes cluster.
If you have access to kube-admin
configuration (typically found under ~/.kube/config
) then you can use it to complete the Kubernetes cluster access setup.
Kubernetes server certificate key
Grab the ‘cluster: certificate-authority-data’ value from your ~/.kube/config
file
apiVersion: v1 clusters: - cluster: certificate-authority-data: LSuperLongBase64EncodedString==
and decode it to get the X509
certificate
echo LSuperLongBase64EncodedString== | base64 -d > ca.crt
Your output should look something like
-----BEGIN CERTIFICATE----- MIIAnotherSuperLongSeritificateValueString -----END CERTIFICATE-----
Copy and paste the content of the ca.crt
file into the Kubernetes server certificate
. For this specific step we only need the certificate value, but the ca.crt
file will be used in subsequent steps.
Without the server certificate, you may get the following SSL Error
You can disable https certificate check by selecting the check box.
Credentials
After you either provided the server certificate (or skipped the SSL check altogether), testing the connection may return following access error:
Now we can add the Kubernetes cluster credentials using Kubernetes user certificates (also found in the ~/.kube.config file
)
First, we need to grab the base64 encoded client-certificate-data
and client-key-data
user: client-certificate-data: LSuperLongBase64EncodedString== client-key-data: LAnotherSuperLongBase64EncodedString==
Using the same step as with ca.crt
we will decode and create:
– client.crt
with client-certificate-data
– client.key
with client-key-data
Client P12 Certificate File
Using all three files we need to generate client certificate file in PKCS12 format
openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt
NOTE: It is important that you provide a passphrase
(as you will see later)
At this point, you are ready to add a new Kubernetes client certificate to Jenkins.
Click Add -> Jenkins
Make sure Kind
is set to Certificate
Select Upload PKCS#12 certificate and then hit Upload Certificate.
You should see a certificate file selector:
Navigate to the client.pfx file you generated and hit Upload.
Note: You will still see the message which you can ignore:
Enter the password you used for client.pfx
. If you provided the correct password you should see the above error message (‘No certificate uploaded’) changed to a warning (‘Could retrieve key “1”. You may need to provide a password’). You can ignore this warning as well.
Complete the form with an ID and a description. I recommend using (or including) the Kubernetes cluster name as a part of both the ID and the description.
Hit Add and that’s all for creating the Kubernetes client certificate. The Jenkins Credential Provider window will close and you should return to the Configuration view.
Select the newly created certificate in the Credentials drop-down.
Now, hit Test Connection again. This time you should see Connection Successful message
PCKS Certificate Without Passphrase
If you set up the PCKS client certificate without a passphrase, Jenkins will not complain and will accept the certificate. However, using this certificate will result in a somewhat obscure error message:
Other tell-tell signs that your certificate wasn’t “successfully” accepted are:
- You won’t get the warning message ‘Could retrieve key “1”. You may need to provide a password’, and the error message ‘No certificate uploaded’ will remain
- The credentials drop-down box will not include ‘CN=kube-admin’ as a part of the certificate name.
Conclusion
Jenkins Kubernetes Plugin provides additional credentials mechanisms to authenticate against the Kubernetes cluster(s) like a Kubernetes service account
However, at this time I was able to configure Kubernetes Cloud credentials using client certificates only. That is not to say that Kubernetes service accounts don’t work, just that I didn’t figure how to get it going.
I hope you find the steps above helpful in configuring your Jenkins against Kubernetes cluster(s). Let me know if you find any inaccuracies or have any questions, comments or suggestions!
Ilya, thank you for this post, you definitely should contribute to jenkinsci/kubernetes-plugin README, as example of external kube cluster config. I lost about 2 days to find out how to config plugin with my externally configured minikube
LikeLiked by 2 people
Very useful, I also spent many hours trying to understand how to accomplish this. I agree that this should be added to the jenkinsci/kubernetes-plugin README. Thanks!
LikeLiked by 1 person
This was super helpful for me. Thank you very much!
LikeLiked by 1 person
This saved me a lot!
LikeLiked by 1 person
i have question.. i am not getting in the kind kubernetes service account since my cluster is outside.. jenkins is not running in side the cluster for this i need to add certificate and all…
without adding certificate and credentials my pod is running in kubernetes but jenkins master is not able to take slave pod for my job. i am not getting what to do?????
LikeLike
Hello krish,
I am not sure I entirely follow. It is not clear is your Jenkins-master runs on Kubernetes cluster or runs outside of Kubernetes cluster?
LikeLike
Hai I am still getting error while trying to connect to Jenkins.. cluster is remote.. and in your steps passphrase you mentioned how to get that passphrase I didn’t understand clearly..
Last step also credentials showing kubernetes service account you got.. that’s commes only when Jenkins is running inside the cluster right?…
I have one more doubt
I have one more Jenkins which connection is successful but after pod is running successfull then Jenkins not able to run the job on that pod?? What would be the issue..
I am not getting any logs???….
LikeLike
Please doulbe check if your Jenkins master host:port(s) a reachable from your cluster where you run your slaves.
LikeLike
Thanks a lot my 1 week struggle comes to an end
LikeLiked by 2 people
This is great! Thanks, Illya!!!
LikeLiked by 1 person
very useful THANK YOU!
LikeLiked by 1 person
Thanks! This is really helpful!!!
LikeLiked by 1 person
Thank you, very useful!
LikeLiked by 1 person
Every every useful. Thanks very much!
LikeLiked by 1 person
Awesome tutorial, thank you very much!
I just got one probleme which made me waste several hours, but I finally solved it: I am behind a corporate proxy, so I needed to add my kubernetes API host to the Jenkins master JVM “no_proxy” option:
-Dhttp.nonProxyHosts=myk8smaster.domain.tld|someotherhost.tld
(despite its name, this option is used both for http and https requests)
LikeLike
Note: I used k8s 1.10.0 and jenkins-kubernetes plugin 1.6.0
LikeLike
Thanks!
LikeLike
Thanks , very helpful for me
LikeLike
Nice blog for kubernetes plugin certifiaction!
LikeLike
Thanks
LikeLike
Thank you very much!
LikeLike
Thank you 100x’s over for this information! It was invaluable in getting Jenkins to play nice with Kubernetes!!
LikeLike
most useful blog
LikeLike
Thanks a lot!
LikeLike
First off, thanks! This helped a ton. I have a weird problem though: the server certificate, which is stored in the Kubernetes plugin section of the config.xml in Jenkins’ home, is always mangled when Jenkins starts.
That is, I paste it into the web interface and save (alternatively: edit the XML directly – same result), and the certificate is stored in the XML correctly, and Jenkins works correctly.
On next restart, Jenkins appears to try to decode it, then saves it, but the encoding is somehow messed up.
For reference, it is supposed to be a string starting with “MIIC”, but ends up being as a string starting with 0 and containing tons of undisplayable symbols.
Any idea whether I’m messing up? Or is this a bug?
LikeLike
Never mind, I got it: Be sure to include the “—–BEGIN CERTIFICATE—–” and “—–END CERTIFICATE—–”, then it works as expected.
LikeLike
Awesome! This post helps me a lot!
LikeLike