This article is a guide for a setup of a new Ubuntu Jenkins master. The setup is made entirely via the terminal, without any GUI client. The migration of the data is done through Windows machine. Ubuntu is selected as OS for Jenkins master because it is very light and with good performance. After the setup the occupied hard disk space was 4GB.

If you do not feel quite comfortable with  terminal commands, see this article.

After you log in on your Ubuntu machine, it is time to install some packages:

1. Add repository key to the machine. The system will return OK when ready.

wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add -

2. Add the Debian package repository address to the server’s sources.list:

echo deb https://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list

3. Update packages:

sudo apt-get update

4. Install Jenkins:

sudo apt-get install jenkins

If you need specific Jenkins version use:

sudo apt-get install jenkins=2.89.3 Just put your preferred version in the place of 2.89.3.

5. Install git:

sudo apt-get install git

6. Install Node.js:

sudo apt-get install nodejs

7. Install tools for mounting Windows folders. This is needed if you will transfer files to/from a Windows machine.

sudo apt-get install cifs-utils

8. Install the WinDirStat GUI-less equivalent in Ubuntu. It is extremely useful when you run out of hard disk space and you need to know what is the cause for that.

sudo apt-get install ncdu
There is a nice NCDU guide if you want to learn more.

9. Make directory on the Ubuntu machine that will contain the mounted Windows share.

mkdir ~/windows-share-migration

The tilde (~) symbol is substitute for /home/yourusername.

10. Mount Windows folder that contains your Jenkins backups or configurations files. Use IP instead of machine name. Jenkins user is with uid 112 in the example below. The uid is needed if you want the Jenkins process to have write permissions in the mounted folder.

sudo mount.cifs //myIP/myWindowsSharedFolder/ ~/windows-share-migration -o user=myusername,domain=mydomain,vers=3.0,uid=112,gid=1000
Unmount share:

sudo umount ~/windows-share-migration

See all mounted:

mount

11. Get admin password so that you can login in the newly created Jenkins server web interface:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

12. Login with admin password on http://YourJenkinsUbuntuMachineIP:8080 on your Windows PC.

13. Install ThinBackup plugin if you plan to restore settings from other Jenkins server with it. I have some mixed feeling for this plugin. From one side, it does a good job in filtering your most important configuration and backing them up. From other side you should be aware that the archive can be corrupted if you select improper setting. So to be on the safe side, do a test migration with it before the real one.

14. Restore the ThinBackup archive or copy files from your old Jenkins server to the new Ubuntu server. Do not forger secrets and email-templates folders.

sudo cp -a ~/windows-share-migration/secrets/. /var/lib/jenkins/secrets/

15. If you have copied files with root, change their owner back to jenkins process, R option will do it recursively:

sudo chown -R jenkins:jenkins /var/lib/jenkins

16. Change Java heap size:

• In Windows this is done in jenkins.xml file.
• In Ubuntu you should change the `JAVA_ARGS=”-Xmx2048m”` in the /etc/default folder, jenkins file.
  Check out this article for more details.

17. Add your SSL certificates to cacerts java security folder if you use any (/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts). This step could be also needed if a build is failing because of this error:

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

If you upgrade the Java version you could follow one of the options below. Option 2 is the recommended one.

1. Copy the old version cacerts file into the new Java home /jre/security folder so that it overwrites the new version of cacerts.
2. Import the certificates into the new java version cacerts file.
   Make a copy of the file just in case the import breaks the file.

sudo cp -a /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts ~/windows-share-migration/

Export the certificate from the Chrome Browser. Place the certificate in the same folder as the cacerts file. Import it in the current Java certificates store.

sudo keytool -import -alias yourcertificate201901 -keystore cacerts -file yourcertificate201901.cer

More info is available in this article.

18. Change Jenkins default session timeout. This is needed if you want the session of the user to last more than the default 30 minutes.

Edit the file /etc/default/jenkins with the settings below and reboot the machine.

1. Add variable session timeout (in minutes).
SESSION_TIMEOUT=660

2. The new variable should be included in JENKINS_ARGS:
JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT --httpListenAddress=$HTTP_HOST --sessionTimeout=$SESSION_TIMEOUT"

If you want to check whether the settings have been applied successfully, execute the following groovy script in the Jenkins script console (http://YourJenkinsUbuntuMachineIP:8080/script):

import org.kohsuke.stapler.Stapler;
Stapler.getCurrentRequest().getSession().getMaxInactiveInterval() / 60

Configuration Files

You can check the folders used by Jenkins in the system info. (http://YourJenkinsUbuntuMachineIP:8080/systemInfo)
If you followed the steps above your Jenkins home will be /var/lib/jenkins.
Several other folders that contain Jenkins configurations (jenkins.xml does nothing in Ubuntu system):
War file is in /usr/share/jenkins/jenkins.war.
Logs are in /var/log/jenkins. If you have suddenly run out of disk space, graphs for the test builds are missing or you see that the log files are big,  check out this article.
Java certificate store is in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security.
Jenkins defaults are in the folder /etc/default/.
Logrotate config is in /etc/logrotate.d/. The file is named jenkins. For logrotate configuration check out this article. If you need a beginner guide on logrotate, check out this article.

Additional Resources

Install Jenkins on Ubuntu 16.04 Guide
Setup firewall
SSL setup

Summary

My opinion is that Jenkins server is much more stable on Ubuntu than it is on Windows. This is why I wrote this article on you how to setup Jenkins server on Ubuntu machine. The article consists of easy to follow steps to install new Jenkins server, configure it and migrate data from a shared Windows folder.

When you have big Jenkins log, the recommended approach is to go through it and fix exceptions causes. Unfortunately sometimes the speed of increasing the log size is so fast, that you need a temporary solution to stop that hard disk eating monster.

Recently we experienced a huge Jenkins master log on Ubuntu. In one night it gained 35 GB. Not bad for a single log file. And it was not larger because the whole hard disk was 40 GB and there was simply no more space. I started looking at the logs, seeing no pattern at first. Then I noticed that when I visit one of the views, the log file at once increases with 200 MB. Then I found jenkins.log contained several thousand warnings connected with uninstalled Inheritance plugin. There were many old builds that used that plugin, although it was not installed any more. The build were visible only in the file system, not in the UI.

The warning was com.thoughtworks.xstream.mapper.CannotResolveClassException: hudson.plugins.project_inheritance.projects.InheritanceBuild, followed by 200+ more lines.  We use a plugin to delete old builds but it had not helped with deleting the problematic ones. As a temporary workaround, before deleting all those old builds, I set the logging level to SEVERE and changed the logrotate settings for Jenkins.

Persist Logging Levels

The log levels can be changed through the Jenkins UI on https://yourjenkinsserver.com/log. Through the UI you can navigate from Manage Jenkins -> System Log -> Log Levels. The log levels will be persisted until you restart the Jenkins service. You can plug in with groovy script on the Jenkins initialization. In order to make the changes permanent, create init.groovy file in Jenkins home (/var/lib/jenkins in our Ubuntu installation) and modify the content below with the names of your loggers. I found this article very useful for the groovy script. The logger “” is the default logger. See more info about the logging level meaning in Java.

import java.util.logging.Level
import java.util.logging.Logger

Logger.getLogger("").setLevel(Level.SEVERE)
Logger.getLogger("org.apache.sshd").setLevel(Level.SEVERE)
Logger.getLogger("winstone").setLevel(Level.SEVERE)

Enable Logrotate

Another part of the problem was that logrotate was not executed regularly. The logrotate file for Jenkins in Ubuntu is in /etc/logrotate.d/jenkins. The log is in /var/log/jenkins. I played with different options, and the current file is:

/var/log/jenkins/jenkins.log {
    daily
    copytruncate
    missingok
    rotate 10
    size 1M
    nocompress
    notifempty
    maxage 30
}

We have set maximum number of 10 log files, each with max size of 1 MB, no file will be compressed, the files will not be rotated if they are empty.

Execute man logrotate in order to see detailed help about the logrotate command options.

Then I tested log rotation with -d option to debug without executing the logrotate.

sudo logrotate -d /etc/logrotate.d/jenkins

The output has been as expected.

Then I entered  sudo logrotate /etc/logrotate.d/jenkinsand nothing happened. The jenkins log was not rotated. The I run the logrotate command with -v (verbose) to see more detailed output from the command:

sudo logrotate -v /etc/logrotate.d/jenkins

There was an error “Ignoring /etc/logrotate.d/jenkins because of bad file mode”.

It turned out that logrotate command was not executed because of a bad file permission. I copied the file from other computer so this must be when I changed the permissions without noticing.

To see the permissions you can run  stat /etc/logrotate.d/jenkins
The file should have 0644 permissions, owner root, group root. If the permissions are not correct use the following:

sudo chmod 0644 /etc/logrotate.d/jenkins
sudo chown -R root:root /etc/logrotate.d/jenkins

Summary

I presented you two ways to cope with big Jenkins log. The first strategy is to decrease the volume of the logged data. Instead of turning logging off completely, you can put it on SEVERE level. The second approach is to rotate, divide into several files and periodically delete old logs. You can find more Ubuntu commands in this article.