Stop That Huge Jenkins Log

When you have big Jenkins log, the recommended approach is to go through it and fix exceptions causes. Unfortunately sometimes the speed of increasing the log size is so fast, that you need a temporary solution to stop that hard disk eating monster.

Recently we experienced a huge Jenkins master log on Ubuntu. In one night it gained 35 GB. Not bad for a single log file. And it was not larger because the whole hard disk was 40 GB and there was simply no more space. I started looking at the logs, seeing no pattern at first. Then I noticed that when I visit one of the views, the log file at once increases with 200 MB. Then I found jenkins.log contained several thousand warnings connected with uninstalled Inheritance plugin. There were many old builds that used that plugin, although it was not installed any more. The build were visible only in the file system, not in the UI.

The warning was com.thoughtworks.xstream.mapper.CannotResolveClassException: hudson.plugins.project_inheritance.projects.InheritanceBuild, followed by 200+ more lines.  We use a plugin to delete old builds but it had not helped with deleting the problematic ones. As a temporary workaround, before deleting all those old builds, I set the logging level to SEVERE and changed the logrotate settings for Jenkins.

Persist Logging Levels

The log levels can be changed through the Jenkins UI on https://yourjenkinsserver.com/log. Through the UI you can navigate from Manage Jenkins -> System Log -> Log Levels. The log levels will be persisted until you restart the Jenkins service. You can plug in with groovy script on the Jenkins initialization. In order to make the changes permanent, create init.groovy file in Jenkins home (/var/lib/jenkins in our Ubuntu installation) and modify the content below with the names of your loggers. I found this article very useful for the groovy script. The logger “” is the default logger. See more info about the logging level meaning in Java.

import java.util.logging.Level
import java.util.logging.Logger

Logger.getLogger("").setLevel(Level.SEVERE)
Logger.getLogger("org.apache.sshd").setLevel(Level.SEVERE)
Logger.getLogger("winstone").setLevel(Level.SEVERE)

Enable Logrotate

Another part of the problem was that logrotate was not executed regularly. The logrotate file for Jenkins in Ubuntu is in /etc/logrotate.d/jenkins. The log is in /var/log/jenkins. I played with different options, and the current file is:

/var/log/jenkins/jenkins.log {
    daily
    copytruncate
    missingok
    rotate 10
    size 1M
    nocompress
    notifempty
    maxage 30
}

We have set maximum number of 10 log files, each with max size of 1 MB, no file will be compressed, the files will not be rotated if they are empty.

Execute man logrotate in order to see detailed help about the logrotate command options.

Then I tested log rotation with -d option to debug without executing the logrotate.

sudo logrotate -d /etc/logrotate.d/jenkins

The output has been as expected.

Then I entered  sudo logrotate /etc/logrotate.d/jenkinsand nothing happened. The jenkins log was not rotated. The I run the logrotate command with -v (verbose) to see more detailed output from the command:

sudo logrotate -v /etc/logrotate.d/jenkins

There was an error “Ignoring /etc/logrotate.d/jenkins because of bad file mode”.

It turned out that logrotate command was not executed because of a bad file permission. I copied the file from other computer so this must be when I changed the permissions without noticing.

To see the permissions you can run  stat /etc/logrotate.d/jenkins
The file should have 0644 permissions, owner root, group root. If the permissions are not correct use the following:

sudo chmod 0644 /etc/logrotate.d/jenkins
sudo chown -R root:root /etc/logrotate.d/jenkins

Summary

I presented you two ways to cope with big Jenkins log. The first strategy is to decrease the volume of the logged data. Instead of turning logging off completely, you can put it on SEVERE level. The second approach is to rotate, divide into several files and periodically delete old logs. You can find more Ubuntu commands in this article.