My log file gets too large! — Python logging to multiple files

Implementing rotating file handlers

These are some big logs (image by Vladimir Kudinov on Unsplash)

After a few weeks of logging that poor single logging file becomes pretty huge. In this short and simple article we’ll create a special file handler that writes to a file until a certain file size is exceeded. Then it stores the file and writes the logs to a new one. Let’s code!

There are two strategies for determining when to write to a new file: based on the file size and based on time. First we’ll check out the rotating file handler, then the timed rotating file handler.

1. RotatingFileHandler

This handler writes to a file until it has reached a set size in bytes. Check out the sample below:

When we run this code we see the following in out logdir:

What happens when the file size exceeds the set limit?
The example above will save all logs to a file called test.log. If that file is full (if its size is 100KB) then test.log will be renamed to test.log.1 and a new test.log will be created. If test.log.1 already exitsts it will be renamed to test.log.2 etc. Also we’ve specified (with backupCount=3) that we don’t want a test.log.4; these files get deleted.

Keep your code secure by using environment variables and env files

2. TimedRotatingFileHandler

This handler doesn’t look at the file size as the RotatingFileHandler but rolls over to a new file according to a specified time interval that we can specify with the when and interval parameters: