BurstLogging

burstLogging is always a compromise between storing maximum amount of context information and maintaining good performance + saving disk space. In other words some errors occur only in production environment or are very difficult to reproduce locally, but you can’t store all debug messages on production systems.

I’d like to propose an alternative solution to this dilemma: BurstLogging.

The idea is simple – log only some debug messages that were created shortly before an error message was logged.

This is achieved by storing all logs in buffer, logging only informational messages during normal operation and dumping all messages when an error occurs. Chronological order of messages is preserved (debug messages that are older than currently logged info message are dropped) and there is no huge performance penalty (messages are formatted only when they are emitted).

Current implementation is written in Python, but it shouldn’t be difficult to port it to other programming languages or implement a language agnostic solution communicating via a port or a pipe.

The project is really new and it hasn’t been used in production yet. Please tell me what do you think about this idea.

2 thoughts on “BurstLogging

  1. This is a brilliant idea. I’d be probably a little worried about losing the messages in the buffer. For example, I log an INFO message and then the application gets stuck, I will never see the INFO message in my logs. I think a better compromise would be to emit all messages above emitLevel as they arive and when you get a message above burstLevel, repost all the messages from the buffer with some marker.

  2. Yeah, I was considering that, but I wanted to retain chronological order for easier reading.

    Another solution would be to emit all the logs of a certain age, for example few seconds.

    Alternatively the whole problem can be avoided by using a NoSQL database as log storage instraf of a flat file.

    I think I will make buffering / retain chronology optional as it has drawbacks you described.

Leave a Reply

Your email address will not be published.