Json Logging in HAProxy: The Right Way

I’ve searched for json logging in haproxy (in order to feed them to graylog, which we’ll get to later in this post) and found some one…

This is a cross-post of my Medium post.

---

I’ve searched for json logging in haproxy (in order to feed them to graylog, which we’ll get to later in this post) and found some one line configs, which presumably worked on the author’s setup, but failed on my installation.

Here’s an example of a non-working configuration. DON’T COPY/PASTE THIS BECAUSE IT DOESN’T WORK IN ALL SITUATIONS.

Although I’m a fan of one-liner tweaks (and have a One-liners repo for collecting those tweaks) sometimes you need more than one line of config and some explanations in order to achieve things (actually we rarely use one-liner configs in order to to achieve things) and this case is one of them.

First of all, you need to understand that haproxy has different modes. tcp and http are the common modes (and health is deprecated). The http mode has some log variables which are doesn’t exist on tcp mode. This is why you can’t have single line of log-format which covers all modes.

Here’s the complete configs we use on this article. I’ll explain what each file means.

Log-Format for each Mode

If you only have mode tcp or mode http on your haproxy setup, use the proper file ( mode-http.conf or mode-tcp.conf) on the defaults section of your config. But if you have both of these modes on your configs, you’ll need to use the proper configuration on each frontend or listen section based on the mode of that section.

One little improvement you can have is to find out which mode you’re using the most and add its config to default section, then you’ll only need to use the log-format when the mode is different than that mode.

Also notice that each log-format has it’s own haproxy_frontend_type field which will be useful when you want to separate log messages on graylog

HTTP Header Logging

The first two lines on mode-http.conf are there because we want to capture Header and Referer headers of the request. They’re used on on the third line in this way:

"host":"%[capture.req.hdr(0)]","referer":"%[capture.req.hdr(1),json(utf8s)]"

You can remove them (or modify them and add more headers) based on your logging requirements.

Log Message’s Max Length

The global-section.conf file is there because haproxy truncates log messages with more than 1024 characters which would be problematic in our case (since there defiantly would be log messages with more than 1024 characters) and you need to modify you log configuration on global section and add len 65535 (that max possible length based on the documentation) after the syslog server in order to avoid that.

Sending Logs to Graylog

I wanted haproxy logs in json format in order to feed them to graylog at the first place. After I achieved proper json logging on the haproxy side, I expected adding log <graylog server>:<graylog syslog tcp input port> len 65535 local0 config on global section to work out of the box but it didn’t. After some trials, I gave up on haproxy and tried to configure rsyslog to send haproxy logs.

HAProxy adds its own rsyslog config on /etc/rsyslog.d/49-haproxy.confwhen you install it using your distro’s package manager (this is the case on Ubuntu and I see no reason why this should be different on other distros). The default config write the logs into /var/log/haproxy.log and we’ll keep it that way (you can remove it if you want).

I modified the config to also send the logs to graylog’s syslog input (the @@ at the beginning of the address tells the rsyslog to send logs in tcp mode. Use @ if you want to use udp mode). You’ll need to restart rsyslog service after you’ve modified your configurations.

service rsyslog restart

Voilà! now we have our haproxy logs on graylog in json format. I hope this article would be useful for you 😉