FLG (Fluentd, Loki, Grafana) stack for apps monitoring

Raman Pandey
3 min readJul 7, 2021

In my previous story, I had discussed PLG (Promtail, Loki, Grafana) for apps monitoring. One can read the article here.

In this post, I will discuss another alternative which is the FLG stack which refers to (Fluentd, Loki, and Grafana) stack for apps log monitoring. unlike PLG stack here we need Fluentd as a logging driver for all docker apps.

Fluentd is a logging driver supported natively by Docker. Docker has a default logging driver json-file but one can configure another docker logging driver. All docker-supported logging drivers can be found here.

All the Fluentd related details can be found here. Fluentd is a CNCF (Cloud Native Computing Foundation) project and it works as a daemon and collects all the logs (access/error) and ships them remotely to other services and can dump logs to AWS S3 or any other storage. It’s a lightweight application which can also send alerts and event to remote system hence will be a great companion to Loki in conjunction with Grafana.

I tried FLG stack along with other services (djangoapp, nodeapp, and nginx) in the same docker-compose file. Below is the Fluentd configuration file.

[INPUT]
Name forward
Listen 0.0.0.0
Port 24224
[Output]
Name grafana-loki
Match *
Url ${LOKI_URL}
RemoveKeys source
Labels {job="fluent-bit"}
LabelKeys container_name
BatchWait 1s
BatchSize 1001024
LineFormat json
LogLevel info

and the docker-compose file will look like below.

version: '3.2'
services:
postgres:
container_name: postgres
image: postgres:12
env_file:
- "secrets/postgres.env"
volumes:
- postgres-data:/var/lib/postgresql/data/
ports:
- '5432:5432'
restart: always
nginx:
container_name: nginx
image: nginx-private-registry-image
ports:
- 80:80
restart: always
depends_on:
- djangoapp
- nodeapp
logging:
driver: fluentd
options:
fluentd-address: host.docker.internal:24224
djangoapp:
container_name: djangoapp
image: django-private-registry-image
ports:
- '8087:8087'
restart: always
depends_on:
- postgres
logging:
driver: fluentd
options:
fluentd-address: host.docker.internal:24224
nodeapp:
container_name: nodeapp
image: node-private-registry-image
ports:
- '1234:1234'
restart: always
logging:
driver: fluentd
options:
fluentd-address: host.docker.internal:24224
loki:
container_name: loki
image: grafana/loki:2.0.0
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./loki/config.yaml:/etc/loki/local-config.yaml
grafana:
container_name: grafana
image: grafana/grafana:latest
ports:
- "3000:3000"
volumes:
- ./grafana/grafana.ini:/etc/grafana/grafana.ini
- ./grafana/provisioning/datasources/datasource.yml:/etc/grafana/provisioning/datasources/datasource.yml
depends_on:
- loki
logging:
driver: fluentd
options:
fluentd-address: host.docker.internal:24224
fluent-bit:
image: grafana/fluent-bit-plugin-loki:latest
container_name: fluent-bit
environment:
- LOKI_URL=http://loki:3100/loki/api/v1/push
volumes:
- ./fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
ports:
- "24224:24224"
- "24224:24224/udp"

volumes:
postgres-data:

Once we run docker-compose up -dit will start other services along side FLG stack. Fluentd will start on port 24224 and as we configured other services logging driver to fluentd then apps will start sending logs to fluentd and fluentd will send app logs to loki which in turn can be displayed to Grafana. we can see app logs in Grafana as below.

Fluentd also send more labels such as container_name, image_id etc. as we configure in fluentd config file. we can see labels in loki log browser as well.

Unlike PLG stack FLG stack some more enhanced features. such as we don’t need to write custom regex to add labels to logs.Fluentd config itself enough for configuring labels. Although, there might be one issue one can see which is below error

ERROR: for <service_name>  Cannot start service <service_name>: failed to initialize logging driver: dial tcp 192.168.65.2:24224: connect: connection refused

which might be because service is not able to send logs to internal docker network and being timed out. To resolve this one can set below environment variables and then run docker-compose up -d which might fix this issue.

export DOCKER_CLIENT_TIMEOUT=120
export COMPOSE_HTTP_TIMEOUT=120

Please let me know your thoughts and any suggestions are welcome. Thanks!

--

--