PLG (Promtail, Loki, Grafana) stack for apps monitoring

Now a days application logs monitoring became very crucial for the microservice architecture. We expect all the application logs should be streamlined with appropriate labels and should be compatible with microservice architecture.

I’ve seen ELK stack (ElasticSearch, LogStash, and Kibana) widely accepted for application monitoring and alerting. I thought to try using other alternative approaches. I came across Loki which is a lightweight log aggregator and can be integrated with Grafana for application logs monitoring and alerting purposes. Loki is similar to Prometheus but in fact, Prometheus uses the Pull model to scrape metrics from different sources and label them and store them in TSDB(Time Series Database) and majorly used for application performance monitoring such as (http_response_time, requests_counts, etc.) But Loki is specifically used for application logs and it works on Push Model i.e. we need another agent which Pushes the data to Loki and then from Loki logs can be shown to Grafana where Loki works as a data source.

As we need another agent to push logs to Loki, we have below 4 options:

  1. Promtail (https://grafana.com/docs/loki/latest/clients/promtail)
  2. Loki Docker Drive Plugin (https://grafana.com/docs/loki/latest/clients/docker-driver)
  3. Fluentd (https://github.com/fluent/fluentd)
  4. Fluent-bit (https://github.com/fluent/fluent-bit)

I have a stack of services (Django application, API Gateway, Node application, and Nginx). I’ve used docker-compose to spin up all the services along with monitoring services (PLG stack).

PLG stack refers to Promtail, Loki and Grafana. Promtail extracts and collects logs from docker containers log files and pushes them to the Loki service which then Grafana uses to show logs in the log panel.

below is the Promtail configuration file:

Below is the sample docker-compose file for all the services.

I’ve provisioned Loki Datasource with Grafana. below is the data source provisioning file.

apiVersion: 1
datasources:
- name: SampleLokiDS
type: loki
access: proxy
url: "http://loki:3100"
editable: false
jsonData:
maxLines: 1000

when we run docker-compose up -d it will start all the services and promtail will start sending the logs to Loki and when we click on Grafana explore there on selecting data source we can see Loki Log Browser where we see all the labels to show application logs.

That’s how we get the logs in Grafana through Loki and Promtail.

The only drawback is in the log browser we get the labels as filenames. so we need to configure the promtail config to extract other useful labels such as container_id, image_id, etc.

The more we refine the labels, the more filtered logs we will get.

In the next article, I’ll explain other ways to get useful labels with Loki and other detials.

Please provide useful thoughts and comments on this article. Thanks!

Software Engineer at BookMyShow