In this article, I will describe how to configure Docker containers to send logs to Google Cloud Logging. That service can parse them, detect severity and based on that you can create filters or alerts.

Context

I have a single GCE Virtual Machine with 2 applications running as a Docker container. I would like to stream logs to the Cloud Logging service and create filters/alerts based on the severity field.

My applications use JSON format for logs.

Problem

Cloud Logging service can parse the input and detect severity, but only if it is pure JSON. It is working, for example, when we stream logs directly from our application using a dedicated programming library. But this solution can be a vendor lock for us.

In most general cases, we'd like to use some wrapper that gets our logs and streams them to Cloud Logging. With Docker, it's possible by using logging drivers.

This is an example of the Docker Compose definition:

services:
  app-1:
    image: myimage
    logging:
      driver: gcplogs
    ...

Source: https://docs.docker.com/config/containers/logging/gcplogs/

There is a huge flaw! Logs are packed into a nested JSON structure. Cloud Logging service cannot find severity field in the nested structure.

The example of the proper logs recognized by Google:

{
  "field-1": "text1",
  "field-2": "text2",
  "severity": "warning",
  ...
}

In the above example, Google can detect the severity field, and we can create better filters/alerts.

The example of logs sent by Docker Compose gcplogs driver:

{
  "some_docker_compose_fileds": "some text",
  "jsonPayload": {
    ...
    "severity": "warning",
    ...
  }
  ...
}

In the above example Google cannot find severity field.

Solution

There is a quite simple solution for that problem with 2 steps.

1. Use fluentd as Docker Compose logging driver.

2. Install and configure Google Ops Agent on the server.

Docker Compose sends logs to the fluentd driver using a specific port. Then, Google Ops Agent listens on that port, parse and modify logs, and sends them to Cloud Logging.

Google Ops Agent can be installed in different ways. Please, check out Google's documentation.

Docker Compose definition:

services:
  app-1:
    image: myimage
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: app-1

The tag field is used instead of container ID in Cloud Logging.

The configuration of Google Ops Agent (/etc/google-cloud-ops-agent/config.yaml):

logging:
  receivers:
    docker-fluentd:
      type: fluent_forward
      listen_host: 127.0.0.1
      listen_port: 24224
  processors:
    parse_json:
      type: parse_json
      field: log
    modify_fields:
      type: modify_fields
      fields:
        severity:
          copy_from: jsonPayload.severity
  service:
    pipelines:
      default_pipeline:
        receivers:
        - docker-fluentd
        processors:
        - parse_json
        - modify_fields

Summary

Let me know if it works for you. If you have a different case describe it in a comment, maybe I will be able to help. I have also other similar examples.

The solution is based on this conversation thread: https://issuetracker.google.com/issues/202309453

Cover image generated by DALLE