OAuth2 Proxy, Traefik and Docker logos.

This how-to is tightly related to the previous one: Protect your websites with oauth2_proxy behind traefik (docker stack edition). This time, I’m going to use docker-compose.

You’ll see how to deploy prometheus, grafana, portainer behind a traefik “cloud native edge router”, all protected by oauth2_proxy with docker-compose.

Make sure you look into the github repository tlex/traefik-oauth2-proxy. You can find all the files over there.

Objectives

For this tutorial, we will build the following:

containous/traefik will receive all http and https requests
pusher/oauth2_proxy will authenticate only the requests for the protected domains
oauth.home.ix.ai will handle the OAUTH responses
These domains are protected by the oauth2_proxy (Sign in with Google):
- prometheus.home.ix.ai - prometheus
- grafana.home.ix.ai - grafana
- portainer.home.ix.ai - portainer
- traefik.home.ix.ai (the traefik dashboard and API)
The authentication domain is .home.ix.ai (the cookie domain)

A couple of more goodies will be added as well:

all HTTP traffic will be redirected to HTTPS
automates certificate generation with Let’s Encrypt for the domains hosted with Cloudflare
using docker secrets to manage the Prometheus configuration

Notes

Networks

In this example, I will not be configuring any extra networks. See my previous post for a working example with docker swarm and dedicated networks.

Storage

The services used here need some form of persistent storage. This article uses the following paths:

./folders/grafana-db
./folders/portainer-db
./folders/prometheus
./folders/traefik

traefik storage folder

I’ll be using ./folders/traefik as storage for the certificates. Make sure you read the documentation, especially this part:

For concurrency reason, the file acme.json file cannot be shared across multiple instances of Traefik.

traefik Cloudflare credentials

In the previous post I’ve used the Cloudflare username and API token. This time, I’ll be using two Cloudflare API tokens, stored in the following variables:

CF_DNS_API_TOKEN
CF_ZONE_API_TOKEN

For more information, please see the Let’s Encrypt client and ACME library written in Go documentation.

The CF_DNS_API_TOKEN (API token with DNS:Edit permission) can be set only for a specific zone (in our case, ix.ai), but the CF_ZONE_API_TOKEN (API token with Zone:Read permission) has to have permissions for all zones.

OAuth2 credentials

Please look directly on the oauth2_proxy Auth Configuration page on how to configure your oauth2 provider. I’m using Google in the example, but you could choose any other one.

Preparation

Create the folders

Execute the following on the console:

for i in grafana-db portainer-db prometheus traefik; do
  mkdir -p "./folders/${i}"
done
# Prometheus runs with UID 99, so we set the permissions here for the folder
sudo chown 99:99 ./folders/prometheus

Configuration files

We create three files:

.env
prometheus.yml
docker-compose.yml

.env

Make sure you set the correct variables below!

# Make sure you set the correct variables below!
# Cloudflare
CF_DNS_API_TOKEN=FIXME: ADD YOUR CF_DNS_API_TOKEN here
CF_ZONE_API_TOKEN=FIXME: ADD YOUR CF_ZONE_API_TOKEN here
# oauth2_proxy
OAUTH2_PROXY_CLIENT_ID=FIXME: Google Client ID for Web application
OAUTH2_PROXY_CLIENT_SECRET=FIXME: Google Client secret
# Note: the cookie secret needs to be 16, 24 or 32 bytes long
OAUTH2_PROXY_COOKIE_SECRET=FIXME: Cookie secret

# Grafana
GF_SECURITY_SECRET_KEY=FIXME: Add a random string here

# The domain name for all services
DOMAIN=home.ix.ai

# MySQL variables
MYSQL_PASSWORD=FIXME: Add a mysql password
MYSQL_ROOT_PASSWORD=FIXME: Add a mysql root password
MYSQL_USER=grafana
MYSQL_DATABASE=grafana

# I'm setting the versions here, for easy change
TRAEFIK_VERSION=latest
OAUTH2_PROXY_VERSION=latest
PORTAINER_VERSION=latest
PORTAINER_AGENT_VERSION=latest
PROMETHEUS_VERSION=latest
MYSQL_VERSION=latest
GRAFANA_VERSION=latest

prometheus.yml

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 5s
scrape_configs:
  - job_name: prometheus
    scheme: http
    static_configs:
      - targets:
        - prometheus:9090
  - job_name: grafana
    scheme: http
    static_configs:
      - targets:
        - grafana:3000
  - job_name: traefik
    scheme: http
    static_configs:
      - targets:
        - traefik:8080

docker-compose.yml

version: "3.7"

services:
  traefik:
    image: traefik:${TRAEFIK_VERSION}
    restart: unless-stopped
    labels:
      traefik.enable: 'true'
      traefik.http.middlewares.default-compress.compress: 'true'
      traefik.http.middlewares.default-http.redirectScheme.scheme: https
      traefik.http.middlewares.default-http.redirectScheme.permanent: 'true'
      traefik.http.middlewares.default-https.chain.middlewares: default-compress
      traefik.http.routers.default-redirect.entrypoints: http
      traefik.http.routers.default-redirect.middlewares: default-http
      traefik.http.routers.default-redirect.rule: "HostRegexp(`{any:.*}`)"
      traefik.http.routers.default-redirect.service: noop@internal
      traefik.http.routers.traefik.entrypoints: https
      traefik.http.routers.traefik.middlewares: oauth-signin,oauth-verify,default-https
      traefik.http.routers.traefik.rule: "Host(`traefik.${DOMAIN?err}`) && PathPrefix(`/api`, `/dashboard`)"
      traefik.http.routers.traefik.tls.certResolver: 'default'
      traefik.http.routers.traefik.service: api@internal
      traefik.http.services.traefik.loadbalancer.server.port: '80'
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./folders/traefik:/etc/traefik/acme
    environment:
      CF_DNS_API_TOKEN: ${CF_DNS_API_TOKEN?err}
      CF_ZONE_API_TOKEN: ${CF_ZONE_API_TOKEN?err}
    command:
      - --accesslog=true
      - --accesslog.fields.defaultmode=keep
      - --accesslog.fields.headers.defaultmode=keep
      - --api.dashboard=true
      - --api.insecure=true
      # For the example only. Remove it for production!
      - --certificatesResolvers.default.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
      - --certificatesResolvers.default.acme.dnsChallenge.provider=cloudflare
      - --certificatesResolvers.default.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53
      - --certificatesResolvers.default.acme.storage=/etc/traefik/acme/acme.json
      - --entrypoints.http.address=:80
      - --entrypoints.https.address=:443
      - --log.level=INFO
      - --metrics.prometheus=true
      - --providers.docker.endpoint=unix:///var/run/docker.sock
      - --providers.docker.defaultRule=Host(`{{ index .Labels "ai.ix.fqdn"}}`)
      - --providers.docker.exposedByDefault=false
  oauth:
    image: quay.io/pusher/oauth2_proxy:${OAUTH2_PROXY_VERSION}
    restart: unless-stopped
    labels:
      ai.ix.expose: 'true'
      traefik.enable: 'true'
      traefik.http.middlewares.oauth-verify.forwardAuth.address: http://oauth:4180/oauth2/auth
      traefik.http.middlewares.oauth-verify.forwardAuth.trustForwardHeader: 'true'
      traefik.http.middlewares.oauth-verify.forwardAuth.authResponseHeaders: X-Auth-Request-User,X-Auth-Request-Email,Set-Cookie
      traefik.http.middlewares.oauth-signin.errors.service: oauth@docker
      traefik.http.middlewares.oauth-signin.errors.status: '401'
      traefik.http.middlewares.oauth-signin.errors.query: /oauth2/sign_in
      traefik.http.routers.oauth.entrypoints: https
      traefik.http.routers.oauth.rule: Host(`oauth.${DOMAIN?err}`) || PathPrefix(`/oauth2`)
      traefik.http.routers.oauth.tls.certResolver: 'default'
      traefik.http.routers.oauth.service: oauth@docker
      traefik.http.services.oauth.loadbalancer.server.port: '4180'
    environment:
      OAUTH2_PROXY_CLIENT_ID: '${OAUTH2_PROXY_CLIENT_ID?err}'
      OAUTH2_PROXY_CLIENT_SECRET: '${OAUTH2_PROXY_CLIENT_SECRET?err}'
      OAUTH2_PROXY_COOKIE_DOMAIN: '.${DOMAIN?err}'
      OAUTH2_PROXY_COOKIE_REFRESH: '1h'
      OAUTH2_PROXY_COOKIE_SECURE: 'true'
      OAUTH2_PROXY_COOKIE_SECRET: '${OAUTH2_PROXY_COOKIE_SECRET?err}'
      OAUTH2_PROXY_EMAIL_DOMAINS: '*'
      OAUTH2_PROXY_FOOTER: '-'
      OAUTH2_PROXY_HTTP_ADDRESS: '0.0.0.0:4180'
      OAUTH2_PROXY_PASS_BASIC_AUTH: 'false'
      OAUTH2_PROXY_PASS_USER_HEADERS: 'true'
      OAUTH2_PROXY_PROVIDER: 'google'
      OAUTH2_PROXY_REVERSE_PROXY: 'true'
      OAUTH2_PROXY_SET_AUTHORIZATION_HEADER: 'true'
      OAUTH2_PROXY_SET_XAUTHREQUEST: 'true'
      OAUTH2_PROXY_WHITELIST_DOMAIN: '.${DOMAIN?err}'
  portainer:
    image: portainer/portainer-ce:${PORTAINER_VERSION}
    restart: unless-stopped
    depends_on:
      - portainer-agent
    labels:
      ai.ix.fqdn: 'portainer.${DOMAIN?err}'
      traefik.enable: 'true'
      traefik.http.routers.portainer.entrypoints: https
      traefik.http.routers.portainer.middlewares: oauth-signin,oauth-verify,default-https
      traefik.http.routers.portainer.tls.certResolver: 'default'
      traefik.http.services.portainer.loadbalancer.server.port: '9000'
    volumes:
      - './folders/portainer:/data:rw'
    # Warning! --no-analytics is deprecated and will probably be removed soon
    command: --no-analytics
  portainer-agent:
    image: portainer/agent:${PORTAINER_AGENT_VERSION}
    restart: unless-stopped
    environment:
      AGENT_CLUSTER_ADDR: tasks.portainer_agent
      CAP_HOST_MANAGEMENT: '1'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
      - /:/host
  prometheus:
    image: prom/prometheus:${PROMETHEUS_VERSION}
    restart: unless-stopped
    labels:
      ai.ix.fqdn: 'prometheus.${DOMAIN?err}'
      traefik.enable: 'true'
      traefik.http.routers.prometheus.entrypoints: https
      traefik.http.routers.prometheus.middlewares: oauth-signin,oauth-verify,default-https
      traefik.http.routers.prometheus.tls.certResolver: 'default'
      traefik.http.services.prometheus.loadbalancer.server.port: '9090'
    secrets:
      - prometheus.yml
    volumes:
      - './folders/prometheus:/data:rw'
    user: '99:99'
    ulimits:
      nofile:
        soft: 200000
        hard: 200000
    command: |
        --config.file=/run/secrets/prometheus.yml
        --web.enable-admin-api
        --web.external-url=https://prometheus.${DOMAIN?err}
        --storage.tsdb.path=/data
        --storage.tsdb.retention.time=30d        
  grafana-db:
    image: mysql:${MYSQL_VERSION}
    restart: unless-stopped
    volumes:
      - './folders/grafana-db:/var/lib/mysql:rw'
    environment:
      MYSQL_USER: '${MYSQL_USER?err}'
      MYSQL_DATABASE: '${MYSQL_DATABASE?err}'
      MYSQL_PASSWORD: '${MYSQL_PASSWORD?err}'
      MYSQL_ROOT_PASSWORD: '${MYSQL_ROOT_PASSWORD?err}'
  grafana:
    image: grafana/grafana:${GRAFANA_VERSION}
    restart: unless-stopped
    depends_on:
      - grafana-db
    labels:
      ai.ix.fqdn: 'grafana.${DOMAIN?err}'
      traefik.enable: 'true'
      traefik.http.routers.grafana.entrypoints: https
      traefik.http.routers.grafana.middlewares: oauth-signin,oauth-verify,default-https
      traefik.http.routers.grafana.tls.certResolver: 'default'
      traefik.http.services.grafana.loadbalancer.server.port: '3000'
    environment:
      GF_ALERTING_EXECUTE_ALERTS: 'true'
      GF_ANALYTICS_CHECK_FOR_UPDATES: 'false'
      GF_ANALYTICS_REPORTING_ENABLED: 'false'
      GF_AUTH_PROXY_ENABLED: 'true'
      GF_AUTH_PROXY_HEADER_NAME: 'X-Auth-Request-User'
      GF_AUTH_PROXY_HEADER_PROPERTY: username
      GF_AUTH_PROXY_AUTO_SIGN_UP: 'true'
      GF_AUTH_PROXY_HEADERS: 'Email:X-Auth-Request-Email'
      GF_AUTH_PROXY_ENABLE_LOGIN_TOKEN: 'true'
      GF_AUTH_DISABLE_LOGIN_FORM: 'true'
      GF_AUTH_DISABLE_SIGNOUT_MENU: 'true'
      GF_DATABASE_HOST: 'grafana-db:3306'
      GF_DATABASE_NAME: 'grafana'
      GF_DATABASE_TYPE: mysql
      GF_DATABASE_PASSWORD: '${MYSQL_PASSWORD?err}'
      GF_DATABASE_USER: '${MYSQL_USER?err}'
      GF_EXPLORE_ENABLED: 'true'
      # These are the plugins I use. Feel free to modify the list
      #GF_INSTALL_PLUGINS: grafana-piechart-panel,natel-plotly-panel,grafana-clock-panel,camptocamp-prometheus-alertmanager-datasource,briangann-datatable-panel,grafana-worldmap-panel
      GF_LOG_LEVEL: info
      GF_LOGIN_COOKIE_NAME: grafana_cheese_cake
      GF_SECURITY_COOKIE_SECURE: 'true'
      GF_SECURITY_SECRET_KEY: ${GF_SECURITY_SECRET_KEY?err}
      GF_SERVER_DOMAIN: 'grafana.${DOMAIN?err}'
      GF_SERVER_ENFORCE_DOMAIN: 'true'
      GF_SERVER_ROOT_URL: https://%(domain)s/
      GF_SERVER_ROUTER_LOGGING: 'true'
      GF_SESSION_LIFE_TIME: 86400
      GF_USERS_ALLOW_SIGN_UP: 'false'
      GF_USERS_AUTO_ASSIGN_ORG: 'true'
      GF_USERS_AUTO_ASSIGN_ORG_ROLE: 'Admin'
      # see https://github.com/grafana/grafana/issues/20096
      GODEBUG: netdns=go
secrets:
  prometheus.yml:
    file: ./prometheus.yml

Deploy

Last step: deploy it all. In the folder with the files, run:

sudo docker-compose up --remove-orphans

If you want it to run in the background, add the option -d above.

Screenshots

Authentication

Traefik

Portainer

Grafana

Prometheus

Updates

This article was updated on:

2021-08-22: Updated the image for portainer, due to the name change in portainer-ce. Also removed the --no-auth from its command, since it’s not supported anymore (thanks Max for the hint!)
2020-02-09: Added the notes on how to configure the oauth2 provider with a link to https://pusher.github.io/oauth2_proxy/auth-configuration
2020-01-29: Added OAUTH2_PROXY_REVERSE_PROXY: 'true' to the oauth2_proxy environment (due to changes brought by oauth2_proxy v5.0.0)

Deploy traefik, prometheus, grafana, portainer and oauth2_proxy with docker-compose