Field Notes: Centralized Log Management with Loki & Promtail on a Cloud VPS

When a small SaaS startup began adding micro‑services—an Nginx reverse proxy, a Node.js API, and a PostgreSQL database—their log files multiplied across three separate locations. Debugging a latency spike required hopping between /var/log/nginx, /var/log/node, and /var/log/postgresql, wasting precious engineering time and increasing the risk of missing critical events. The team needed a lightweight, cost‑effective way to aggregate logs, retain them for a week, and query them quickly without moving to a heavyweight ELK stack.

Problem Statement – Scattered Logs in a Growing SaaS

Key constraints were:

• Budget‑first approach: The company runs on a modest Cloud VPS plan (1 vCPU, 1 GB RAM).
• Low operational overhead: Engineers already juggle CI/CD, monitoring, and incident response.
• Retention policy: Only seven days of log history are required for troubleshooting.
• Searchability: Ability to filter by service, severity, and request ID.

Traditional syslog aggregation was insufficient because it stores logs as plain text, making ad‑hoc queries painful. A full ELK stack would exceed the VPS resources. Loki, paired with Promtail, offered a “prometheus‑style” log indexing model that fits the resource envelope while delivering fast queries via Grafana.

Why Loki & Promtail Over Traditional Syslog

Loki stores log streams indexed only by labels (e.g., app=nginx, env=prod), avoiding full‑text indexing and dramatically reducing RAM and CPU usage. Promtail runs as a lightweight agent that tails files and ships them to Loki with the appropriate labels. This model aligns with the SaaS’s existing Prometheus metrics pipeline, allowing a single Grafana instance to visualize both metrics and logs.

Design Decisions and Architecture

The final architecture consists of three components on a single Cloud VPS:

• Loki server: Runs in a Docker container, persisting logs to the VPS SSD.
• Promtail agents: One per service, also containerized, reading local log files.
• Grafana (optional): Connected to Loki for log exploration; can be hosted elsewhere if the VPS is too tight on resources.

All containers share a dedicated Docker network (lognet) to keep traffic internal. Persistent volumes are mounted to survive container restarts.

Provisioning the Cloud VPS

Before diving into the setup, a reliable Cloud VPS is provisioned with Ubuntu 22.04 LTS. The chosen plan (1 vCPU, 1 GB RAM, 25 GB SSD) comfortably hosts the three containers, given Loki’s modest footprint when configured for a seven‑day retention window.

Implementation Steps

1. Install Docker Engine

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
sudo usermod -aG docker $USER

2. Create a Docker network for log components

docker network create lognet

3. Deploy Loki

Create a configuration file loki-config.yml with a seven‑day retention policy:

cat > loki-config.yml <<EOF
auth_enabled: false
server:
  http_listen_port: 3100
ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 5m
  max_chunk_age: 1h
schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/cache
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks
compactor:
  working_directory: /loki/compactor
  shared_store: filesystem
limits_config:
  retention_period: 168h   # 7 days
EOF

Run Loki with persistent volumes:

docker run -d --name=loki \
  --network=lognet \
  -p 3100:3100 \
  -v $(pwd)/loki-config.yml:/etc/loki/local-config.yml \
  -v $(pwd)/loki-data:/loki \
  grafana/loki:2.9.1 -config.file=/etc/loki/local-config.yml

4. Deploy Promtail for each service

Below is a generic Promtail configuration; create separate files per service (e.g., promtail-nginx.yml, promtail-node.yml).

cat > promtail-nginx.yml <<EOF
server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /tmp/positions.yaml
clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: nginx
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginx
          __path__: /var/log/nginx/*.log
EOF

Start the Promtail container for Nginx:

docker run -d --name=promtail-nginx \
  --network=lognet \
  -v /var/log/nginx:/var/log/nginx \
  -v $(pwd)/promtail-nginx.yml:/etc/promtail/config.yml \
  -v /tmp/promtail-positions:/tmp \
  grafana/promtail:2.9.1 -config.file=/etc/promtail/config.yml

Repeat the same steps for the Node.js and PostgreSQL logs, adjusting __path__ and label values accordingly.

5. Verify the pipeline

Query Loki directly to ensure logs are arriving:

curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
  --data-urlencode "query={job=\"nginx\"}" \
  --data-urlencode "limit=10"

Successful JSON output confirms the ingestion path works. If you have Grafana elsewhere, add Loki as a data source using http://:3100 and explore logs via the “Explore” view.

Operational Checks and Ongoing Maintenance

• Container health: Set Docker restart policies to unless-stopped and monitor via docker ps.
• Disk usage: Loki stores compressed chunks; run du -sh /path/to/loki-data daily. Configure a cron job to alert when usage exceeds 70 % of the 25 GB SSD.
• Log rotation: Although Loki handles retention, ensure the source log files are rotated (e.g., logrotate) to keep the tailer’s file handles manageable.
• Backup strategy: Snapshot the loki-data volume weekly using docker run --rm -v loki-data:/data -v $(pwd):/backup alpine tar czf /backup/loki-backup-$(date +%F).tar.gz -C /data ..
• Security: Loki’s default configuration is unauthenticated. If the VPS is internet‑facing, place a reverse proxy (Nginx) with basic auth or TLS client certificates before exposing port 3100.

Risks and Mitigations

Resource contention: Loki can consume CPU spikes during compaction. Mitigate by limiting Docker CPU shares (--cpus="0.8") and scheduling compaction during off‑peak hours via the compactor schedule.

Data loss on abrupt shutdown: Ensure Docker’s stop_grace_period is sufficient (e.g., 30 s) so Promtail can flush pending batches before Loki stops.

Log format changes: Promtail relies on static __path__ globs. When a service updates its logging path, update the corresponding Promtail config and reload the container.

Conclusion

By consolidating Nginx, Node.js, and PostgreSQL logs into a single Loki instance on a modest Cloud VPS, the SaaS team reduced mean time to resolution (MTTR) dramatically while staying within budget. The lightweight architecture scales comfortably to additional services, and the Prometheus‑compatible label model keeps query performance snappy. Regular operational checks—disk monitoring, container health, and secure exposure—ensure the pipeline remains reliable as the business grows.