Loki¶

Connect HolmesGPT to Loki for log analysis through Grafana or direct API access. Provides access to historical logs and advanced log queries.

When to Use This¶

✅ Your Kubernetes logs are centralized in Loki
✅ You need historical log data beyond what's in pods
✅ You want advanced log search capabilities

Prerequisites¶

Loki instance with logs from your Kubernetes cluster
Grafana with Loki datasource configured (recommended) OR direct Loki API access

Available Log Sources

Multiple logging toolsets can be enabled simultaneously. HolmesGPT will use the most appropriate source for each investigation.

Kubernetes logs - Direct pod log access (enabled by default)
Loki - Centralized logs via Loki
Elasticsearch / OpenSearch - Logs from Elasticsearch/OpenSearch
Coralogix - Logs via Coralogix platform
DataDog - Logs from DataDog

Configuration¶

Choose one of the following methods:

Option 1: Through Grafana (Recommended)¶

Required:

Grafana service account token with Viewer role
Loki datasource UID from Grafana

Find your Loki datasource UID:

# Port forward to Grafana
kubectl port-forward svc/grafana 3000:80

# Get Loki datasource UID
curl -s -u admin:admin http://localhost:3000/api/datasources | jq '.[] | select(.type == "loki") | .uid'

Configuration (Grafana Proxy)¶

Holmes CLI Holmes Helm Chart Robusta Helm Chart

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  grafana/loki:
    enabled: true
    config:
      api_key: <your grafana API key>
      api_url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
      grafana_datasource_uid: <the UID of the loki data source in Grafana>

When using the standalone Holmes Helm Chart, update your values.yaml:

toolsets:
  grafana/loki:
    enabled: true
    config:
      api_key: <your grafana API key>
      api_url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
      grafana_datasource_uid: <the UID of the loki data source in Grafana>

Apply the configuration:

helm upgrade holmes holmes/holmes --values=values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  toolsets:
    grafana/loki:
      enabled: true
      config:
        api_key: <your grafana API key>
        api_url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
        grafana_datasource_uid: <the UID of the loki data source in Grafana>

Apply the configuration:

helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Direct Connection¶

The toolset can directly connect to a Loki instance without proxying through a Grafana instance. This is done by not setting the grafana_datasource_uid field. Not setting this field makes HolmesGPT assume that it is directly connecting to Loki.

Configuration (Direct Connection)¶

Holmes CLI Holmes Helm Chart Robusta Helm Chart

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  grafana/loki:
    enabled: true
    config:
      api_url: http://loki.logging
      additional_headers:
        X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled

When using the standalone Holmes Helm Chart, update your values.yaml:

toolsets:
  grafana/loki:
    enabled: true
    config:
      api_url: http://loki.logging
      additional_headers:
        X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled

Apply the configuration:

helm upgrade holmes holmes/holmes --values=values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  toolsets:
    grafana/loki:
      enabled: true
      config:
        api_url: http://loki.logging
        additional_headers:
          X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled

Apply the configuration:

helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Advanced Configuration¶

SSL Verification¶

For self-signed certificates, you can disable SSL verification:

Holmes CLI Holmes Helm Chart Robusta Helm Chart

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  grafana/loki:
    enabled: true
    config:
      api_url: https://loki.internal
      verify_ssl: false  # Disable SSL verification (default: true)

When using the standalone Holmes Helm Chart, update your values.yaml:

toolsets:
  grafana/loki:
    enabled: true
    config:
      api_url: https://loki.internal
      verify_ssl: false  # Disable SSL verification (default: true)

Apply the configuration:

helm upgrade holmes holmes/holmes --values=values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  toolsets:
    grafana/loki:
      enabled: true
      config:
        api_url: https://loki.internal
        verify_ssl: false  # Disable SSL verification (default: true)

Apply the configuration:

helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

External URL¶

If HolmesGPT accesses Loki through an internal URL but you want clickable links in results to use a different URL:

Holmes CLI Holmes Helm Chart Robusta Helm Chart

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  grafana/loki:
    enabled: true
    config:
      api_url: http://loki.internal:3100  # Internal URL for API calls
      external_url: https://loki.example.com  # URL for links in results

When using the standalone Holmes Helm Chart, update your values.yaml:

toolsets:
  grafana/loki:
    enabled: true
    config:
      api_url: http://loki.internal:3100  # Internal URL for API calls
      external_url: https://loki.example.com  # URL for links in results

Apply the configuration:

helm upgrade holmes holmes/holmes --values=values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  toolsets:
    grafana/loki:
      enabled: true
      config:
        api_url: http://loki.internal:3100  # Internal URL for API calls
        external_url: https://loki.example.com  # URL for links in results

Apply the configuration:

helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Capabilities¶

Tool Name	Description
fetch_pod_logs	Fetches pod logs from Loki