Documentation Grafana Alloy Reference Components discovery discovery.kubernetes

discovery.kubernetes

discovery.kubernetes allows you to find scrape targets from Kubernetes resources. It watches cluster state and ensures targets are continually synced with what’s running in your cluster.

If you supply no connection information, this component defaults to an in-cluster configuration. You can use a kubeconfig file or manual connection settings to override the defaults.

Performance considerations

By default, discovery.kubernetes discovers resources across all namespaces in your cluster.

For better performance and reduced API load:

• Use the namespaces block to limit discovery to specific namespaces.
• Use selectors to filter resources by labels or fields.
• Consider the node-local example in Limit to only Pods on the same node.
• Use discovery.kubelet for DaemonSet deployments to discover only Pods on the local node.
• Use clustering mode for larger deployments to distribute the discovery load.
• Monitor API server metrics like request rate, throttling, and memory usage, especially on managed clusters.

Usage

Alloy Copy

discovery.kubernetes "<LABEL>" {
  role = "<DISCOVERY_ROLE>"
}

Arguments

You can use the following arguments with discovery.kubernetes:

You can provide at most one of the following:

• authorization block
• basic_auth block
• bearer_token_file argument
• bearer_token argument
• oauth2 block

The role argument specifies the type of targets to discover. role must be one of node, pod, service, endpoints, endpointslice, or ingress.

node role

The node role discovers one target per cluster node with the address defaulting to the HTTP port of the kubelet daemon. The target address defaults to the first address of the Kubernetes node object in the address type order of NodeInternalIP, NodeExternalIP, NodeLegacyHostIP, and NodeHostName.

Discovered nodes include the following labels:

• __meta_kubernetes_node_address_<address_type>: The first address for each node address type, if it exists.
• __meta_kubernetes_node_annotation_<annotationname>: Each annotation from the node object.
• __meta_kubernetes_node_annotationpresent_<annotationname>: Set to true for each annotation from the node object.
• __meta_kubernetes_node_label_<labelname>: Each label from the node object.
• __meta_kubernetes_node_labelpresent_<labelname>: Set to true for each label from the node object.
• __meta_kubernetes_node_name: The name of the node object.
• __meta_kubernetes_node_provider_id: The cloud provider’s name for the node object.

In addition, the component sets the instance label for the node to the node name retrieved from the API server.

service role

The service role discovers a target for each port of each service. This is generally useful for externally monitoring a service. The component sets the address to the Kubernetes DNS name of the service and respective service port.

Discovered services include the following labels:

• __meta_kubernetes_namespace: The namespace of the service object.
• __meta_kubernetes_service_annotation_<annotationname>: Each annotation from the service object.
• __meta_kubernetes_service_annotationpresent_<annotationname>: true for each annotation of the service object.
• __meta_kubernetes_service_cluster_ip: The cluster IP address of the service. This doesn’t apply to services of type ExternalName.
• __meta_kubernetes_service_external_name: The DNS name of the service. This only applies to services of type ExternalName.
• __meta_kubernetes_service_label_<labelname>: Each label from the service object.
• __meta_kubernetes_service_labelpresent_<labelname>: true for each label of the service object.
• __meta_kubernetes_service_name: The name of the service object.
• __meta_kubernetes_service_port_name: Name of the service port for the target.
• __meta_kubernetes_service_port_number: Number of the service port for the target.
• __meta_kubernetes_service_port_protocol: Protocol of the service port for the target.
• __meta_kubernetes_service_type: The type of the service.

pod role

The pod role discovers all Pods and exposes their containers as targets. The component generates a single target for each declared port of a container.

If a container has no specified ports, the component creates a port-free target per container. You must manually inject a port using a discovery.relabel component before you can collect metrics from these targets.

Discovered Pods include the following labels:

• __meta_kubernetes_namespace: The namespace of the Pod object.
• __meta_kubernetes_pod_annotation_<annotationname>: Each annotation from the Pod object.
• __meta_kubernetes_pod_annotationpresent_<annotationname>: true for each annotation from the Pod object.
• __meta_kubernetes_pod_container_id: ID of the container the target address points to. The ID is in the form <type>://<container_id>.
• __meta_kubernetes_pod_container_image: The container image.
• __meta_kubernetes_pod_container_init: true if the container is an InitContainer.
• __meta_kubernetes_pod_container_name: Name of the container the target address points to.
• __meta_kubernetes_pod_container_port_name: Name of the container port.
• __meta_kubernetes_pod_container_port_number: Number of the container port.
• __meta_kubernetes_pod_container_port_protocol: Protocol of the container port.
• __meta_kubernetes_pod_controller_kind: Object kind of the Pod controller.
• __meta_kubernetes_pod_controller_name: Name of the Pod controller.
• __meta_kubernetes_pod_host_ip: The current host IP of the Pod object.
• __meta_kubernetes_pod_ip: The Pod IP of the Pod object.
• __meta_kubernetes_pod_label_<labelname>: Each label from the Pod object.
• __meta_kubernetes_pod_labelpresent_<labelname>: true for each label from the Pod object.
• __meta_kubernetes_pod_name: The name of the Pod object.
• __meta_kubernetes_pod_node_name: The name of the node where the Pod runs.
• __meta_kubernetes_pod_phase: Set to Pending, Running, Succeeded, Failed, or Unknown in the lifecycle.
• __meta_kubernetes_pod_ready: Set to true or false for the Pod’s ready state.
• __meta_kubernetes_pod_uid: The UID of the Pod object.

endpoints role

The endpoints role discovers targets from listed endpoints of a service. The component discovers one target per port for each endpoint address. If a Pod backs the endpoint, the component discovers all container ports of the Pod as targets even if they aren’t bound to an endpoint port.

Discovered endpoints include the following labels:

• __meta_kubernetes_endpoints_label_<labelname>: Each label from the endpoints object.

• __meta_kubernetes_endpoints_labelpresent_<labelname>: true for each label from the endpoints object.

• __meta_kubernetes_endpoints_name: The name of the endpoints object.

• __meta_kubernetes_namespace: The namespace of the endpoints object.

• The component attaches the following labels to all targets discovered directly from the endpoints list:

  • __meta_kubernetes_endpoint_address_target_kind: Kind of the endpoint address target.
  • __meta_kubernetes_endpoint_address_target_name: Name of the endpoint address target.
  • __meta_kubernetes_endpoint_hostname: Hostname of the endpoint.
  • __meta_kubernetes_endpoint_node_name: Name of the node hosting the endpoint.
  • __meta_kubernetes_endpoint_port_name: Name of the endpoint port.
  • __meta_kubernetes_endpoint_port_protocol: Protocol of the endpoint port.
  • __meta_kubernetes_endpoint_ready: Set to true or false for the endpoint’s ready state.

• If the endpoints belong to a service, all labels of the service role discovery are also included.

• If a Pod backs the target, all labels of the pod role discovery are also included.

endpointslice role

The endpointslice role discovers targets from Kubernetes endpoint slices. The component discovers one target for each endpoint address referenced in the EndpointSlice object. If a Pod backs the endpoint, the component discovers all container ports of the Pod as targets even if they’re not bound to an endpoint port.

Discovered endpoint slices include the following labels:

• __meta_kubernetes_endpointslice_name: The name of the endpoint slice object.

• __meta_kubernetes_namespace: The namespace of the endpoints object.

• The component attaches the following labels to all targets discovered directly from the endpoint slice list:

  • __meta_kubernetes_endpointslice_address_target_kind: Kind of the referenced object.
  • __meta_kubernetes_endpointslice_address_target_name: Name of the referenced object.
  • __meta_kubernetes_endpointslice_address_type: The IP protocol family of the target address.
  • __meta_kubernetes_endpointslice_endpoint_conditions_ready: Set to true or false for the referenced endpoint’s ready state.
  • __meta_kubernetes_endpointslice_endpoint_topology_kubernetes_io_hostname: Name of the node hosting the referenced endpoint.
  • __meta_kubernetes_endpointslice_endpoint_topology_present_kubernetes_io_hostname: true if the referenced object has a kubernetes.io/hostname annotation.
  • __meta_kubernetes_endpointslice_endpoint_hostname: Hostname of the referenced endpoint.
  • __meta_kubernetes_endpointslice_endpoint_node_name: Name of the node hosting the referenced endpoint.
  • __meta_kubernetes_endpointslice_endpoint_zone: The zone where the referenced endpoint exists. Only available when using the discovery.k8s.io/v1 API group.
  • __meta_kubernetes_endpointslice_port_name: Name of the port for the referenced endpoint.
  • __meta_kubernetes_endpointslice_port_protocol: Protocol of the referenced endpoint.
  • __meta_kubernetes_endpointslice_port: Port of the referenced endpoint.

• If the endpoints belong to a service, all labels of the service role discovery are also included.

• If a Pod backs the target, all labels of the pod role discovery are also included.

ingress role

The ingress role discovers a target for each path of each ingress. This is generally useful for externally monitoring an ingress. The component sets the address to the host specified in the Kubernetes Ingress’s spec block.

Discovered ingress objects include the following labels:

• __meta_kubernetes_ingress_annotation_<annotationname>: Each annotation from the ingress object.
• __meta_kubernetes_ingress_annotationpresent_<annotationname>: true for each annotation from the ingress object.
• __meta_kubernetes_ingress_class_name: Class name from ingress spec, if present.
• __meta_kubernetes_ingress_label_<labelname>: Each label from the ingress object.
• __meta_kubernetes_ingress_labelpresent_<labelname>: true for each label from the ingress object.
• __meta_kubernetes_ingress_name: The name of the ingress object.
• __meta_kubernetes_ingress_path: Path from ingress spec. Defaults to /.
• __meta_kubernetes_ingress_scheme: Protocol scheme of ingress, https when using TLS. Defaults to http.
• __meta_kubernetes_namespace: The namespace of the ingress object.

Blocks

You can use the following blocks with discovery.kubernetes:

The > symbol indicates deeper levels of nesting. For example, oauth2 > tls_config refers to a tls_config block defined inside an oauth2 block.

attach_metadata

The attach_metadata block allows you to attach node metadata to discovered targets.

authorization

The authorization block configures generic authorization to the endpoint.

basic_auth

The basic_auth block configures basic authentication to the endpoint.

namespaces

The namespaces block limits the namespaces to discover resources in. If you omit this block, the component searches all namespaces.

oauth2

The oauth2 block configures OAuth 2.0 authentication to the endpoint.

selectors

The selectors block contains optional label and field selectors to limit the discovery process to a subset of resources.

Refer to Kubernetes’ documentation for Field selectors and Labels and selectors to learn more about the filters you can use.

The endpoints role supports Pod, service, and endpoints selectors. The Pod role supports node selectors when configured with attach_metadata: {node: true}. Other roles only support selectors matching the role itself. For example, node role can only contain node selectors.

tls_config

The tls_config block configures TLS settings for connecting to the endpoint.

Exported fields

discovery.kubernetes exports the following fields that other components can reference:

Component health

discovery.kubernetes reports as unhealthy when you provide an invalid configuration. In those cases, exported fields retain their last healthy values.

Debug information

discovery.kubernetes doesn’t expose any component-specific debug information.

Debug metrics

discovery.kubernetes doesn’t expose any component-specific debug metrics.

Examples

In-cluster discovery

This example uses in-cluster authentication to discover all Pods:

Alloy Copy

discovery.kubernetes "k8s_pods" {
  role = "pod"
}

prometheus.scrape "demo" {
  targets    = discovery.kubernetes.k8s_pods.targets
  forward_to = [prometheus.remote_write.demo.receiver]
}

prometheus.remote_write "demo" {
  endpoint {
    url = "<PROMETHEUS_REMOTE_WRITE_URL>"

    basic_auth {
      username = "<USERNAME>"
      password = "<PASSWORD>"
    }
  }
}

Replace the following:

• <PROMETHEUS_REMOTE_WRITE_URL>: The URL of the Prometheus remote_write-compatible server to send metrics to.
• <USERNAME>: The username to use for authentication to the remote_write API.
• <PASSWORD>: The password to use for authentication to the remote_write API.

kubeconfig file authentication

This example uses a kubeconfig file to authenticate to the Kubernetes API:

Alloy Copy

discovery.kubernetes "k8s_pods" {
  role = "pod"
  kubeconfig_file = "/etc/k8s/kubeconfig.yaml"
}

prometheus.scrape "demo" {
  targets    = discovery.kubernetes.k8s_pods.targets
  forward_to = [prometheus.remote_write.demo.receiver]
}

prometheus.remote_write "demo" {
  endpoint {
    url = "<PROMETHEUS_REMOTE_WRITE_URL>"

    basic_auth {
      username = "<USERNAME>"
      password = "<PASSWORD>"
    }
  }
}

Replace the following:

• <PROMETHEUS_REMOTE_WRITE_URL>: The URL of the Prometheus remote_write-compatible server to send metrics to.
• <USERNAME>: The username to use for authentication to the remote_write API.
• <PASSWORD>: The password to use for authentication to the remote_write API.

Limit searched namespaces and filter by label

This example limits the searched namespaces and selects only Pods with a specific label:

Alloy Copy

discovery.kubernetes "k8s_pods" {
  role = "pod"

  selectors {
    role = "pod"
    label = "app.kubernetes.io/name=prometheus-node-exporter"
  }

  namespaces {
    names = ["myapp"]
  }
}

prometheus.scrape "demo" {
  targets    = discovery.kubernetes.k8s_pods.targets
  forward_to = [prometheus.remote_write.demo.receiver]
}

prometheus.remote_write "demo" {
  endpoint {
    url = "<PROMETHEUS_REMOTE_WRITE_URL>"

    basic_auth {
      username = "<USERNAME>"
      password = "<PASSWORD>"
    }
  }
}

Replace the following:

• <PROMETHEUS_REMOTE_WRITE_URL>: The URL of the Prometheus remote_write-compatible server to send metrics to.
• <USERNAME>: The username to use for authentication to the remote_write API.
• <PASSWORD>: The password to use for authentication to the remote_write API.

Limit to only Pods on the same node

This example limits the search to Pods on the same node as this Alloy. Use this configuration when running Alloy as a DaemonSet to significantly reduce API server load and memory usage by only watching local Pods instead of all Pods cluster-wide.

Alloy Copy

discovery.kubernetes "k8s_pods" {
  role = "pod"
  selectors {
    role = "pod"
    field = "spec.nodeName=" + coalesce(sys.env("HOSTNAME"), constants.hostname)
  }
}

prometheus.scrape "demo" {
  targets    = discovery.kubernetes.k8s_pods.targets
  forward_to = [prometheus.remote_write.demo.receiver]
}

prometheus.remote_write "demo" {
  endpoint {
    url = "<PROMETHEUS_REMOTE_WRITE_URL>"

    basic_auth {
      username = "<USERNAME>"
      password = "<PASSWORD>"
    }
  }
}

Replace the following:

• <PROMETHEUS_REMOTE_WRITE_URL>: The URL of the Prometheus remote_write-compatible server to send metrics to.
• <USERNAME>: The username to use for authentication to the remote_write API.
• <PASSWORD>: The password to use for authentication to the remote_write API.

Compatible components

discovery.kubernetes has exports that can be consumed by the following components:

• Components that consume Targets