Default configuration of the EDOT Collector (Kubernetes)
Elastic Stack Serverless Observability
The Kubernetes setup uses the OpenTelemetry Operator to automate orchestration of EDOT Collectors:
- EDOT Collector Cluster: Collection of cluster metrics.
- EDOT Collector Daemon: Collection of node metrics, logs and application telemetry.
- EDOT Collector Gateway: Pre-processing, aggregation and ingestion of data into Elastic.
The following values.yaml files are used depending on the ingest scenario:
The following sections describe the default pipelines for the different roles of EDOT collectors in a Kubernetes setup.
The main purpose of the Cluster collector is to collect Kubernetes cluster-level metrics and events using the k8s_cluster and the k8sobjects receivers.
The resource and resourcedetection processors enrich the cluster-level data with corresponding meta information. Data then goes to the Gateway collector through OTLP.
The Daemonset collectors gather telemetry associated with corresponding, individual Kubernetes nodes:
The filelog and hostmetrics receivers are used to gather container logs and host metrics, respectively. The kubeletstats receiver collects additional Kubernetes Node, Pod and Container metrics.
Logs and metrics are batched for better performance (batch processor) and then enriched with meta information using the k8sattributes, resourcedetection and resource processors.
The Daemonset collectors also receive the application telemetry from OTel SDKs that instrument services and pods running on corresponding Kubernetes nodes.
The Daemonset collectors receive that data through OTLP, batch the data (batch processor) and pass it on to the Gateway Collector through the OTLP exporter.
The Gateway collectors pipelines differ between the two different deployment use cases, direct ingestion into Elasticsearch and using Elastic's Managed OTLP Endpoint.
In self-managed and Elastic Cloud Hosted Stack deployment use cases, the main purpose of the Gateway collector is the central enrichment of data before the OpenTelemetry data is being ingested directly into Elasticsearch using the elasticsearch exporter.
The Gateway collector configuration comprises the pipelines for data enrichment of application telemetry and host metrics. For more details, refer to the linked descriptions of the corresponding standalone use cases.
The routing connector separates the infrastructure metrics from other metrics and routes them into the ECS-based pipeline, with ECS-compatibility exporter mode. Other metrics are exported in OTel-native format to Elasticsearch.
With the managed OTLP Endpoint, the Gateway collector configuration pipes all the data from the OTLP receiver through a batch processor before the data is being exported through OTLP to the managed endpoint.
With this scenario there's no need to do any Elastic-specific enrichment in your Kubernetes cluster, as all of that happens behind the managed OTLP endpoint.