从多个集群收集可观测性数据 - Red Hat 版本的 OpenTelemetry | 可观测性 | OpenShift Container Platform 4.17

对于多集群配置，您可以在每个远程集群中创建一个 OpenTelemetry Collector 实例，然后将所有遥测数据转发到一个 OpenTelemetry Collector 实例。

先决条件

已安装 Red Hat 版本的 OpenTelemetry Operator。
已安装 Tempo Operator。
已在集群上部署 TempoStack 实例。
以下已挂载的证书：颁发者、自签名证书、CA 颁发者、客户端和服务器证书。要创建任何这些证书，请参阅步骤 1。

步骤

在 OpenTelemetry Collector 实例中挂载以下证书，跳过已挂载的证书。

一个颁发者，用于使用 Red Hat OpenShift 的 cert-manager 运算符生成证书。

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}

一个自签名证书。

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca
spec:
  isCA: true
  commonName: ca
  subject:
    organizations:
      - Organization # <your_organization_name>
    organizationalUnits:
      - Widgets
  secretName: ca-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer
    group: cert-manager.io

一个 CA 颁发者。

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: test-ca-issuer
spec:
  ca:
    secretName: ca-secret

客户端和服务器证书。

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: server
spec:
  secretName: server-tls
  isCA: false
  usages:
    - server auth
    - client auth
  dnsNames:
  - "otel.observability.svc.cluster.local" (1)
  issuerRef:
    name: ca-issuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: client
spec:
  secretName: client-tls
  isCA: false
  usages:
    - server auth
    - client auth
  dnsNames:
  - "otel.observability.svc.cluster.local" (2)
  issuerRef:
    name: ca-issuer

1	要映射到服务器 OpenTelemetry Collector 实例中的求解器的确切 DNS 名称列表。
2	要映射到客户端 OpenTelemetry Collector 实例中的求解器的确切 DNS 名称列表。

为 OpenTelemetry Collector 实例创建一个服务帐户。

示例 ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector-deployment

为服务帐户创建一个集群角色。

示例 ClusterRole

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector
rules:
  (1)
  (2)
- apiGroups: ["", "config.openshift.io"]
  resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
  verbs: ["get", "watch", "list"]

1	`k8sattributesprocessor` 需要 Pod 和命名空间资源的权限。
2	`resourcedetectionprocessor` 需要基础架构和状态的权限。

将集群角色绑定到服务帐户。

示例 ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-collector
subjects:
- kind: ServiceAccount
  name: otel-collector-deployment
  namespace: otel-collector-<example>
roleRef:
  kind: ClusterRole
  name: otel-collector
  apiGroup: rbac.authorization.k8s.io

创建 YAML 文件以定义边缘集群中的OpenTelemetryCollector自定义资源 (CR)。

边缘集群的OpenTelemetryCollector自定义资源示例

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: otel-collector-<example>
spec:
  mode: daemonset
  serviceAccount: otel-collector-deployment
  config: |
    receivers:
      jaeger:
        protocols:
          grpc: {}
          thrift_binary: {}
          thrift_compact: {}
          thrift_http: {}
      opencensus:
      otlp:
        protocols:
          grpc: {}
          http: {}
      zipkin: {}
    processors:
      batch: {}
      k8sattributes: {}
      memory_limiter:
        check_interval: 1s
        limit_percentage: 50
        spike_limit_percentage: 30
      resourcedetection:
        detectors: [openshift]
    exporters:
      otlphttp:
        endpoint: https://observability-cluster.com:443 (1)
        tls:
          insecure: false
          cert_file: /certs/server.crt
          key_file: /certs/server.key
          ca_file: /certs/ca.crt
    service:
      pipelines:
        traces:
          receivers: [jaeger, opencensus, otlp, zipkin]
          processors: [memory_limiter, k8sattributes, resourcedetection, batch]
          exporters: [otlp]
  volumes:
    - name: otel-certs
      secret:
        name: otel-certs
  volumeMounts:
    - name: otel-certs
      mountPath: /certs

1	Collector 导出器配置为导出 OTLP HTTP 并指向来自中心集群的 OpenTelemetry Collector。

创建 YAML 文件以定义中心集群中的OpenTelemetryCollector自定义资源 (CR)。

中心集群的OpenTelemetryCollector自定义资源示例

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otlp-receiver
  namespace: observability
spec:
  mode: "deployment"
  ingress:
    type: route
    route:
      termination: "passthrough"
  config: |
    receivers:
      otlp:
        protocols:
          http:
            tls: (1)
              cert_file: /certs/server.crt
              key_file: /certs/server.key
              client_ca_file: /certs/ca.crt
    exporters:
      logging: {}
      otlp:
        endpoint: "tempo-<simplest>-distributor:4317" (2)
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [otlp]
  volumes:
    - name: otel-certs
      secret:
        name: otel-certs
  volumeMounts:
    - name: otel-certs
      mountPath: /certs

1	Collector 接收器需要步骤一中列出的证书。
2	Collector 导出器配置为导出 OTLP 并指向 Tempo 分发器端点，在此示例中为`"tempo-simplest-distributor:4317"`，并且已创建。