Skip to content

Roadmap

Jump to bottom
Peter Wilcsinszky edited this page Feb 13, 2024 · 35 revisions

Collector - Tenant relationship

Currently only one collector can manage a tenant which we enforce through the tenant status. We want to allow however multiple different external or internal sources to implement the same tenancy rules. The idea to implement it is to dedicate the current Controller resource to the kubernetes log collection use case and introduce separate CRDs for use cases such as receiving telemetry from external sources (where we process not just logs but metrics and traces as well). Even for the Kubernetes collector there is a use case we can think about where the one to many relationship implemented currently is too limited, because we would need multiple connector to be able to implement the global tenant configuration. (the use case is the multiple isolated node groups with a single global infra tenant)

Verify and fix the backpressure problem

When we deal with lots of outputs, one slow output can fill up the queues. If queues are limited there will be backpressure. If there is backpressure the source will stop. The idea here is to use separate receivers per tenant, but this need to be verified.

Optimization by merging subscriptions

We could possibly optimize for the case when subscriptions have lots of overlap in their labelselector, thus might be sending the same data multiple times to the same destination. Instead of using a routerconnector for subscriptions we could possibly use a single pipeline to add all the subscriptions as subsequent processors and then use a routerconnector for the messages already labeled with the subscription id to route them to the right output.

Hot reload

Implemented in the collector but need to be wired through the operator which doesn't support it just yet.

Configcheck

Go with the simplest possible solution.

Existing alternatives currently (and possible improvement ideas)

  • silly config check is available by default
  • there is an option in the collector for syntax check, not implemented for the operator
  • implementing a full config check by running an isolated job (for the future)

See the following issue: https://github.com/open-telemetry/opentelemetry-collector/issues/4205

Persistent buffering and file position

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/filelogreceiver/README.md

Buffering and retry helper: https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md

Bindplane example: https://github.com/search?q=repo%3AobservIQ%2Fbindplane-agent+sending_queue&type=code

Please note that the collector is yet to fully support a graceful shutdown:

Collector does not yet support graceful shutdown but we plan to add it.

TODO:

Metrics

We lack a complete solution for collecting byte metrics, although we plan to use the count connector already. There is another approach that doesn't involve duplicating logs which is implemented in bindplane: https://github.com/observIQ/bindplane-agent/tree/release/v1.43.0/processor/metricextractprocessor

We have to keep considering both approaches until we can have a good measurement.

(Discussion ftr https://axoflow.slack.com/archives/C04HPRT4JH3/p1705938608863419)

TODO:

  • check opamp what kind of metrics it provides

Support other sources

  • host logs
  • file based logs through a managed sidecar container
  • logs sent to a network/otlp endpoint directly
  • kubernetes event log
Clone this wiki locally