Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PrometheusAlerts should filter by namespace (conflicts with cortex) #5260

Open
nabadger opened this issue Jun 15, 2023 · 1 comment · May be fixed by #9659
Open

PrometheusAlerts should filter by namespace (conflicts with cortex) #5260

nabadger opened this issue Jun 15, 2023 · 1 comment · May be fixed by #9659

Comments

@nabadger
Copy link

Since mimir / cortex share the same metric names, it's possible for say a Mimir alert to fire that is actually related to Cortex (and visa-versa). We've seen this on other grafana components too, such as Tempo.

There are many examples here: https://github.com/grafana/mimir/blob/main/operations/mimir-mixin-compiled/alerts.yaml

In our case, MimirRulerTooManyFailedQueries was firing but it was actually Cortex that was causing it (we are doing a migration).

I would propose that we filter these queries by namespace.
@nabadger nabadger mentioned this issue Jun 16, 2023
Open
@pracucci
Copy link
Collaborator

An alternative may be making the metrics prefix configurable so that we can export metrics with mimir_ prefix instead of cortex_ and then allow to configure it in the mixin too.

@mtweten mtweten linked a pull request Oct 17, 2024 that will close this issue
Open
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@pracucci @nabadger