/prometheus endpoint gives 404 error page

[ryayon]

Describe the bug

http://<ingress-host>/prometheus gives 404 error page.
Also get 404 error from mimir-nginx pod using curl http://mimir-distributed-nginx.<namespace>.svc/prometheus

Both api/v1/status/buildinfo and api/v1/status/buildinfo endpoints are OK both Ingress and mimir-nginx pod (using internal service name)

To Reproduce

Just follow the documentation

Expected behavior

/prometheus endpoint should be requestable both from navigator or Grafana

Environment

• Infrastructure: Kubernetes
• Deployment tool: helm
  [EDIT] v5.2.0-weekly.270

Additional Context

alertmanager:
  enabled: false

mimir:
  structuredConfig:
    common:
      storage:
        backend: s3
        s3:
          endpoint: 'XXXXXX'
          access_key_id: 'YYYYYYYY'
          secret_access_key: 'ZZZZZZZZZ'
          insecure: true
          bucket_name: mimir
    blocks_storage:
      backend: s3
      storage_prefix: blocks

nginx:
  ingress:
    enabled: true
    hosts:
      - host: mimir.sndsdskd.qdqdqd
        paths:
          - path: /
            pathType: Prefix

overrides_exporter:
  enabled: false

query_scheduler:
  enabled: false

ruler:
  enabled: false

[rosh0450]

Yes even i am getting the same issue

I can see the /prometheus is already set to be forwarded to query-frontend's service in nginx configmap

# Rest of /prometheus goes to the query frontend
location /prometheus {
  set $query_frontend mimir-query-frontend.mimir.svc.cluster.local;
  proxy_pass      http://$query_frontend:8080$request_uri;
}

@pracucci @krajorama @pstibrany How to access prometheus dashboard for grafana mimir.

Similarly, there are various other endpoints mentioned in the https://grafana.com/docs/mimir/latest/references/http-api/#grafana-mimir-http-api docs which are not accessible. What's the reason behind this?

Thanks.

[pstibrany]

Hello. http://<ingress-host>/prometheus is URL that should be used in Grafana datasource URL field ("Prometheus server URL"). Grafana adds additional path segments to it to form final URL, for example http://<ingress-host>/prometheus/api/v1/query_range for querying, or http://<ingress-host>/prometheus/api/v1/labels for fetching available labels. (These endpoints return JSON data, and are not suitable for usage in your browser)

Mimir itself doesn't have any UI for querying, and can only be used in combination with Grafana, or other tool that can query Prometheus-compatible API.

[ryayon]

Thank you @pstibrany for the explanation.
Indeed, I get something via https://<ingress-host>/prometheus/api/v1/labels.
However, using the connection from Grafana, I get the error Post "https://<ingress-host>/prometheus/api/v1/query": dial XXX.XXX.XXX.XXX:443: i/o timeout - There was an error returned querying the Prometheus API..
The only possible issue I see is in the querier logs where I get a component="memberlist TCPTransport" msg="unknown message type" error everytime.
I close this discussion and might open a new one for this last issue if I can't find a solution.

[rosh0450]

Thanks @pstibrany

[celesteking]

ingress-host is incorrect denomination of the host willing to serve /prometheus endpoint. It should be called a host that is configured to run Querier, Query-frontend, Rules services instead.

The list of endpoints is at https://grafana.com/docs/mimir/latest/references/http-api/#endpoints