High distributor % for single node in monolith deploy.

[jzakhar]

We've have 3 Mimir nodes setup for sometime now and recently during a normal update -> rolling reboot one of our ops engineers noticed that the distributor ring was being pinned to one node.

Instance ID | Availability Zone | State | Address | Registered At | Last Heartbeat | Tokens | Ownership | Actions

instance1.domain.com |   | ACTIVE | 1.1.1.185:9095 | 2024-09-17T14:01:24Z | 14.134s ago (17:19:39) | 1 | 3.84% | Forget
instance2.domain.com |   | ACTIVE | 1.1.1.194:9095 | 2024-09-17T14:07:27Z | 11.135s ago (17:19:42) | 1 | 0.77% | Forget
instance3.domain.com |   | ACTIVE | 1.1.1.140:9095 | 2024-09-17T14:11:14Z | 9.135s ago (17:19:44) | 1 | 95.4% | Forget

That third instance is 95%, if we reload the process that 95% moves to somewhere else. I also see that there is only 1 token available.

The config we run is :

alertmanager:
    data_dir: /data/alertmanager
    external_url: http://localhost:9009/alertmanager
alertmanager_storage:
    backend: s3
    s3:
        access_key_id: abc123
        bucket_name: mimirmetrics
        endpoint: 1.1.1.95:9000
        insecure: true
        secret_access_key: '********'
blocks_storage:
    backend: s3
    bucket_store:
        chunks_cache:
            backend: memcached
            memcached:
                addresses: 1.1.1.50:11211,1.1.1.51:11211
        index_cache:
            backend: memcached
            memcached:
                addresses: 1.1.1.50:11211,1.1.1.51:11211
        metadata_cache:
            backend: memcached
            memcached:
                addresses: 1.1.1.50:11211,1.1.1.51:11211
        sync_dir: /datastore/mimir/tsdb-sync/
    s3:
        access_key_id: abc123
        bucket_name: mimirblocks
        endpoint:1.1.1.95:9000
        insecure: true
        secret_access_key: '********'
    tsdb:
        dir: /datastore/mimir/tsdb/
common:
    storage:
        backend: s3
        s3:
            access_key_id: abc123
            bucket_name: mimirmetrics
            endpoint: 1.1.1.95:9000
            insecure: true
            secret_access_key: '********'
distributor:
    max_recv_msg_size: 209715200
    remote_timeout: 10s
frontend:
    align_queries_with_step: true
    cache_results: true
    results_cache:
        backend: memcached
        memcached:
            addresses: 1.1.1.50:11211,1.1.1.51:11211
ingester_client:
    grpc_client_config:
        max_recv_msg_size: 209715200
        max_send_msg_size: 209715200
limits:
    compactor_block_upload_enabled: true
    ingestion_burst_size: 8000000
    ingestion_rate: 400000
    max_global_series_per_user: 12000000
    max_label_names_per_series: 120
    out_of_order_time_window: 12d
memberlist:
    cluster_label: somelabel
    join_members:
        - 1.1.1.185
        - 1.1.1.194
        - 1.1.1.140
ruler:
    alertmanager_url: http://127.0.0.1:8080/alertmanager
    ring:
        heartbeat_period: 2s
        heartbeat_timeout: 10s
    rule_path: /data/ruler
ruler_storage:
    backend: s3
    cache:
        backend: memcached
        memcached:
            addresses: 1.1.1.50:11211,1.1.1.51:11211
    s3:
        access_key_id: abc123
        bucket_name: mimirrules-storage
        endpoint: 1.1.1.95:9000
        insecure: true
        secret_access_key: '********'
runtime_config:
    file: /etc/mimir/runtime_config.yml
server:
    grpc_server_max_recv_msg_size: 209715200
    grpc_server_max_send_msg_size: 209715200
target: all,alertmanager,overrides-exporter
usage_stats:
    enabled: false

Do we need to add num_tokens under the ring stanza for the distributer like the ingester has ?

The ingester defaults + our changes come out looking like this :

ingester:
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 15s
        heartbeat_timeout: 1m0s
        replication_factor: 3
        zone_awareness_enabled: false
        excluded_zones: ""
        tokens_file_path: ""
        num_tokens: 128
        instance_id: <hostname>
        instance_interface_names:
            - team0
        instance_port: 0
        instance_addr: ""
        instance_enable_ipv6: false
        instance_availability_zone: ""
        unregister_on_shutdown: true
        observe_period: 0s
        min_ready_duration: 15s
        final_sleep: 0s
        token_generation_strategy: random
        spread_minimizing_join_ring_in_order: false
        spread_minimizing_zones: ""
    partition_ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        min_partition_owners_count: 1
        min_partition_owners_duration: 10s
        delete_inactive_partition_after: 13h0m0s
    metadata_retain_period: 10m0s
    rate_update_period: 15s
    active_series_metrics_enabled: true
    active_series_metrics_update_period: 1m0s
    active_series_metrics_idle_timeout: 10m0s
    tsdb_config_update_period: 15s
    instance_limits:
        max_ingestion_rate: 0
        max_tenants: 0
        max_series: 0
        max_inflight_push_requests: 30000
        max_inflight_push_requests_bytes: 0
    ignore_series_limit_for_metric_names: ""
    read_path_cpu_utilization_limit: 0
    read_path_memory_utilization_limit: 0
    log_utilization_based_limiter_cpu_samples: false
    limit_inflight_requests_using_grpc_method_limiter: true
    error_sample_rate: 10
    return_only_grpc_errors: true
    use_ingester_owned_series_for_limits: false
    track_ingester_owned_series: false
    owned_series_update_interval: 15s
    push_circuit_breaker:
        enabled: false
        failure_threshold_percentage: 10
        failure_execution_threshold: 100
        thresholding_period: 1m0s
        cooldown_period: 10s
        initial_delay: 0s
        request_timeout: 2s
    read_circuit_breaker:
        enabled: false
        failure_threshold_percentage: 10
        failure_execution_threshold: 100
        thresholding_period: 1m0s
        cooldown_period: 10s
        initial_delay: 0s
        request_timeout: 30s
    push_grpc_method_enabled: true

Which has num_tokens while the distributor does not.

distributor:
    pool:
        client_cleanup_period: 15s
        health_check_ingesters: true
    retry_after_header:
        enabled: false
        base_seconds: 3
        max_backoff_exponent: 5
    ha_tracker:
        enable_ha_tracker: false
        ha_tracker_update_timeout: 15s
        ha_tracker_update_timeout_jitter_max: 5s
        ha_tracker_failover_timeout: 30s
        kvstore:
            store: consul
            prefix: ha-tracker/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
    max_recv_msg_size: 209715200
    max_request_pool_buffer_size: 0
    remote_timeout: 10s
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 15s
        heartbeat_timeout: 1m0s
        instance_id: <instance>
        instance_interface_names:
            - team0
        instance_port: 0
        instance_addr: ""
        instance_enable_ipv6: false
    instance_limits:
        max_ingestion_rate: 0
        max_inflight_push_requests: 2000
        max_inflight_push_requests_bytes: 0
    write_requests_buffer_pooling_enabled: true
    limit_inflight_requests_using_grpc_method_limiter: true
    reusable_ingester_push_workers: 2000
    direct_otlp_translation_enabled: true

anyway, any help is much appreciated.

TYIA