Successfully uploaded TSDB block data via mimirtool, but no data on Grafana dashboard

[nhyphamnt]

Hi everyone,
I am currently having issue with the mimir backfill data. I successfully uploaded all the tsdb block to mimir, and can confirmed it in our Object Storage (nfs server via minio). But data was not shown in our Grafana metrics explore. I tried to restart the memory cache pod and also waited almost a day following this post (#3735) but it did not help. Could you please advise if I am doing wrong in some points. I put details in the comment. Many thanks.
Condition:

• Grafana, Prometheus, Mimir are hosted on a Kubernetes cluster
• Using minio Object storage to have mimir data in our nfs server

Steps done:

1. Created data as Openmetric format

data_created{project="A",customer="a"} 170364955325 1726876808
data_created{project="A",customer="a"} 170364955325 1726876808
data_created{project="A",customer="a"} 170364955325 1726876808
data_created{project="A",customer="a"} 170364955325 1726876808
data_created{project="A",customer="b"} 170364955325 1726876808

2. Using promtool to create tsdb data block

Data$ promtool tsdb list tsdb_output                                                                                                                                              
BLOCK ULID                  MIN TIME       MAX TIME       DURATION     NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE                                                                                                 
01J8MGZ11ZB190S52MC0BJ22FP  1726876808000  1726876808001  1ms          3            3            3            904                                                                                                  
01J8MGZ16KT48W8GMH07QDDTSC  1726876808000  1726876808001  1ms          1            1            1            684 

3. Uploaded this tsdb block to mimir using mimir tool. Got the successful message

./tsdb_output1/01J8MGZ11ZB190S52MC0BJ22FP  
INFO[0000] Backfilling                                   blocks=./tsdb_output1/01J8MGZ11ZB190S52MC0BJ22FP  user=anonymous
INFO[0000] making request to start block upload          block=01J8MGZ11ZB190S52MC0BJ22FP  file=meta.json path=./tsdb_output1/01J8Q12KH2HCC58DKP2YEKAYQ7
INFO[0000] uploading block file                          block=01J8Q12KH2HCC58DKP2YEKAYQ7 file=index path=./tsdb_output1/01J8Q12KH2HCC58DKP2YEKAYQ7 size=904
INFO[0000] uploading block file                          block=01J8Q12KH2HCC58DKP2YEKAYQ7 file=chunks/000001 path=./tsdb_output1/01J8MGZ11ZB190S52MC0BJ22FP  size=904
INFO[0020] block uploaded successfully                   block=01J8MGZ11ZB190S52MC0BJ22FP  path=./tsdb_output1/01J8MGZ11ZB190S52MC0BJ22FP  
INFO[0020] finished uploading blocks                     already_exists=0 failed=0 succeeded=1

After uploading, the folder has this data hiar:

  rwxr-xr-x 2 root root 29 Sep 26 23:44 uploading-meta.json
drwxr-xr-x 2 root root 29 Sep 26 23:44 index
drwxr-xr-x 3 root root 28 Sep 26 23:44 chunks
drwxr-xr-x 2 root root 29 Sep 26 23:44 validation.json
drwxr-xr-x 2 root root 29 Sep 26 23:44 meta.json

Currently the status of these blocks was changed to delete-mark.

drwxr-xr-x 2 root root 29 Sep 25 22:08 uploading-meta.json
drwxr-xr-x 2 root root 29 Sep 25 22:08 index
drwxr-xr-x 3 root root 28 Sep 25 22:08 chunks
drwxr-xr-x 2 root root 29 Sep 25 22:08 validation.json
drwxr-xr-x 2 root root 29 Sep 25 22:08 meta.json
drwxr-xr-x 2 root root 29 Sep 25 22:56 deletion-mark.json

I am not sure what I did wrong here, data did not appear on Grafana metric explore. Please kindly help. Many thanks.

[pstibrany]

Hello. There are two possible reasons why block may be marked for deletion:

• either it fell out of retention window
• or block was merged (compacted) with another block from the same time range

Based on your

promtool tsdb list tsdb_output
BLOCK ULID                  MIN TIME       MAX TIME       DURATION     NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE
01J8MGZ11ZB190S52MC0BJ22FP  1726876808000  1726876808001  1ms          3            3            3            904
01J8MGZ16KT48W8GMH07QDDTSC  1726876808000  1726876808001  1ms          1            1            1            684 

and mimirtool backfill output, block 01J8MGZ11ZB190S52MC0BJ22FP was uploaded. This block has 3 series, each with exactly 1 sample at Sat Sep 21 00:00:08 UTC 2024. Did you query the right time range to see this one sample?

[nhyphamnt]

Many thanks @pstibrany for your prompt response. The idea is every day we will upload the data usage then we want to sum the data for every month. That's why we want to upload it just once a day, as specific time. This will be reflected using textfile_collector once a day later. Currently we want to put the historic data first.
Thus, we have data for just 1 sample for one series at the same time range, with different labels. Is this ok to have data like that?

# HELP dmf_data_volume_created My metric.
# TYPE dmf_data_volume_created counter
data_created{project="A",vg="v1_8",pool="a"} 1477410031051 1725321608
data_created{project="A",vg="v_8",pool="a"} 519875 1725321608
data_created{project="A",vg="v2_8",pool="b"} 519875 1725321608
data_created{project="A",vg="v_8",pool="b"} 1477410031051 1725321608
data_created{project="A",vg="v_8",pool="c"} 1477410031051 1725321608
data_created{project="C",vg="v3_8",pool="a"} 2073645102173 1725321608
data_created{project="C",vg="vc_8",pool="c"} 2073645102173 1725321608
data_created{project="R",vg="v_8",pool="a"} 9546910033 1725321608
data_created{project="R",vg="v_8",pool="b"} 9546910033 1725321608
data_created{project="R",vg="v_6",pool="c"} 9546910033 1725321608
# EOF

Thank you again and I am waiting for your advice.

[pstibrany]

Hello. It's fine to have data like this. As mentioned earlier your block was uploaded correctly, but you have trouble querying it and block was also marked for deletion.

To figure out why block is marked for deletion, I'd try to find compactor logs for given block ID, to see which of the above mentioned cases (retention or compaction) made compactor to mark block for deletion.

For querying, what is your query and time range?

[nhyphamnt]

Thank you @pstibrany and I apologise for the delayed response. I am currently on vacation and will be back in 10 days. I will collect logs and posted here as soon as I have access again to my work computer.