Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Additional storage backends #638

Open
15 of 22 tasks
yurishkuro opened this issue Jan 8, 2018 · 53 comments
Open
15 of 22 tasks

Additional storage backends #638

yurishkuro opened this issue Jan 8, 2018 · 53 comments
Labels
area/storage enhancement meta-issue An tracking issue that requires work in other repos

Comments

@yurishkuro
Copy link
Member

yurishkuro commented Jan 8, 2018

Opening this issue to keep track of other related issues.

Relevant issue: plugin support #422 (done).

@nbettiol
Copy link

nbettiol commented Jan 9, 2018

Did you remove the flags for elasticsearch in jaeger-collector? Because I'm doing a test using the image docker, which version is:

{"gitCommit":"dbd5db721fc59431b1e64874cc7d6265d89ec917","GitVersion":"v1.1.0","BuildDate":"2018-01-08T21:56:21Z"}

and I cannot see the elasticsearch flags.

@black-adder
Copy link
Contributor

It looks like you're using latest instead of 1.1. We recently moved around some of the flags so that we can support plugins better #625. Using latest, you have to instead use env variable SPAN_STORAGE=elasticsearch to use the elasticsearch flags. I'd recommend that you use 1.1 since this change will be apart of 1.2 and will be documented at that time.

@nbettiol
Copy link

nbettiol commented Jan 9, 2018

Thanks for the reply, yes I was using the latest version. I will use the 1.1

@fzakaria
Copy link

I would love to see a SQL option (whatever ANSI SQL that will be least vendor lock-in).
Setting up Cassandra / ElasticSearch might be too ambitious for projects that want distributed tracing but honestly don't have the TPS to warrant a distributed datastore.

@ringerc
Copy link

ringerc commented Feb 3, 2018

Since I work with PostgreSQL, I sure wouldn't complain. But honestly I'm not sure a SQL db is an optimal store for largely free-form metrics of this nature. PostgreSQL at least offers the jsonb type for indexable free-form data. If you're trying to do this in a vendor neutral way you'll land up with your own json blobs, or doing EAV, and both of those are terrible. ANSI SQL is a poor fit for variable-structured or key/value form data and you'll need some vendor extensions to get usable performance.

But you inevitably land up with someone putting an ORM on top to "abstract" the DB. Then the ORM performs terribly, gobbles memory and everyone says "the SQL backend is slow, use instead".

@pavolloffay
Copy link
Member

Related issue to this one is #551. Upvote if you are interested in it.

@SwarnimRaj
Copy link

New related issue-
Files - #894

@wy100101
Copy link

wy100101 commented Aug 1, 2018

We are looking at using BigQuery as a storage layer. Presumably this could work with a SQL storage option. SQL can be a generic way to deal with columnar data stores in a generic way. I would complain about a BigQuery specific solution, but I think there is a place for generic SQL interface beyond RDBs.

@yurishkuro
Copy link
Member Author

I assume that even if some database can be treated as SQL and accessed via standard database/sql API, we still need to statically import the actual driver. Granted, this may be less maintenance than a dedicated SpanStorage implementation. However, now that the protobuf model has been merged, nothing is blocking us from moving on the storage plugin dev, eg using something like harshicorp grpc plugin framework.

@isaachier
Copy link
Contributor

isaachier commented Aug 1, 2018

Our model is sufficiently simple to warrant looking into using an ORM to support a large number of backends. I'll take a look at what's available. Reread above and understand what @yurishkuro means.

@bruth
Copy link

bruth commented Aug 6, 2018

Giving my two cents.. an ANSI SQL could work for small workloads, so may be useful for lower-throughput applications that still want to benefit from this tool.

I will also throw out there that Timescale (a Postgres extension) may be a good fit for the required high write throughput.

@mcarbonneaux
Copy link

mcarbonneaux commented May 27, 2019

Clickhouse are SQL high performance storage very efficient for log and trace storage and whold be perfect storage alternative to cassandra original one... they are a true column db... distributed...compressed...

they are near to the CQL (sql like query language)... they use an SQL like language to...

https://clickhouse.yandex/

@chvck
Copy link
Contributor

chvck commented May 31, 2019

I just thought that I'd drop something here to say that there is also support for using Couchbase as a storage backend (via the grpc plugin), currently at https://github.com/chvck/couchbase-jaeger-storage-plugin. Will likely move to the couchbase-labs organisation in time.

@omerlh
Copy link

omerlh commented Jul 18, 2019

Has someone started to work on Azure CosmosDB integration? It has support for Cassandra API, but I couldn't manage to make it work...

@rleiwang
Copy link

rleiwang commented Oct 2, 2020

I just created an issue proposing Chronowave as storage backend. #2534

@DjinNO
Copy link

DjinNO commented Oct 15, 2020

What about ClickHouse? Clickhouse is very cool

@jkowall
Copy link
Contributor

jkowall commented Apr 20, 2021

Cool! Nice job @muhammadn I'm curious how the performance is with a search using S3 in that manner. Are you planning on deploying or using this setup?

@yurishkuro
Copy link
Member Author

@muhammadn I re-opened #2633 and linked your repo at the top. Suggest moving discussions/updates there.

@em135
Copy link

em135 commented May 17, 2021

@Xitric and I are working in on a storage backend for Humio using the grpc plugin. The repository can currently be found at https://github.com/em135/humio-jaeger-plugin. I have opened an issue for this: #3005

@galan
Copy link

galan commented Jun 1, 2021

If S3 is supported, I would suggest to support GCS as well, which is also an objects-storage for all Google Cloud users. For our use-case that would be tremendous helpful!

@muhammadn
Copy link

muhammadn commented Jun 4, 2021

@galan it does actually, despite the name is jaeger-s3, i had already added support for GCS (and Azure Storage) for quite some time but i have not tested it.

Maybe you can go to https://github.com/muhammadn/jaeger-s3 to try it out.

Related code to GCS:
https://github.com/muhammadn/jaeger-s3/blob/main/config/config.go#L26
https://github.com/muhammadn/jaeger-s3/blob/main/s3store/store.go#L50

All you need is to modify the configuration for jaeger-s3 to this:

https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/#example-configuration

which has the GCS config.

I will update the documentation to jaeger-s3 include GCS and Azure as well. (and probably change the project name entirely)

Do tell me if you need help. But i think we can move this discussion to #2633

@galan Update: I have updated the documentation - https://github.com/muhammadn/jaeger-s3/blob/main/README.md

Also just a question from the community, should i rename this project as a more generic name rather than jaeger-s3 since this plugins will support GCS and Azure as well?

@qiansheng91
Copy link

@jpkrohling The Alibaba cloud log service has supported the jaeger, and here are the gifs of the plugin. Link: https://github.com/qiansheng91/jaeger-sls#quick-start

@acceptMyPR
Copy link

Can i use other jaeger collector as backend for my jaeger collector?

@pavolloffay
Copy link
Member

Jaeger collector cannot send data (e.g. over gRPC) to other jaeger collector. However this capability is supported with OTEL collector.

@nitinsaprumaersk
Copy link

nitinsaprumaersk commented Mar 17, 2022

@yurishkuro Would highly recommend to add Azure Table Storage as a backend storage option as well.

@arajkumar
Copy link
Contributor

@yurishkuro Could you please add PostgreSQL with Promscale into the list.? Now Promscale is Jaeger storage complaint too :) Thanks.

@nicolastakashi
Copy link

@yurishkuro could you please add RediSearch to the list? I've worked on a GRPC Plugin to Store Traces on Redis Search and this is close to the first release, I'm just adding a few performance tests.

https://github.com/nicolastakashi/jaeger-redisearch

@coverthesea
Copy link

Zinc https://github.com/zinclabs/zinc

@vemula-anu
Copy link

i use jaeger with timescaledb in kubernetes
{"level":"info","ts":1677823818.9793034,"caller":"querysvc/query_service.go:137","msg":"Archive storage not created","reason":"archive storage not supported"}
i use this configurations
spec:
containers:
- env:
- name: GRPC_STORAGE_SERVER
value: promscale:9202
- name: SPAN_STORAGE_TYPE
value: grpc-plugin
image: jaegertracing/jaeger-query:1.30
name: jaeger
ports:
- containerPort: 16685
- containerPort: 16686
restartPolicy: Always
the problem getting can anyone give solution for this

@yurishkuro
Copy link
Member Author

@vemula-anu please do not post support questions to this issue, create a new question in Discussions.

@diondew
Copy link

diondew commented Apr 19, 2023

Could you please add Yugabyte to the list? #4354

@jkowall
Copy link
Contributor

jkowall commented Jun 27, 2023

Could you please add Yugabyte to the list? #4354

Added, sorry for the delay.

@paulgrav
Copy link

The creator of Tempo @joe-elliott is a maintainer here at Jaeger and is also an engineer with Grafana Labs. We had this discussion, but right now Tempo cannot support the types of queries the Jaeger UI does today. I know that may change, at which point Tempo would be a good Jaeger backend, but today it cannot do everything necessary.

I know Tempo today is a lot more capable in terms of its search compared to back in 2021. Is it now a potentially good Jaeger backend or are there still gaps?

@joe-elliott
Copy link
Member

joe-elliott commented Nov 15, 2023

Tempo has a broader set of search capabilities (via TraceQL) than Jaeger search. So, generally, Tempo could be used to back Jaeger. There are two gaps I'm aware of.

First, Jaeger search currently returns the entire trace to the frontend which then renders the search results pane you see in the UI. Tempo, however, only returns metadata to the frontend. I don't believe this metadata is enough to render every element of the Jaeger search results. For instance, I don't think you could list the services in the trace.

Second, we currently only retrieve auto complete tags from recent traces. So if you were searching a time range from yesterday the auto complete would still be based on the traces received in the last half hour or so. We are working to address this.

@yurishkuro yurishkuro added the meta-issue An tracking issue that requires work in other repos label Feb 3, 2024
@muhammadn
Copy link

muhammadn commented Feb 25, 2024

@paulgrav

Just what @joe-elliott explained but i want to add that it had been already done. I had completely overhauled jaeger-objectstorage to use tempo as a backend to store traces to AWS S3/GCS/AzureBlob since i believe tempo is already mature to support multiple cloud storage providers. Back in the early days of tempo there wasn't support for GCS and AzureBlob so we used loki as the interface to store trace data.

I've posted in the forums on how it would look like so you can take a look.

The codes are already published (both the tempo fork and jaeger-objectstorage) but the documentation needs more polishing.

@jiajiayang
Copy link

Is it possible to use loki as a back-end store, which can be a good correlation between logging and tracing?

@jkowall
Copy link
Contributor

jkowall commented Jun 8, 2024

Is it possible to use loki as a back-end store, which can be a good correlation between logging and tracing?

Unfortunately, no @jiajiayang as loki is a logging system, however Tempo is the tracing backend and doesn't support full text search which is how Jaeger does the querying you see in the search dialog box.

Even in Grafana stack you still run multiple backends.

@joker-star-l
Copy link

joker-star-l commented Sep 4, 2024

Hello community, I am going to implement a apache doris grpc storage.
I have done some work in open telemetry community. The open telemetry collector will be able to write traces data directly to apache doris.
So can I only implement the SpanReaderPlugin interface in the grpc service, or I still need to implement the SpanWriterPlugin interface?

Could you give me some advice? Thank you!

@yurishkuro
Copy link
Member Author

@joker-star-l it's up to you. You can maintain it in your own repository and explain the existing limitations. If people find it useful they may ask to support the writer API.

@rajeshksv
Copy link

Any plans of supporting Pinot in future ?

@yurishkuro
Copy link
Member Author

No

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/storage enhancement meta-issue An tracking issue that requires work in other repos
Projects
None yet
Development

No branches or pull requests