Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@nimarezainia Should this issue be moved to the https://github.com/elastic/integrations/ repo as that's where the custom log integration is defined?

@nimarezainia Should this issue be moved to the https://github.com/elastic/integrations/ repo as that's where the custom log integration is defined?

No I believe that this is owned by the data plane team. (fyi @flexitrev )

Linking these here for record keeping:

elastic/integrations#4609
https://github.com/elastic/ingest-dev/issues/1384

Could we migrate the Custom Logs integration to use Filestream instead? We have the take-over mode in Filestream that is capable of getting the state from a file being harvested by the Log input and use it in the Filestream input, the only thing missing is an automated way to convert the configuration (specially the custom configuration field). It might also need some extra work because the way the Elastic-Agent divides the integrations per input type when starting component.s

Just adding a new integration might make it harder to migrate the state and confusing for users finding both when searching.

Could we migrate the Custom Logs integration to use Filestream instead?

We could, but it is a lot more work, in addition to what you mentioned it is blocked by:

• Support migrating persisted state from one input to another #5304

Creating a new integration at least stops new instances of the deprecated input being used with low effort. The migration problem should still be solved but it doesn't need to block creating a filestream based integration given how long solving migration correctly is likely to take.

@belimawr would you be merging this PR again as part of this issue?

@belimawr would you be merging this PR again as part of this issue?

Do you mean reverting it? That's an option and it's an interesting starting point. My first idea was to just copy the Custom Logs integration, edit it to use filestream and update the docs, but I can start from reverting this PR as well.

elastic/integrations#4609 was using a package spec version of 1.0.0 so there might be some changes required.

Folks, I create a draft PR, there are I'd like your opinion on (@nimarezainia, @jlind23)
1. Should we expose file identities other than fingerprint?
File identity docs: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#_file_identity_2

Currently I'm using fingerprint as the only option and allowing to configure the offset and length. I believe that is the best option because this is the file identity we implemented to solve the issues we were having with inode reuse and the one we recommend to our users.

The reason I did not add any of the other ones is because while I can define a variable/input field as a selection from a list, the template language used is somehow limited and there is no way to use this selection values to render the correct policy. Having all options a boolean/toggle is very confusing because only one can be used at a time. I asked on slack for some suggestions/help, if there is a way

2. Which name do we want for the integration?
Currently it's "Custom Filestream Logs" (from "reverting" and updating the PR removing it), however having "Custom logs" and "Custom Filestream Logs" can be confusing... Even the description is not very helpful in answering which one should be used.

I was thinking about something like "Custom Logs V2","Custom Logs enhanced", something that makes it clear this one should be used instead of the "Custom Logs".

3. Is there any preference for the initial version?
Currently I put 0.0.1 because it is the very first release, the old one had 1.0.0 as the initial release. I'm not 100% sure what is the standard we use for integrations.

4. When do we need to release the first version?
I'm asking because if question 1., one way to solve is is to get a small change in Kibana to add template helpers that will allow me to implement the logic I need. my JS foo is rusty, but I could even give it a shot, but I don't know how long it would take to get the change in and it will make the integration require the newest version of Kibana, which might not be ideal.

Currently I'm using fingerprint as the only option and allowing to configure the offset and length. I believe that is the best option because this is the file identity we implemented to solve the issues we were having with inode reuse and the one we recommend to our users.

If people want to use the take over mode to migrate from custom logs one day, do we want to give them the option to do that with the same file identity? I think this might actually be a requirement of the take over mode (once it actually works for Elastic Agent).

Currently it's "Custom Filestream Logs" (from "reverting" and updating the elastic/integrations#4609 removing it), however having "Custom logs" and "Custom Filestream Logs" can be confusing... Even the description is not very helpful in answering which one should be used.

IIRC we spoke with @nimarezainia and @strawgate about renaming "Custom logs" to "Custom logs (Legacy)" so naming this one "Custom Filestream Logs" would make sense.

Currently it's "Custom Filestream Logs" (from "reverting" and updating the elastic/integrations#4609 removing it), however having "Custom logs" and "Custom Filestream Logs" can be confusing... Even the description is not very helpful in answering which one should be used.

IIRC we spoke with @nimarezainia and @strawgate about renaming "Custom logs" to "Custom logs (Legacy)" so naming this one "Custom Filestream Logs" would make sense.

Would this rename happen at the same time/version where we release the Custom Filestream logs?

Currently I'm using fingerprint as the only option and allowing to configure the offset and length. I believe that is the best option because this is the file identity we implemented to solve the issues we were having with inode reuse and the one we recommend to our users.

If people want to use the take over mode to migrate from custom logs one day, do we want to give them the option to do that with the same file identity? I think this might actually be a requirement of the take over mode (once it actually works for Elastic Agent).

That makes sens. Then we need to find the best way to help users not to create an invalid configuration for their Filestream integration. I'll look into this.

Would this rename happen at the same time/version where we release the Custom Filestream logs?

Once the Custom Filestream logs integration is published we would have to create a new PR that will update the Custom Log title, yes.

Would this rename happen at the same time/version where we release the Custom Filestream logs?

Once the Custom Filestream logs integration is published we would have to create a new PR that will update the Custom Log title, yes.

@belimawr
Yes this can happen at some other point. the idea is that at some point the whole "Custom Log" integration will be deprecated (and that would show in the title).

Regarding configurations available: all the options available to configure filestream should be configurable. e need to accommodate a user migrating from Custom Logs integration to Custom Filestream Logs.