azure: add related.entity field to activitylogs default ingest pipeline #11233
Conversation
orouz
force-pushed
the
azure_activitylogs_cdr_pipeline
branch
from
47f2eea
to
21b7189
Compare
|
/test |
orouz
force-pushed
the
azure_activitylogs_cdr_pipeline
branch
from
dd66af9
to
f34cd4a
Compare
orouz
force-pushed
the
azure_activitylogs_cdr_pipeline
branch
from
f34cd4a
to
da652ec
Compare
🚀 Benchmarks reportTo see the full report comment with |
andrewkroh
added
the
Team:Obs-InfraObs
packages/azure/changelog.yml
Outdated
| @@ -1,3 +1,8 @@ | |||
| - version: "1.17.1-preview01" | |||
There was a problem hiding this comment.
What's the plan for when this change is safe to be included in a non pre-release version? Is this a pre-release because of elastic/ecs#2360 not being finalized?
Based on semver.org rules, as an enhancement, I would expect the versioning to be based on 1.18 and not a patch version.
There was a problem hiding this comment.
I've commented on this kind of thing before, but now having experienced the impact of it first hand, I'll repeat that the use of preview commits with an undefined TTL based on external constraints and that cannot become part of a new release without potentially breaking users as a bad idea from an ongoing maintenance perspective.
There was a problem hiding this comment.
initially the plan was to release a preview version so we can test it before it hits users, but i see now how it affects other development teams so i've updated it to 1.18.0 as suggested and did the testing using elastic package install
| on_failure: | ||
| - set: | ||
| description: Add error reason | ||
| field: error.message | ||
| value: "{{{ _ingest.on_failure_message }}}" |
There was a problem hiding this comment.
This looks incomplete. Also conventionally, on_failure handlers are placed last.
| - script: | ||
| description: Appends principal_id and resource_id to `related.entity` | ||
| lang: painless | ||
| ignore_failure: true | ||
| on_failure: | ||
| - set: | ||
| description: Add error reason | ||
| field: error.message | ||
| value: "{{{ _ingest.on_failure_message }}}" | ||
| source: | | ||
| ctx.related = ctx.related ?: [:]; | ||
| ctx.related.entity = new HashSet(); | ||
| ctx.related.entity.add(field("azure.activitylogs.identity.authorization.evidence.principal_id").get(null)); | ||
| ctx.related.entity.add(field("azure.resource_id").get(null)); | ||
| ctx.related.entity.remove(""); | ||
| ctx.related.entity.remove(null); | ||
|
|
There was a problem hiding this comment.
Suggest rather the simpler
| - script: | |
| description: Appends principal_id and resource_id to `related.entity` | |
| lang: painless | |
| ignore_failure: true | |
| on_failure: | |
| - set: | |
| description: Add error reason | |
| field: error.message | |
| value: "{{{ _ingest.on_failure_message }}}" | |
| source: | | |
| ctx.related = ctx.related ?: [:]; | |
| ctx.related.entity = new HashSet(); | |
| ctx.related.entity.add(field("azure.activitylogs.identity.authorization.evidence.principal_id").get(null)); | |
| ctx.related.entity.add(field("azure.resource_id").get(null)); | |
| ctx.related.entity.remove(""); | |
| ctx.related.entity.remove(null); | |
| - append: | |
| field: related.entity | |
| value: '{{{azure.resource_id}}}' | |
| allow_duplicates: false | |
| if: > | |
| ctx.azure?.resource_id != null && ctx.azure.resource_id != '' | |
| - append: | |
| field: related.entity | |
| value: '{{{azure.activitylogs.identity.authorization.evidence.principal_id}}}' | |
| allow_duplicates: false | |
| if: > | |
| ctx.azure?.activitylogs?.identity?.authorization?.evidence?.principal_id != null && | |
| ctx.azure.activitylogs.identity.authorization.evidence.principal_id != '' |
which is shorter, does not require an error handler, and does not allocate in the case that the fields are absent.
There was a problem hiding this comment.
i did this, but currently if one of these append processors fails, the pipeline won't continue, which is something i'd like to avoid. i think adding an empty on_failure field to the processors should cover that, but i didn't manage to test this with invalid values. wdyt? how could i ensure a failure won't stop the pipeline from executing the next steps, but still log the error? (ignore_failure is also not preferable)
There was a problem hiding this comment.
What is the failure mode that could lead to an append failing?
There was a problem hiding this comment.
tbh i don't know. i don't see how it can fail now, but as a general good measure i assumed it'd be better to account for it in anyway. but if you're ok with it then i guess it's fine.
| contains multiple entities, identifiers belonging to different entities | ||
| will be present. Example identifiers include cloud resource IDs, ARNs, | ||
| email addresses, or hostnames. | ||
| type: keyword |
|
💚 Build Succeeded
History
|
efd6
changed the title
Proposed commit message
this PR is part of the cloud security CDR epic. it adds a painless script processor that appends both
principal_id(event/action origin) andresource_id(event/action target) to a new field -related.entitythe
related.entityfield is an upcoming ECS field meant to facilitate pivoting around a piece of data.Related issues
benchmark (30 runs)