Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[corelight] initial release of Corelight #11288

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

sharadcrest
Copy link

Proposed commit message

Create New integration package corelight and add below four dashboards:

  • Name Resolution Insights
  • Remote Activity Insights
  • Secure Channel Insights
  • Security Posture

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/corelight directory.
  • Run the following command to run tests.

elastic-package test

--- Test results for package: corelight - START ---
╭───────────┬─────────────┬───────────┬────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                                                          │ RESULT │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ corelight │             │ asset     │ dashboard corelight-45197477-c13f-4e52-a5dd-fb4f53564963 is loaded │ PASS   │      1.146µs │
│ corelight │             │ asset     │ dashboard corelight-7c0946bc-acd0-4ec3-ab3b-8a92853f4a3b is loaded │ PASS   │        181ns │
│ corelight │             │ asset     │ dashboard corelight-8546a96c-86c9-4edf-9d46-88338d6ac40e is loaded │ PASS   │        185ns │
│ corelight │             │ asset     │ dashboard corelight-f4864774-ed73-4b78-b861-5b8235ec12cf is loaded │ PASS   │        164ns │
╰───────────┴─────────────┴───────────┴────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: corelight - END   ---
Done

Screenshots

image
image

Copy link

cla-checker-service bot commented Oct 1, 2024

💚 CLA has been signed

@sharadcrest sharadcrest changed the title {corelight] initial release of Corelight [corelight] initial release of Corelight Oct 1, 2024
@andrewkroh andrewkroh added needs CLA User must sign the Elastic Contributor License before review. New Integration labels Oct 1, 2024

## Prerequisites:

**Add ECS Mappings**: Start by adding the ECS (Elastic Common Schema) mappings from the [Corelight GitHub repository](https://github.com/corelight). You can find the required templates here: [Corelight ECS Templates](https://github.com/corelight/ecs-templates). These mappings will ensure that Corelight data is correctly formatted and aligned with Elastic's schema.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are users been asked to manually download an input ECS mappings from Github. The ingest pipeline can be included with the integration to avoid this additional step. Any reason why we can't include the ingest pipeline as part of the integration (inline with all other integrations?)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corelight have mapped custom mapping differently in their GitHub repo, also they have Elastic as a configuration page where they have build pipeline along with ECS mapping for the fields, so the only ask from them is to build dashboards inline to dashboards for other vendors(Splunk, Looker etc..)

@kcreddy
Copy link
Contributor

kcreddy commented Oct 8, 2024

/test

@andrewkroh andrewkroh removed the needs CLA User must sign the Elastic Contributor License before review. label Oct 8, 2024
@elasticmachine
Copy link

💚 Build Succeeded

@sharadcrest
Copy link
Author

@jamiehynds Are we okay to start review for this PR, please?

@jamiehynds
Copy link

@kcreddy are you ok to review this PR for Corelight (developed by Crest). Would be great to understand if the work you had done previously with Corelight, could be leveraged here? The current workflow goes against the experience of most integrations, as you're required have to manually download the mappings from Github.

@kcreddy kcreddy added Crest Integration:corelight [Integration not found in source] Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Oct 16, 2024
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The integration is unconventional and requests users to install assets such as mappings, settings, templates, ingest pipelines, etc. using an installation script provided by Corelight.

Although the Corelight repositories (providing the script and assets) are well-maintained, it should be noted and documented that this integration only provides dashboards and nothing more. The users are responsible for the updates and have to frequently check and update assets from the script provided by the Corelight repository. This should also be documented in the README.

Also it should be documented that any issues with the installation during initial setup or updates must be followed up with Corelight as we don't own them. This applies to any non-dashboard issues.

In the future, if we were to add some or all of these assets to our integration, users will need to manually delete existing stale Corelight assets.

cc: @jamiehynds


[Corelight](https://corelight.com/) provides network detection and response (NDR) solutions that enhance visibility, threat detection, and incident response by leveraging open-source technologies like Zeek. Its platform integrates with existing security tools to deliver high-fidelity network data, helping organizations detect and respond to threats more effectively across both on-premises and cloud environments​.

This integration includes only the Corelight dashboards for Security Posture, Remote Activity Insights, Name Resolution Insights, and Secure Channel Insights.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a bulleted list?


**Add ECS Mappings**: Start by adding the ECS (Elastic Common Schema) mappings from the [Corelight GitHub repository](https://github.com/corelight). You can find the required templates here: [Corelight ECS Templates](https://github.com/corelight/ecs-templates). These mappings will ensure that Corelight data is correctly formatted and aligned with Elastic's schema.

**Send Data from Corelight to Elastic**: Once the ECS mappings are in place, configure Corelight to send data directly to your Elastic environment.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be explained step-by-step.


**Send Data from Corelight to Elastic**: Once the ECS mappings are in place, configure Corelight to send data directly to your Elastic environment.

**Note**: Use the default index (logs-*) name instead of a custom index.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get this. Where do users set this? This can be explained better.

Comment on lines +17 to +24
### Enabling the integration in Elastic:

1. In Kibana go to Management > Integrations.
2. In "Search for integrations" search bar, type Corelight.
3. Click on the "Corelight" integration from the search results.
4. Go to Settings.
5. Click on the "Install Corelight assets".
6. Go to Assets to get list of dashboards.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To conform the docs across integration, we should make this section wording exactly same as https://github.com/elastic/integrations/blob/main/packages/abnormal_security/_dev/build/docs/README.md#enabling-the-integration-in-elastic

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was there any requirement to make the title Remote Access Hygiene while the left navigation calls it Remote Activity Insights? I think they should be alike to avoid confusion.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for the rest of the dashboards.

description: Collect logs from Corelight with Elastic Agent.
type: integration
categories:
- security
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add following as well:

- dns_security
- network
- network_security
- vpn_security

@kcreddy
Copy link
Contributor

kcreddy commented Oct 18, 2024

@piyush-elastic, related to #11288 (comment), can you update Crest template for README with changes from #11210? This is to conform the README docs across integrations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Crest Integration:corelight [Integration not found in source] New Integration Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants