Field mapping

Introduction

Field mapping standardizes data from integrated third-party sources for display on the [REDACTED] investigation dashboard. This formatting consistency allows reliable data correlation and automation.

Mappings are configured for the following fetch commands:

fetchEvent: Retrieves data from external sources and ingests them as events into [REDACTED].
fetchIncident: Ingests data as incidents into [REDACTED].

Note

For fetchEvent commands, you need to configure Event Field Mapping. On the other hand, both Event and Incident Field Mappings need to be configured with fetchIncident commands.

What’s the difference between events and incidents?

Some systems only provide a single layer of raw data, while others offer two. For instance, Microsoft Sentinel differentiates between alerts and incidents. [REDACTED] also employs a two-level approach to organize ingested data:

Events: Raw data from systems like SIEMs, email protection, EDRs, and many more. Events are undetermined data, meaning they haven't yet been checked for potential security risks.
Incidents: Consists of one or more events that are flagged as potentially malicious and need investigating.

Event field mapping

For built-in integrations, [REDACTED] provides a set of default event field mappings. However, [REDACTED] provides the flexibility for you to modify and add mappings as you see fit, which opens up endless configurability. For instance, you may want to map the flagStatus field for the Office 365 connector, which isn’t mapped by default in [REDACTED]. Event field mapping can be configured by navigating to Configuration > Integrations [Your Desired Integration] > Event Intake Setup Event Field Mapping.

Event source

Some third-party platforms offer multiple types of retrievable data. While most built-in integrations come with a single default event source to cater to the majority of data ingestion scenarios, there are exceptions. For instance, integrations such as Microsoft Sentinel provide various data types for ingestion, including incidents, events, and alerts, requiring three additional corresponding event sources. You can also create new event sources for various edge cases.

Main Event JSON Path: This is the parent JSON array in the ingested data’s structure for the event source. This is configured by specifying the JSON path in dot notation.

For example, take a look at the following condensed JSON structure representing event data from Office 365:

json
{
   "@odata.context":"https://graph.microsoft.com/v1.0/$metadata#users(&#39;sysint%40devcyber.onmicrosoft.com&#39;)/mailFolders(&#39;Inbox&#39;)/messages(attachments())",
   "value":[
      {
         "@odata.etag":"W/\"CQAAABYAAAA5S8SNezXtS61ZDvwztvPVAAAkd5by\"",
         "id":"AAMkAGI5ODVlN2I2LWVlODktNDhkYS1iZjI4LTU1NTQ1YTliMmZmYQBGAAAAAADOVKOjlhkSQK6WF_M4xWz-BwA5S8SNezXtS61ZDvwztvPVAAAAAAEMAAA5S8SNezXtS61ZDvwztvPVAAAkgSisAAA=",
         "createdDateTime":"2021-05-13T21:48:49Z",
         "lastModifiedDateTime":"2021-05-13T21:48:51Z",
         "changeKey":"CQAAABYAAAA5S8SNezXtS61ZDvwztvPVAAAkd5by",
         "categories":[],
         "receivedDateTime":"2021-05-13T21:48:50Z",
         "sentDateTime":"2021-05-13T21:46:50Z",
         "hasAttachments":true,
         "internetMessageId":"&lt;CAO78ce_dnvP+PYXnYyczkMsFwczG6T4vP71YQUNi1hjgi-3ntA@mail.gmail.com&gt;",
         "subject":"Phishing1",
         "bodyPreview":"",
         "importance":"normal",
         "parentFolderId":"AAMkAGI5ODVlN2I2LWVlODktNDhkYS1iZjI4LTU1NTQ1YTliMmZmYQAuAAAAAADOVKOjlhkSQK6WF_M4xWz-AQA5S8SNezXtS61ZDvwztvPVAAAAAAEMAAA=",
         "conversationId":"AAQkAGI5ODVlN2I2LWVlODktNDhkYS1iZjI4LTU1NTQ1YTliMmZmYQAQAPuXocauAD5AukCNPirqico=",
         "conversationIndex":"AQHXSEHC+5ehxq4APkC6QI0+KuqJyg==",
         "isDeliveryReceiptRequested":null,
         "isReadReceiptRequested":false,
         "isRead":false,
         "isDraft":false,
         "webLink":"https://outlook.office365.com/owa/?ItemID=AAMkAGI5ODVlN2I2LWVlODktNDhkYS1iZjI4LTU1NTQ1YTliMmZmYQBGAAAAAADOVKOjlhkSQK6WF%2BM4xWz%2FBwA5S8SNezXtS61ZDvwztvPVAAAAAAEMAAA5S8SNezXtS61ZDvwztvPVAAAkgSisAAA%3D&amp;exvsurl=1&amp;viewmodel=ReadMessageItem",
         "inferenceClassification":"focused",
         "body":{},
         "sender":{},
         "from":{},
         "toRecipients":[],
         "ccRecipients":[],
         "bccRecipients":[],
         "replyTo":[],
         "flag":{},
         "attachments":[],
         "originalEmail":[],
         "originalMessageFile":[]
      }
   ]
}

The most relevant event data from Office 365 reside under the key $.value. However, there's also the $[.@odata.context](mailto:.@odata.context) key, which provides the payload's context URL, though this is generally not relevant for incident responses. By setting the Main Event JSON Path, the system knows to associate fields starting with $.value. This means that instead of writing out the full path as $.value.[field] when defining additional field mappings for the event source, you can shorthand it to .[field].

Search String: This feature enables an event source to sift through ingested data, focusing solely on specific values or based on regular expressions. For instance, if you wish to only capture events from Office 365 that have a "high" importance level, found at the path $.value.importance in the JSON object, you would set the search string to $.value.importance=high when establishing a new event source. Regex expressions can be used to tell [REDACTED] to only extract a certain type of data from the data source. For example, in the Splunk v2 integration, there is a built-in event source for WinEventLog, or Windows Event Log. The search string with the Regex condition for this event source is {sourcetype}=WinEventLog:*.

Note
Search strings can be specified using one of the following formats: {jsonpath}=value, {jsonpath}={{regex}}, or jsonpath=value.
Order Priority: In situations where several event sources are set up for a fetch command, the system gives precedence to the event source with the lowest order priority number, especially in cases of overlapping field mappings.

Mapped fields

Event sources have fields that map the relationship between ingested data and an event field.

Source Field: The JSONPath key that specifies the exact data field to map from the ingested JSON payload. If you have already configured a Main Event JSON Path, you do not need to repeat the full path. Instead, use shorthand syntax. For example, if your desired field is $.value.importance and the Main Event JSON Path is $.value, enter .importance as the Source Field.

Source Type: The data type of the mapped field, which can be Text, DatetimeRegex, and Placeholder. This lets [REDACTED] know how the data will be formatted within the application.

Source Type	Description
Text	Formatted as a plain text field.
Datetime	Formatted as a datetime field. The default datetime format is `yyyy-MM-dd HH:mm:ss`. Other formats are also available for mapping.
Regex	Applies a regular expression to filter the field value (e.g. `userdata:s*(.?)` ).
Placeholder	Concatenates the values from multiple mapped fields as a single plain text field (e.g. `{EventID} started on {FirstTime}`).

Unique key field mapping

The Unique Key event field mapping tells [REDACTED] how to distinguish each event upon ingestion and prevent duplicated events. This is done by pinpointing a UUID or primary key found within the event data and mapping it to [REDACTED]’s event field. Events are ingested differently based on whether a field mapping has been configured. Here's how it works:

If the unique key field is mapped:

Only events with a unique key value are ingested. A new event containing an existing Unique Key with a different set of raw event data will not be ingested.

If the unique key field is not mapped:

Only events containing unique raw data are ingested. Events with identical raw event data as any existing event in the system will not be ingested.

Incident field mapping

Incident field mapping is exclusively available for fetchIncident commands. In most cases, default incident field mappings are already available. If you wish to customize them, you can do so by navigating to Configuration > Integrations [Your Desired Integration] > Incident Intake Setup Incident Field Mapping.

Please note that event field mapping must also be configured for fetchIncident commands. Incident field mapping allows you to map the following set of incident-specific system fields with the ingested data of a command:

Incident Fields	Description
Title	The title of the incident.
Description	A description text about the incident.
Severity	The triage severity level of the incident.
Incident Type	The default incident type to apply to the incident after ingestion. Note: Incident types can be configured within the incident form editor.
Incident Creator	The default creator of the incident after ingestion.
Incident Owner	The default owner of the incident after ingestion. Owners have the permission to modify incident details, run the playbook and assign investigators.
Incident Playbook	The default incident playbook which will be run after ingestion.
Due in Date	The deadline for the incident to be concluded and closed. This is the current date plus x number of days.
Unique Key	The unique key to identify the incident upon ingestion. Refer to the Unique Key Field Mapping section below for more details.
Tactics	The JSON path in the incident data to extract MITRE tactics. Any matching values will appear as a new instance in the Monitor module.
Techniques	The JSON path in the incident data to extract MITRE techniques. Any matching values will appear as a new instance in the Monitor module.

Main JSON path

Some third-party systems may offer their incidents in the form of an overarching incident dataset with sub-events embedded inside. In that case, we need to tell [REDACTED] which JSON path where the incident portion of the data starts. If no Main JSON Path has been specified, it will default to the root path.

Consider the following condensed JSON structure representing incident data from Cybereason:

json
{
   "data":{
      "resultIdToElementDataMap":[],
      "suspicionsMap":{},
      "evidenceMap":{},
      "totalPossibleResults":1,
      "guessedPossibleResults":0,
      "queryLimits":{},
      "queryTerminated":false,
      "pathResultCounts":[],
      "guids":[]
   },
   "status":"SUCCESS",
   "hidePartialSuccess":false,
   "message":"",
   "expectedResults":1,
   "failures":0
}

If our primary interest lies in the data under the "resultIdToElementDataMap" key, the Main JSON Path can be set to $.data.resultIdToElementDataMap. The rest of the arrays in this JSON are not relevant for our analysis.

Note

When you set a Main JSON path, it will be universally applied to all incident sources within the fetchIncident command.

Site

Each site can have its own unique incident field mappings, allowing various sites to have distinct incident sources and associated mappings. By default, all sites utilize the “Default Site” mappings.

To customize incident field mappings for a particular site:

Choose the desired site.
View its "Default Incident Source."
If necessary, you can either adjust the "Default Incident Source" or create a new incident source.
Click on the "+ Add Default Mappings" button to apply the standard mappings to your chosen site’s "Default Incident Source".
Once added, you can then modify individual mappings as needed.

Incident source

For the specified site, multiple incident sources can be configured. Unless you want configure a customized search string for the incident source, you do not need to add a new incident source. Incident sources have the following configurable components:

Search String: This feature enables an incident source to sift through ingested data, focusing solely on specific values or based on Regex expressions.

Note
Search strings can be specified using one of the following formats: {jsonpath}=value, {jsonpath}={{regex}}, or jsonpath=value.
Order Priority: In situations where several incident sources are set up for a fetch command, the system gives precedence to the incident source with the lowest order priority number, especially in cases of overlapping field mappings.

Unique key field mapping

The Unique Key incident field mapping tells [REDACTED] SOAR how to distinguish each event upon ingestion and prevent duplicated events. This is done by pinpointing a UUID or primary key found within the incident data and mapping it to [REDACTED]’s incident field. Incidents are ingested differently based on whether a field mapping has been configured. Here's how it works:

If the unique key field is mapped:

Incidents with a unique key value are ingested. Should an incoming incident share the same unique key as an existing one, it will only be considered for ingestion if it presents a different data set. In such cases, the original incident is replaced with the updated data.

If the unique key field is not mapped:

Only incidents containing unique raw data are ingested. Any incident exhibiting raw incident data identical to what's already in the system will be excluded from ingestion.

Introduction​

What’s the difference between events and incidents?​

Event field mapping​

Event source​

Mapped fields​

Unique key field mapping​

Incident field mapping​

Main JSON path​

Site​

Incident source​

Unique key field mapping​

Introduction

What’s the difference between events and incidents?

Event field mapping

Event source

Mapped fields

Unique key field mapping

Incident field mapping

Main JSON path

Site

Incident source

Unique key field mapping