Quick Links

Overview

This is the documentation of ECS version 1.11.0-7.

What is ECS?

The Elastic Common Schema (ECS) is an open source specification, developed with support from the Elastic user community. ECS defines a common set of fields to be used when storing event data in Elasticsearch, such as logs and metrics.

ECS specifies field names and Elasticsearch datatypes for each field, and provides descriptions and example usage. ECS also groups fields into ECS levels, which are used to signal how much a field is expected to be present. You can learn more about ECS levels in [ecs-guidelines]. Finally, ECS also provides a set of naming guidelines for adding custom fields.

The goal of ECS is to enable and encourage users of Elasticsearch to normalize their event data, so that they can better analyze, visualize, and correlate the data represented in their events. ECS has been scoped to accommodate a wide variety of events, spanning:

  • Event sources: whether the source of your event is an Elastic product, a third- party product, or a custom application built by your organization.

  • Ingestion architectures: whether the ingestion path for your events includes Beats processors, Logstash, Elasticsearch ingest node, all of the above, or none of the above.

  • Consumers: whether consumed by API, Kibana queries, dashboards, apps, or other means.

New to ECS?

If you’re new to ECS and looking for an introduction to its benefits and examples of the core concepts, [ecs-getting-started] is a great place to start.

My events don’t map with ECS

ECS is a permissive schema. If your events have additional data that cannot be mapped to ECS, you can simply add them to your events, using custom field names.

Maturity

ECS improvements are released following Semantic Versioning. Major ECS releases are planned to be aligned with major Elastic Stack releases.

Any feedback on the general structure, missing fields, or existing fields is appreciated. For contributions please read the Contribution Guidelines.

Using ECS

ECS fields follow a series of guidelines, to ensure a consistent and predictable feel, across various use cases.

If you’re new to ECS and would like an introduction on implementing and using the schema, check out the [ecs-getting-started] guide.

Whether you’re trying to recall a field name, implementing a solution that follows ECS, or proposing a change to the schema, the [ecs-guidelines] and [ecs-conventions] will help get you there.

If you’re wondering how to best capture event details that don’t map to existing ECS fields, head over to [ecs-custom-fields-in-ecs].

Mapping network events provides a detailed walk-through of how to best map and categorize an example network event to the schema.

Getting Started

ECS enables and encourages users to normalize event data in order to better analyze, visualize, and correlate their events. Collected events can be normalized at ingest time, consistently searched across indices, and visualized predictably.

Note that when adopting an Elastic solution, such as Observability or Security, all events will map to ECS out of the box. Elastic provides an extensive set of integrations to simplify ingesting your data sources.

If you rely on custom data pipelines and/or building content around specific needs, ECS can still help to alleviate the challenges of searching, analyzing, and visualizing across your data. Let’s see how using a common schema can simplify the search experience, and then take a look at how an event’s contents can be mapped to ECS field sets.

Simplified Search

With ECS defining a normalized schema across all of your data sources, querying against those sources is simplified. Consider searching for a particular source IP address prior to adopting ECS. All the various data sources and their field mappings would need to be considered in your query:

src:10.42.42.42 OR client_ip:10.42.42.42 OR apache.access.remote_ip:10.42.42.42 OR
context.user.ip:10.42.42.42 OR src_ip:10.42.42.42

With all sources mapped to ECS, the query becomes much simpler:

source.ip:10.42.42.42

Not only does this simplify writing queries, but saved queries shared with other users become much more obvious. To gain familiarity with ECS fields, you can also take a look at the [ecs-field-reference] section.

Unified Visualizations

With normalized data from different data sources, building insightful visualizations across sources is simple. From a single, centralized dashboard, events from web servers, IDS/IPS devices, and firewalls can be aggregated and visualized, and enhanced with drill-downs, and pivoting for delving into deeper investigations. Centralized monitoring of diverse data sources is straightforward with normalized ECS data.

Simplify visualization using ECS

Translating Data Sources

To align events to ECS, some sort of parsing will usually be necessary to transform the contents of the original event into the relevant ECS fields. Depending on how you’ve designed your Elastic Stack data ingestion pipelines, the amount of work to parse your events will vary.

For example, an Apache web server log event:

10.42.42.42 - - [15/Jul/2020:20:48:32 +0000] "GET /content HTTP/1.1" 200 2571 "-"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/83.0.4103.106 Safari/537.36"

In order to map this event to ECS, the contents of the event is associated with the appropriate ECS fields.

Field Name Value

@timestamp

2020-07-15T20:20:48.000Z

event.original

10.42.42.42 - - [15/Jul/2020:20:48:32 +0000] "GET /content HTTP/1.1" 200 2571 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36

http.request.method

GET

http.response.body.bytes

2571

http.response.status_code

200

http.version

1.1

message

GET /content HTTP/1.1" 200 2571 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36

source.address

10.42.42.42

source.ip

10.42.42.42

url.original

/content

user_agent.original

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36

Beyond extracting values that are present in the original event, we also populate other fields to provide additional context about the event itself.

  • ecs.version: States which version of ECS the ingest pipeline was developed against.

  • event.dataset and event.module: Answers "where is this event from" and are expected to have a hardcoded value per pipeline, per source.

  • event.kind, event.category, event.type, and event.outcome: The [ecs-category-field-values-reference] should also be hardcoded using knowledge of each type of event the source emits. The contents of these fields are limited to the specifically allowed values detailed in the ECS documentation.

Field Name Value

ecs.version

1.5.0

event.module

apache

event.dataset

apache.access

event.kind

event

event.category

[ "network", "web" ]

event.type

[ "access" ]

event.outcome

success

Lastly, existing field values can be interpreted or enriched using a processor, with the results populating additional fields in the final event.

  • The user_agent processor extracts details from the original user agent string, user_agent.original.

  • IP fields like source.ip can provide enrichment using the geoip processor to add information about the location and autonomous system number (ASN) associated with an IP address.

  • The registered domain processor reads a field containing a hostname and writes the registered domain to another field

  • each event with metadata from the machine’s hosting provider (cloud) and/or from the host machine (host).

Here are some examples of additional fields processed by metadata or parser processors.

Field Name Value Processor

host.architecture

x86_64

add_host_metadata

host.hostname

mbp.example.com

add_host_metadata

host.ip

[ "192.168.1.100" ]

add_host_metadata

host.os.family

darwin

add_host_metadata

host.os.kernel

19.4.0

add_host_metadata

host.os.name

Mac OS X

add_host_metadata

host.os.version

10.15.4

add_host_metadata

user_agent.name

Chrome

user_agent

user_agent.os.full

Mac OS X 10.15.4

user_agent

user_agent.os.name

Mac OS X

user_agent

user_agent.os.version

10.15.4

user_agent

user_agent.version

83.0.4103.106

user_agent

Field Mapping Reference Guides

We’ve covered at a high level how to map your events to ECS. Now if you’d like your events to render well in the Elastic solutions, check out the reference guides below to learn more about each:

Guidelines and Best Practices

The ECS schema serves best when you follow schema guidelines and best practices.

ECS Field Levels

ECS defines "Core" and "Extended" fields.

  • Core fields. Fields that are most common across all use cases are defined as core fields.

    These generalized fields are used by analysis content (searches, visualizations, dashboards, alerts, machine learning jobs, reports) across use cases. Analysis content designed to operate on these fields should work properly on data from any relevant source.

    Focus on populating these fields first.

  • Extended fields. Any field that is not a core field is defined as an extended field. Extended fields may apply to more narrow use cases, or may be more open to interpretation depending on the use case. Extended fields are more likely to change over time.

Each ECS field in a table is identified as core or extended.

General guidelines

  • The document MUST have the @timestamp field.

  • defined for an ECS field.

  • Use the ecs.version field to define which version of ECS is used.

  • Map as many fields as possible to ECS.

Guidelines for field names

  • Field names must be lower case

  • Combine words using underscore

  • No special characters except underscore

  • Use present tense unless field describes historical information.

  • Use singular and plural names properly to reflect the field content.

    • For example, use requests_per_sec rather than request_per_sec.

  • Use prefixes for all fields, except for the base fields.

    • For example, all host fields are prefixed with host.. Such a grouping is called a field set.

  • Nest fields inside a field set with dots

    • The document structure should be nested JSON objects. If you use Beats or Logstash, the nesting of JSON objects is done for you automatically. If you’re ingesting to Elasticsearch using the API, your fields must be nested objects, not strings containing dots.

    • See [dot-notation] for more details.

  • General to specific. Organise the nesting of field sets from general to specific, to allow grouping fields into objects with a prefix like host.*.

  • Avoid repetition or stuttering of words

    • If part of the field name is already in the name of the field set, avoid repeating it. Example: host.host_ip should be host.ip.

    • Exceptions can be made, when changing the name of the field would break a strong convention in the community. Example: host.hostname is an exception to this rule.

  • Avoid abbreviations when possible

    • Exceptions can be made, when the name used for the concept is too strongly in favor of the abbreviation. Example: ip fields, or field sets such as os, geo.

Conventions

The implementation of ECS follows a few conventions. Understanding them will help you understand ECS better.

Datatype for integers

Unless otherwise noted, the datatype used for integer fields should be long.

IDs and most codes are keywords, not integers

Despite the fact that IDs and codes (such as error codes) are often integers, this is not always the case. Since we want to make it possible to map as many systems and data sources to ECS as possible, we default to using the keyword type for IDs and codes.

Some specific kinds of codes are always integers, like HTTP status codes. If those have a specific corresponding specific field (as HTTP status does), its type can safely be an integer type. But generic fields like error.code cannot have this guarantee, and are therefore keyword.

Text indexing and multi-fields

Elasticsearch can index text using datatypes:

  • text Text indexing allows for full text search, or searching arbitrary words that are part of the field.

  • keyword Keyword indexing offers faster exact match filtering, prefix search (like autocomplete),

Default Elasticsearch convention for indexing text fields

Unless your index mapping or index template specifies otherwise (as the ECS index template does), Elasticsearch indexes a text field as text at the canonical field name, and indexes a second time as keyword, nested in a multi-field.

Default Elasticsearch convention:

  • Canonical field: myfield is text

  • Multi-field: myfield.keyword is keyword

ECS convention for indexing text fields

ECS flips the above convention around.

For monitoring use cases, keyword indexing is needed almost exclusively, with full text search needed on very few fields. Moreover, indexing for full text search on lots of fields, where it’s not expected to be used is wasteful of resources.

Given these two premises, ECS defaults all text indexing to keyword datatype (with very few exceptions). Any use case that requires full text search indexing on additional fields for full text search. Doing so does not conflict with ECS, as the canonical field name will remain keyword indexed.

So the ECS multi-field convention for text is:

  • Canonical field: myfield is keyword

  • Multi-field: myfield.text is text

Exceptions

The only exceptions to this convention are fields message and error.message, which are indexed for full text search only, with no multi-field. These two fields don’t follow the new convention because they are deemed too big of a breaking change with these two widely used fields in Beats.

Any future field that will be indexed for full text search in ECS will however follow the multi-field convention where text indexing is nested in the multi-field.

Custom Fields

ECS defines fields, their datatypes and their usage, and classifies them in "core" and "extended" levels.

However, ECS does not define anything about custom fields. By definition, they are additional fields, exactly as the user or the integration defines them, independently of ECS.

Users and integrations are welcome to capture additional information in their events, as custom fields. This flexibility is by design, and ensures that no one is ever blocked by something not being supported by ECS yet.

ECS is under active development, however. Adding custom fields carries a small risk of conflicting with a future ECS field. There are ways of modeling custom fields that will lead to lower chances of conflict with future versions of ECS. This section outlines a few of these strategies.

Modeling to Reduce Chances of Conflict

The labels Field

Any time a data source has a few extra fields that can be modeled with the keyword data type, the simplest way to capture them is with the ECS field labels.

Example:

{ "labels": { "foo_id": "beef42", "env": "production" },
  "message": "...",
  "event": { ... }
}

If labels doesn’t work for your use case, here’s a few more tips to avoid conflicts.

Proper Names

ECS tries to model information by using the name of concepts, and avoids proper names such as tool names, project names or company names. By extension, nesting custom fields under a proper name is a relatively safe approach to adding custom fields. This is the approach taken by Filebeats modules, for example.

As an example, an HTTP log from HAProxy will contain typical HTTP information, as well as proxy details and statistics. The standard HTTP information can be captured in the ECS field sets http and url, and the extra details in a custom haproxy section:

{ "http": { "request": { "method": "get", ... },
            "response": { "status_code": 200, ... } },
  "url": { "original": "/favicon.ico", ... },
  "haproxy": { "frontend_name": "myfrontend", "backend_name": "mybackend_prod",
               "backend_queue": 0, ... }
}
Capitalization

ECS strives for a consistent feel by using nesting to group related concepts, and underscores to join words. Following these guidelines for custom fields ensures you preserve the same consistent experience throughout your schemas.

Note however, that breaking away from these guidelines for your custom fields can be used to your advantage. It can be a good way to differentiate between ECS fields and custom fields. Since ECS doesn’t use capitalization for field names, this approach virtually guarantees that custom fields will not conflict with future ECS fields.

Common proxy concepts could modelled via a capitalized, but generic concept name.

HAProxy example:

{ "http": { "request": { "method": "get", ... } },
  "url": { "original": "/favicon.ico", ... },
  "Proxy": { "FrontendName": "myfrontend", "BackendName": "mybackend_prod" },
  "event": { "module": "haproxy" }
}

NGINX example:

{ "http": { "request": { "method": "get", ... } },
  "url": { "original": "/favicon.ico", ... },
  "Proxy": { "FrontendName": "another_frontend", "BackendName": "another_backend_prod" },
  "event": { "module": "nginx" }
}

The above demonstrates that using a common concept name in custom fields can still be beneficial to correlate among multiple sources that populate them. Using capitalization ensures a future version of ECS that defines a proxy field set will not conflict.

Here’s a sample event, during a migration from the custom field, to using a new equivalent ECS field set:

{ "http": { "request": { "method": "get", ... } },
  "Proxy": { "FrontendName": "myfrontend", "BackendName": "mybackend_prod" },
  "proxy": { "frontend_name": "myfrontend", "backend_name": "mybackend_prod" }
}

The above will look strange during the migration. However the ability to start populating ECS fields while custom fields are still present in your events makes it possible to decouple the upgrade to a new version of ECS from the time you adjust your pipelines and analysis content.

Mapping Network Events

Network events capture the details of one device communicating with another. The initiator is referred to as the source, and the recipient as the destination. Depending on the data source, a network event can contain details of addresses, protocols, headers, and device roles.

This guide describes the different field sets available for network-related events in ECS and provides direction on the ECS best practices for mapping to them.

Source and destination baseline

When an event contains details about the sending and receiving hosts, the baseline for capturing these values will be the source and destination fields.

Some events may also indicate each host’s role in the exchange: client or server. When this information is available, the client and server fields should be used in addition to the source and destination fields. The fields and values mapped under source/destination should be copied under client/server.

Network event mapping example

Below is a DNS network event. The source device (192.168.86.222) makes a DNS query, acting as the client and the DNS server is the destination (192.168.86.1).

Note this event contains additional details that would populate additional fields (such as the [ecs-dns]) if this was a complete mapping example. These additional fields are omitted here to focus on the network details.

{
  "ts":1599775747.53056,
  "uid":"CYqFPH3nOAa0kPxA0d",
  "id.orig_h":"192.168.86.222",
  "id.orig_p":54162,
  "id.resp_h":"192.168.86.1",
  "id.resp_p":53,
  "proto":"udp",
  "trans_id":28899,
  "rtt":0.02272200584411621,
  "query":"example.com",
  "qclass":1,
  "qclass_name":"C_INTERNET",
  "qtype":1,
  "qtype_name":"A",
  "rcode":0,
  "rcode_name":"NOERROR",
  "AA":false,
  "TC":false,
  "RD":true,
  "RA":true,
  "Z":0,
  "answers":["93.184.216.34"],
  "TTLs":[21209.0],
  "rejected":false
}

Source and destination fields

First, the source.* and destination.* field sets are populated:

  "source": {
    "ip": "192.168.86.222",
    "port": 54162
  }
  "destination": {
    "ip": "192.168.86.1",
    "port": 53
  }

Client and server fields

Looking back at the original event, it shows the source device is the DNS client and the destination device is the DNS server. The values mapped under source and destination are copied and mapped under client and server, respectively:

  "client": {
    "ip": "192.168.86.222",
    "port": 54162
  }
  "server": {
    "ip": "192.168.86.1",
    "port": 53
  }

Mapping both pairs of field sets gives query visibility of the same network transaction in two ways.

  • source.ip:192.168.86.222 returns all events sourced from 192.168.86.222, regardless its role in a transaction

  • client.ip:192.168.86.222 returns all events with host 192.168.86.222 acting as a client

The same applies for the destination and server fields:

  • destination.ip:192.168.86.1 returns all events destined to 192.168.86.1

  • server.ip:192.168.86.1 returns all events with 192.168.86.1 acting as the server

It’s important to note that while the values for the source and destination fields may reverse between events in a single network transaction, the values for client and server typically will not. The following two tables demonstrate how two DNS transactions involving two clients and one server would map to source.ip/destination.ip vs. client.ip/server.ip:

Table 1. Source/Destination
source.ip destination.ip event

192.168.86.222

192.168.86.1

DNS query request 1

192.168.86.1

192.168.86.222

DNS answer response 1

192.168.86.42

192.168.86.1

DNS answer request 2

192.168.86.1

192.168.86.42

DNS answer request 2
Table 2. Client/Server
client.ip server.ip event

192.168.86.222

192.168.86.1

DNS query request 1

192.168.86.222

192.168.86.1

DNS answer response 1

192.168.86.42

192.168.86.1

DNS query request 2

192.168.86.42

192.168.86.1

DNS answer response 2

Related fields

The related.ip field captures all the IPs present in the event in a single array:

  "related": {
    "ip": [
      "192.168.86.222",
      "192.168.86.1",
      "93.184.216.34"
    ]
  }

The related fields are meant to facilitate pivoting. Since these IP addresses can appear in many different fields (source.ip, destination.ip, client.ip, server.ip, etc.), you can search for the IP trivially no matter what field it appears using a single query, e.g. related.ip:192.168.86.222.

Network events are not only limited to using related.ip. If hostnames or other host identifiers were present in the event, related.hosts should be populated too.

Categorization using event fields

When considering the event categorization fields, the category and type fields are populated using their respective allowed values which best classify the source network event.

  "event": {
    "category": [
      "network"
    ],
    "type": [
      "connection",
      "protocol"
    ],
    "kind": "event"
  }

Most event.category/event.type ECS pairings are complete on their own. However, the pairing of event.category:network and event.type:protocol is an exception. When these two fields/value pairs both used to categorize an event, the network.protocol field should also be populated:

  "network": {
    "protocol": "dns",
    "type": "ipv4",
    "transport": "udp"
  }

Result

Putting everything together covered so far, we have a final ECS-mapped event:

{
  "event": {
    "category": [
      "network"
    ],
    "type": [
      "connection",
      "protocol"
    ],
    "kind": "event"
  },
  "network": {
    "protocol": "dns",
    "type": "ipv4",
    "transport": "udp"
  },
  "source": {
    "ip": "192.168.86.222",
    "port": 54162
  },
  "destination": {
    "ip": "192.168.86.1",
    "port": 53
  },
  "client": {
    "ip": "192.168.86.222",
    "port": 64734
  },
  "server": {
    "ip": "192.168.86.1",
    "port": 53
  },
  "related": {
    "ip": [
      "192.168.86.222",
      "192.168.86.1",
      "93.184.216.34"
    ]
  },
  "dns": { ... }, <= Again, not diving into the DNS fields here but included for completeness.
  "zeek": { "ts":1599775747.53056, ... } <= Original fields can optionally be kept around as custom fields.
}

ECS Field Reference

This is the documentation of ECS version 1.11.0-7.

ECS defines multiple groups of related fields. They are called "field sets". The Base field set is the only one whose fields are defined at the root of the event.

all fields are defined.

For a single page representation of all fields, please see the /generated/csv/fields.csv[generated CSV of fields].

Field Sets

Field Set Description

Base

All fields defined directly at the root of the events.

Autonomous System

Fields describing an Autonomous System (Internet routing prefix).

Client

Fields about the client side of a network connection, used with server.

Container

Fields describing the container that generated this event.

Destination

Fields about the destination side of a network connection, used with source.

ECS

Meta-information specific to ECS.

Error

Fields about errors of any kind.

Event

Fields breaking down the event details.

Geo

Fields describing a location.

Git

Fields related to git repositories

Hash

Hashes, usually file hashes.

Host

Fields describing the relevant computing instance.

HTTP

Fields describing an HTTP request.

Interface

Fields to describe observer interface information.

Log

Details about the event’s logging mechanism.

Metrics

Fields for metrics generated from logs.

Network

Fields describing the communication path over which the event happened.

Normalized

Fields generated and normalized by the logging pipeline.

Observer

Fields describing an entity observing the event from outside the host.

Orchestrator

Fields relevant to container orchestrators.

Operating System

OS fields contain information about the operating system.

Package

These fields contain information about an installed software package.

Process

These fields contain information about a process.

Rule

Fields to capture details about rules used to generate alerts or other notable events.

Server

Fields about the server side of a network connection, used with client.

Service

Fields describing the service for or from which the data was collected.

Source

Fields about the source side of a network connection, used with destination.

TLS

Fields describing a TLS connection.

Tracing

Fields related to distributed tracing.

URL

Fields that let you store URLs in various forms.

User

Fields to describe the user relevant to the event.

User agent

Fields to describe a browser user_agent string.

Base Fields

The base field set contains all fields which are at the root of the events. These fields are common across all types of events.

Field Description Level

@timestamp

Date/time when the event originated.

This is the date/time extracted from the event, typically representing when the event was generated by the source.

If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline.

Required field for all events.

type: date

example: 2016-05-23T08:05:34.853Z

core

labels

Custom key/value pairs.

Can be used to add meta information to events. Should not contain nested objects. All values are stored as keyword.

Example: docker and k8s labels.

type: object

example: {"application": "foo-bar", "env": "production"}

core

message

For log events the message field contains the log message, optimized for viewing in a log viewer.

For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event.

If multiple messages exist, they can be combined into one message.

type: text

example: Hello World

core

tags

List of keywords used to tag each event.

type: keyword

Note: this field should contain an array of values.

example: ["production", "env2"]

core

Autonomous System Fields

An autonomous system (AS) is a collection of connected Internet Protocol (IP) routing prefixes under the control of one or more network operators on behalf of a single administrative entity or domain that presents a common, clearly defined routing policy to the internet.

Field Description Level

as.number

Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.

type: long

example: 15169

extended

as.organization.name

Organization name.

type: keyword

Multi-fields:

* as.organization.name.text (type: text)

example: Google LLC

extended

The as fields are expected to be nested at:

  • client.as

  • destination.as

  • server.as

  • source.as

  • threat.enrichments.indicator.as

  • threat.indicator.as

Note also that the as fields are not expected to be used directly at the root of the events.

Client Fields

A client is defined as the initiator of a network connection for events regarding sessions, connections, or bidirectional flow records.

For TCP events, the client is the initiator of the TCP connection that sends the SYN packet(s). For other protocols, the client is generally the initiator or requestor in the network transaction. Some systems use the term "originator" to refer the client in TCP connections. The client fields describe details about the system acting as the client in the network event. Client fields are usually populated in conjunction with server fields. Client fields are generally not populated for packet-level events.

Client / server representations can add semantic context to an exchange, which is helpful to visualize the data in certain situations. If your context falls in that category, you should still ensure that source and destination are filled appropriately.

Field Description Level

client.address

Some event client addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field.

Then it should be duplicated to .ip or .domain, depending on which one it is.

type: keyword

extended

client.bytes

Bytes sent from the client to the server.

type: long

example: 184

core

client.domain

Client domain.

type: keyword

core

client.ip

IP address of the client (IPv4 or IPv6).

type: ip

core

client.mac

MAC address of the client.

The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen.

type: keyword

example: 00-00-5E-00-53-23

core

client.nat.ip

Translated IP of source based NAT sessions (e.g. internal client to internet).

Typically connections traversing load balancers, firewalls, or routers.

type: ip

extended

client.nat.port

Translated port of source based NAT sessions (e.g. internal client to internet).

Typically connections traversing load balancers, firewalls, or routers.

type: long

extended

client.packets

Packets sent from the client to the server.

type: long

example: 12

core

client.port

Port of the client.

type: long

core

client.registered_domain

The highest registered client domain, stripped of the subdomain.

For example, the registered domain for "foo.example.com" is "example.com".

This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last two labels will not work well for TLDs such as "co.uk".

type: keyword

example: example.com

extended

client.subdomain

The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain. In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain.

For example the subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.

type: keyword

example: east

extended

client.top_level_domain

The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com".

This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".

type: keyword

example: co.uk

extended
Location Field Set Description

client.as.*

as

Fields describing an Autonomous System (Internet routing prefix).

client.geo.*

geo

Fields describing a location.

client.user.*

user

Fields to describe the user relevant to the event.

Container Fields

Container fields are used for meta information about the specific container that is the source of information.

These fields help correlate data based containers from any runtime.

Field Description Level

container.id

Unique container id.

type: keyword

core

container.image.name

Name of the image the container was built on.

type: keyword

extended

container.image.tag

Container image tags.

type: keyword

Note: this field should contain an array of values.

extended

container.labels

Image labels.

type: object

extended

container.name

Container name.

type: keyword

extended

container.runtime

Runtime managing this container.

type: keyword

example: docker

extended

Destination Fields

Destination fields capture details about the receiver of a network exchange/packet. These fields are populated from a network event, packet, or other event containing details of a network transaction.

Destination fields are usually populated in conjunction with source fields. The source and destination fields are considered the baseline and should always be filled if an event contains source and destination details from a network transaction. If the event also contains identification of the client and server roles, then the client and server fields should also be populated.

Field Description Level

destination.address

Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field.

Then it should be duplicated to .ip or .domain, depending on which one it is.

type: keyword

extended

destination.bytes

Bytes sent from the destination to the source.

type: long

example: 184

core

destination.domain

Destination domain.

type: keyword

core

destination.ip

IP address of the destination (IPv4 or IPv6).

type: ip

core

destination.mac

MAC address of the destination.

The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen.

type: keyword

example: 00-00-5E-00-53-23

core

destination.nat.ip

Translated ip of destination based NAT sessions (e.g. internet to private DMZ)

Typically used with load balancers, firewalls, or routers.

type: ip

extended

destination.nat.port

Port the source session is translated to by NAT Device.

Typically used with load balancers, firewalls, or routers.

type: long

extended

destination.packets

Packets sent from the destination to the source.

type: long

example: 12

core

destination.port

Port of the destination.

type: long

core

destination.registered_domain

The highest registered destination domain, stripped of the subdomain.

For example, the registered domain for "foo.example.com" is "example.com".

This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last two labels will not work well for TLDs such as "co.uk".

type: keyword

example: example.com

extended

destination.subdomain

The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain. In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain.

For example the subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.

type: keyword

example: east

extended

destination.top_level_domain

The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com".

This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".

type: keyword

example: co.uk

extended
Location Field Set Description

destination.as.*

as

Fields describing an Autonomous System (Internet routing prefix).

destination.geo.*

geo

Fields describing a location.

destination.user.*

user

Fields to describe the user relevant to the event.

ECS Fields

Meta-information specific to ECS.

Field Description Level

ecs.version

ECS version this event conforms to. ecs.version is a required field and must exist in all events.

Must match ^(?<major>[0-9]+)\.(?<minor>[0-9]+)\.(?<patch>[0-9]+)$.

When querying across multiple indices — which may conform to slightly different ECS versions — this field lets integrations adjust to the schema version of the events.

type: keyword

example: 1.0.0

core

Error Fields

These fields can represent errors of any kind.

Use them for errors that happen while fetching events or in cases where the event itself contains an error.

Field Description Level

error.code

Error code describing the error.

type: keyword

core

error.id

Unique identifier for the error.

type: keyword

core

error.message

Error message.

type: text

core

error.stack.head

The first line of the stack trace useful for aggregation.

This is an attribute of the stack object, unlike stack_trace.

type: keyword

custom

error.stack.previous_trace

The previous stack trace.

type: keyword

Multi-fields:

* error.stack.previous_trace.text (type: text)

custom

error.stack_trace

The stack trace of this error in plain text.

type: keyword

Multi-fields:

* error.stack_trace.text (type: text)

extended

error.type

The type of the error, for example the class name of the exception.

type: keyword

example: java.lang.NullPointerException

extended

Event Fields

The event fields are used for context information about the log or metric event itself.

A log is defined as an event containing details of something that happened. Log events must include the time at which the thing happened. Examples of log events include a process starting on a host, a network packet being sent from a source to a destination, or a network connection between a client and a server being initiated or closed. A metric is defined as an event containing one or more numerical measurements and the time at which the measurement was taken. Examples of metric events include memory pressure measured on a host and device temperature. See the event.kind definition in this section for additional details about metric and state events.

Field Description Level

event.action

The action captured by the event.

This describes the information in the event. It is more specific than event.category. Examples are group-add, process-started, file-created. The value is normally defined by the implementer.

type: keyword

example: user-password-change

core

event.agent_id_status

Agents are normally responsible for populating the agent.id field value. If the system receiving events is capable of validating the value based on authentication information for the client then this field can be used to reflect the outcome of that validation.

For example if the agent’s connection is authenticated with mTLS and the client cert contains the ID of the agent to which the cert was issued then the agent.id value in events can be checked against the certificate. If the values match then event.agent_id_status: verified is added to the event, otherwise one of the other allowed values should be used.

If no validation is performed then the field should be omitted.

The allowed values are:

verified - The agent.id field value matches expected value obtained from auth metadata.

mismatch - The agent.id field value does not match the expected value obtained from auth metadata.

missing - There was no agent.id field in the event to validate.

auth_metadata_missing - There was no auth metadata or it was missing information about the agent ID.

type: keyword

example: verified

extended

event.category

This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy.

event.category represents the "big buckets" of ECS categories. For example, filtering on event.category:process yields all events relating to process activity. This field is closely related to event.type, which is used as a subcategory.

This field is an array. This will allow proper categorization of some events that fall in multiple categories.

type: keyword

Note: this field should contain an array of values.

Important: The field value must be one of the following:

authentication, configuration, database, driver, file, host, iam, intrusion_detection, malware, network, package, process, registry, session, threat, web

To learn more about when to use which value, visit the page allowed values for event.category

core

event.code

Identification code for this event, if one exists.

Some event sources use event codes to identify messages unambiguously, regardless of message language or wording adjustments over time. An example of this is the Windows Event ID.

type: keyword

example: 4648

extended

event.created

event.created contains the date/time when the event was first read by an agent, or by your pipeline.

This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event.

In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent’s or pipeline’s ability to keep up with your event source.

In case the two timestamps are identical, @timestamp should be used.

type: date

example: 2016-05-23T08:05:34.857Z

core

event.dataset

Name of the dataset.

If an event source publishes more than one type of log or events (e.g. access log, error log), the dataset is used to specify which one the event comes from.

It’s recommended but not required to start the dataset name with the module name, followed by a dot, then the dataset name.

type: keyword

example: apache.access

core

event.duration

Duration of the event in nanoseconds.

If event.start and event.end are known this value should be the difference between the end and start time.

type: long

core

event.end

event.end contains the date when the event ended or when the activity was last observed.

type: date

extended

event.hash

Hash (perhaps logstash fingerprint) of raw field to be able to demonstrate log integrity.

type: keyword

example: 123456789012345678901234567890ABCD

extended

event.id

Unique ID to describe the event.

type: keyword

example: 8a4f500d

core

event.ingested

Timestamp when an event arrived in the central data store.

This is different from @timestamp, which is when the event originally occurred. It’s also different from event.created, which is meant to capture the first time an agent saw the event.

In normal conditions, assuming no tampering, the timestamps should chronologically look like this: @timestamp < event.created < event.ingested.

type: date

example: 2016-05-23T08:05:35.101Z

core

event.kind

This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy.

event.kind gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events.

The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not.

type: keyword

Important: The field value must be one of the following:

alert, enrichment, event, metric, state, pipeline_error, signal

To learn more about when to use which value, visit the page allowed values for event.kind

core

event.module

Name of the module this data is coming from.

If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), event.module should contain the name of this module.

type: keyword

example: apache

core

event.original

Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex.

This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from _source. If users wish to override this and index this field, please see Field data types in the Elasticsearch Reference.

type: keyword

example: Sep 19 08:26:10 host CEF:0&#124;Security&#124; threatmanager&#124;1.0&#124;100&#124; worm successfully stopped&#124;10&#124;src=10.0.0.1 dst=2.1.2.2spt=1232

core

event.outcome

This is one of four ECS Categorization Fields, and indicates the lowest level in the ECS category hierarchy.

event.outcome simply denotes whether the event represents a success or a failure from the perspective of the entity that produced the event.

Note that when a single transaction is described in multiple events, each event may populate different values of event.outcome, according to their perspective.

Also note that in the case of a compound event (a single event that contains multiple logical events), this field should be populated with the value that best captures the overall success or failure from the perspective of the event producer.

Further note that not all events will have an associated outcome. For example, this field is generally not populated for metric events, events with event.type:info, or any events for which an outcome does not make logical sense.

type: keyword

Important: The field value must be one of the following:

failure, success, unknown

To learn more about when to use which value, visit the page allowed values for event.outcome

core

event.provider

Source of the event.

Event transports such as Syslog or the Windows Event Log typically mention the source of an event. It can be the name of the software that generated the event (e.g. Sysmon, httpd), or of a subsystem of the operating system (kernel, Microsoft-Windows-Security-Auditing).

type: keyword

example: kernel

extended

event.reason

Reason why this event happened, according to the source.

This describes the why of a particular action or outcome captured in the event. Where event.action captures the action from the event, event.reason describes why that action was taken. For example, a web proxy with an event.action which denied the request may also populate event.reason with the reason why (e.g. blocked site).

type: keyword

example: Terminated an unexpected process

extended

event.reference

Reference URL linking to additional information about this event.

This URL links to a static definition of this event. Alert events, indicated by event.kind:alert, are a common use case for this field.

type: keyword

example: https://system.example.com/event/#0001234

extended

event.risk_score

Risk score or priority of the event (e.g. security solutions). Use your system’s original value here.

type: float

core

event.risk_score_norm

Normalized risk score or priority of the event, on a scale of 0 to 100.

This is mainly useful if you use more than one system that assigns risk scores, and you want to see a normalized value across all systems.

type: float

extended

event.sequence

Sequence number of the event.

The sequence number is a value published by some event sources, to make the exact ordering of events unambiguous, regardless of the timestamp precision.

type: long

extended

event.severity

The numeric severity of the event according to your event source.

What the different severity values mean can be different between sources and use cases. It’s up to the implementer to make sure severities are consistent across events from the same source.

The Syslog severity belongs in log.syslog.severity.code. event.severity is meant to represent the severity according to the event source (e.g. firewall, IDS). If the event source does not publish its own severity, you may optionally copy the log.syslog.severity.code to event.severity.

type: long

example: 7

core

event.start

event.start contains the date when the event started or when the activity was first observed.

type: date

extended

event.timezone

This field should be populated when the event’s timestamp does not include timezone information already (e.g. default Syslog timestamps). It’s optional otherwise.

Acceptable timezone formats are: a canonical ID (e.g. "Europe/Amsterdam"), abbreviated (e.g. "EST") or an HH:mm differential (e.g. "-05:00").

type: keyword

extended

event.type

This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy.

event.type represents a categorization "sub-bucket" that, when used along with the event.category field values, enables filtering events down to a level appropriate for single visualization.

This field is an array. This will allow proper categorization of some events that fall in multiple event types.

type: keyword

Note: this field should contain an array of values.

Important: The field value must be one of the following:

access, admin, allowed, change, connection, creation, deletion, denied, end, error, group, indicator, info, installation, protocol, start, user

To learn more about when to use which value, visit the page allowed values for event.type

core

event.url

URL linking to an external system to continue investigation of this event.

This URL links to another system where in-depth investigation of the specific occurrence of this event can take place. Alert events, indicated by event.kind:alert, are a common use case for this field.

type: keyword

example: https://mysystem.example.com/alert/5271dedb-f5b0-4218-87f0-4ac4870a38fe

extended

Geo Fields

Geo fields can carry data about a specific location related to an event.

This geolocation information can be derived from techniques such as Geo IP, or be user-supplied.

Field Description Level

geo.city_name

City name.

type: keyword

example: Montreal

core

geo.continent_code

Two-letter code representing continent’s name.

type: keyword

example: NA

core

geo.continent_name

Name of the continent.

type: keyword

example: North America

core

geo.country_iso_code

Country ISO code.

type: keyword

example: CA

core

geo.country_name

Country name.

type: keyword

example: Canada

core

geo.location

Longitude and latitude.

type: geo_point

example: { "lon": -73.614830, "lat": 45.505918 }

core

geo.name

User-defined description of a location, at the level of granularity they care about.

Could be the name of their data centers, the floor number, if this describes a local physical entity, city names.

Not typically used in automated geolocation.

type: keyword

example: boston-dc

extended

geo.postal_code

Postal code associated with the location.

Values appropriate for this field may also be known as a postcode or ZIP code and will vary widely from country to country.

type: keyword

example: 94040

core

geo.region_iso_code

Region ISO code.

type: keyword

example: CA-QC

core

geo.region_name

Region name.

type: keyword

example: Quebec

core

geo.timezone

The time zone of the location, such as IANA time zone name.

type: keyword

example: America/Argentina/Buenos_Aires

core

The geo fields are expected to be nested at:

  • client.geo

  • destination.geo

  • host.geo

  • observer.geo

  • server.geo

  • source.geo

  • threat.enrichments.indicator.geo

  • threat.indicator.geo

Note also that the geo fields are not expected to be used directly at the root of the events.

Git Fields

Fields related to git repositories

Field Description Level

git.commit.author

The author of the git commit

type: keyword

custom

git.commit.hash.sha1.full

The full sha1 hash of a git commit

type: keyword

custom

git.commit.hash.sha1.short

The short sha1 hash of a git commit

type: keyword

custom

git.repo.name

The name of the git repo

type: keyword

custom

git.repo.url.domain

Domain of the url

type: keyword

custom

git.repo.url.full

Unmodified original url as seen in the event source.

type: keyword

custom

git.repo.url.original

Unmodified original url as seen in the event source.

type: keyword

custom

git.repo.url.scheme

Scheme of the request, such as "https" or "ssh"

type: keyword

custom

Hash Fields

The hash fields represent different bitwise hash algorithms and their values.

Field names for common hashes (e.g. MD5, SHA1) are predefined. Add fields for other hashes by lowercasing the hash algorithm name and using underscore separators as appropriate (snake case, e.g. sha3_512).

Note that this fieldset is used for common hashes that may be computed over a range of generic bytes. Entity-specific hashes such as ja3 or imphash are placed in the fieldsets to which they relate (tls and pe, respectively).

Field Description Level

hash.md5

MD5 hash.

type: keyword

extended

hash.sha1

SHA1 hash.

type: keyword

extended

hash.sha256

SHA256 hash.

type: keyword

extended

hash.sha512

SHA512 hash.

type: keyword

extended

hash.ssdeep

SSDEEP hash.

type: keyword

extended

The hash fields are expected to be nested at:

  • dll.hash

  • file.hash

  • process.hash

  • threat.enrichments.indicator.hash

  • threat.indicator.hash

Note also that the hash fields are not expected to be used directly at the root of the events.

Host Fields

A host is defined as a general computing instance.

ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.

Field Description Level

host.architecture

Operating system architecture.

type: keyword

example: x86_64

core

host.cpu.usage

Percent CPU used which is normalized by the number of CPU cores and it ranges from 0 to 1.

Scaling factor: 1000.

For example: For a two core host, this value should be the average of the two cores, between 0 and 1.

type: scaled_float

extended

host.disk.read.bytes

The total number of bytes (gauge) read successfully (aggregated from all disks) since the last metric collection.

type: long

extended

host.disk.write.bytes

The total number of bytes (gauge) written successfully (aggregated from all disks) since the last metric collection.

type: long

extended

host.domain

Name of the domain of which the host is a member.

For example, on Windows this could be the host’s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host’s LDAP provider.

type: keyword

example: CONTOSO

extended

host.hostname

Hostname of the host.

It normally contains what the hostname command returns on the host machine.

type: keyword

core

host.id

Unique host id.

As hostname is not always unique, use values that are meaningful in your environment.

Example: The current usage of beat.name.

type: keyword

core

host.ip

Host ip addresses.

type: ip

Note: this field should contain an array of values.

core

host.mac

Host MAC addresses.

The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen.

type: keyword

Note: this field should contain an array of values.

example: ["00-00-5E-00-53-23", "00-00-5E-00-53-24"]

core

host.name

Name of the host.

It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.

type: keyword

core

host.network.egress.bytes

The number of bytes (gauge) sent out on all network interfaces by the host since the last metric collection.

type: long

extended

host.network.egress.packets

The number of packets (gauge) sent out on all network interfaces by the host since the last metric collection.

type: long

extended

host.network.ingress.bytes

The number of bytes received (gauge) on all network interfaces by the host since the last metric collection.

type: long

extended

host.network.ingress.packets

The number of packets (gauge) received on all network interfaces by the host since the last metric collection.

type: long

extended

host.type

Type of host.

For Cloud providers this can be the machine type like t2.medium. If vm, this could be the container, for example, or other information meaningful in your environment.

type: keyword

core

host.uptime

Seconds the host has been up.

type: long

example: 1325

extended
Location Field Set Description

host.geo.*

geo

Fields describing a location.

host.os.*

os

OS fields contain information about the operating system.

host.user.*

user

This reuse is deprecated and will be removed in the next major ECS version.

HTTP Fields

Fields related to HTTP activity. Use the url field set to store the url of the request.

Field Description Level

http.request.body.bytes

Size in bytes of the request body.

type: long

example: 887

extended

http.request.body.content

The full HTTP request body.

type: keyword

Multi-fields:

* http.request.body.content.text (type: text)

example: Hello world

extended

http.request.bytes

Total size in bytes of the request (body and headers).

type: long

example: 1437

extended

http.request.headers

Request headers in key/value pairs.

Can be used add request header information to events. Should not contain nested objects. All values are stored as keyword.

type: object

example: {"content-type": "application/json", "cache-status": "Miss"}

custom

http.request.id

A unique identifier for each HTTP request to correlate logs between clients and servers in transactions.

The id may be contained in a non-standard HTTP header, such as X-Request-ID or X-Correlation-ID.

type: keyword

example: 123e4567-e89b-12d3-a456-426614174000

extended

http.request.method

HTTP request method.

Prior to ECS 1.6.0 the following guidance was provided:

"The field value must be normalized to lowercase for querying."

As of ECS 1.6.0, the guidance is deprecated because the original case of the method may be useful in anomaly detection. Original case will be mandated in ECS 2.0.0

type: keyword

example: GET, POST, PUT, PoST

extended

http.request.mime_type

Mime type of the body of the request.

This value must only be populated based on the content of the request body, not on the Content-Type header. Comparing the mime type of a request with the request’s Content-Type header can be helpful in detecting threats or misconfigured clients.

type: keyword

example: image/gif

extended

http.request.referrer

Referrer for this HTTP request.

type: keyword

example: https://blog.example.com/

extended

http.response.body.bytes

Size in bytes of the response body.

type: long

example: 887

extended

http.response.body.content

The full HTTP response body.

type: keyword

Multi-fields:

* http.response.body.content.text (type: text)

example: Hello world

extended

http.response.bytes

Total size in bytes of the response (body and headers).

type: long

example: 1437

extended

http.response.headers

Response headers in key/value pairs.

Can be used add response header information to events. Should not contain nested objects. All values are stored as keyword.

type: object

example: {"content-type": "application/json", "cache-status": "Miss"}

custom

http.response.mime_type

Mime type of the body of the response.

This value must only be populated based on the content of the response body, not on the Content-Type header. Comparing the mime type of a response with the response’s Content-Type header can be helpful in detecting misconfigured servers.

type: keyword

example: image/gif

extended

http.response.status_code

HTTP response status code.

type: long

example: 404

extended

http.version

HTTP version.

type: keyword

example: 1.1

extended

Interface Fields

The interface fields are used to record ingress and egress interface information when reported by an observer (e.g. firewall, router, load balancer) in the context of the observer handling a network connection. In the case of a single observer interface (e.g. network sensor on a span port) only the observer.ingress information should be populated.

Field Description Level

interface.alias

Interface alias as reported by the system, typically used in firewall implementations for e.g. inside, outside, or dmz logical interface naming.

type: keyword

example: outside

extended

interface.id

Interface ID as reported by an observer (typically SNMP interface ID).

type: keyword

example: 10

extended

interface.name

Interface name as reported by the system.

type: keyword

example: eth0

extended

The interface fields are expected to be nested at:

  • observer.egress.interface

  • observer.ingress.interface

Note also that the interface fields are not expected to be used directly at the root of the events.

Log Fields

Details about the event’s logging mechanism or logging transport.

The log.* fields are typically populated with details about the logging mechanism used to create and/or transport the event. For example, syslog details belong under log.syslog.*.

The details specific to your event source are typically not logged under log.*, but rather in event.* or in other ECS fields.

Field Description Level

log.file.path

Full path to the log file this event came from, including the file name. It should include the drive letter, when appropriate.

If the event wasn’t read from a log file, do not populate this field.

type: keyword

example: /var/log/fun-times.log

extended

log.level

Original log level of the log event.

If the source of the event provides a log level or textual severity, this is the one that goes in log.level. If your source doesn’t specify one, you may put your event transport’s severity here (e.g. Syslog severity).

Some examples are warn, err, i, informational.

type: keyword

example: error

core

log.logger

The name of the logger inside an application. This is usually the name of the class which initialized the logger, or can be a custom name.

type: keyword

example: org.elasticsearch.bootstrap.Bootstrap

core

log.origin.file.line

The line number of the file containing the source code which originated the log event.

type: integer

example: 42

extended

log.origin.file.name

The name of the file containing the source code which originated the log event.

Note that this field is not meant to capture the log file. The correct field to capture the log file is log.file.path.

type: keyword

example: Bootstrap.java

extended

log.origin.function

The name of the function or method which originated the log event.

type: keyword

example: init

extended

log.original

Deprecated for removal in next major version release. This field is superseded by event.original.

This is the original log message and contains the full log message before splitting it up in multiple parts.

In contrast to the message field which can contain an extracted part of the log message, this field contains the original, full log message. It can have already some modifications applied like encoding or new lines removed to clean up the log message.

This field is not indexed and doc_values are disabled so it can’t be queried but the value can be retrieved from _source.

type: keyword

example: Sep 19 08:26:10 localhost My log

core

log.syslog

The Syslog metadata of the event, if the event was transmitted via Syslog. Please see RFCs 5424 or 3164.

type: object

extended

log.syslog.facility.code

The Syslog numeric facility of the log event, if available.

According to RFCs 5424 and 3164, this value should be an integer between 0 and 23.

type: long

example: 23

extended

log.syslog.facility.name

The Syslog text-based facility of the log event, if available.

type: keyword

example: local7

extended

log.syslog.priority

Syslog numeric priority of the event, if available.

According to RFCs 5424 and 3164, the priority is 8 * facility + severity. This number is therefore expected to contain a value between 0 and 191.

type: long

example: 135

extended

log.syslog.severity.code

The Syslog numeric severity of the log event, if available.

If the event source publishing via Syslog provides a different numeric severity value (e.g. firewall, IDS), your source’s numeric severity should go to event.severity. If the event source does not specify a distinct severity, you can optionally copy the Syslog severity to event.severity.

type: long

example: 3

extended

log.syslog.severity.name

The Syslog numeric severity of the log event, if available.

If the event source publishing via Syslog provides a different severity value (e.g. firewall, IDS), your source’s text severity should go to log.level. If the event source does not specify a distinct severity, you can optionally copy the Syslog severity to log.level.

type: keyword

example: Error

extended

Metrics Fields

Fields for metrics generated from logs.

Subkeys of this field should generally follow Prometheus naming convention, paying particular attention to the suffix.

"A metric name should have a suffix indicating the unit, in plural form." e.g. ttfb.seconds, heap.bytes, size.megabytes etc.

https://prometheus.io/docs/practices/naming/

Field Description Level

metrics.puppet.changes.total

total number of changes performed in a puppet run

type: long

custom

metrics.puppet.resources.changed.total

number of resources puppet changed

type: long

custom

metrics.puppet.resources.corrective_change.total

number of resources puppet corrective changes made

type: long

custom

metrics.puppet.resources.failed.total

number of resources puppet failed to manage

type: long

custom

metrics.puppet.resources.failed_to_restart.total

number of resources puppet failed to restart

type: long

custom

metrics.puppet.resources.out_of_sync.total

number of resources puppet found to be out_of_sync

type: long

custom

metrics.puppet.resources.restarted.total

number of resources puppet restarted

type: long

custom

metrics.puppet.resources.scheduled.total

number of resources puppet scheduled

type: long

custom

metrics.puppet.resources.skipped.total

number of resources puppet skipped

type: long

custom

metrics.puppet.resources.total

Total number of resources affected

type: long

custom

metrics.puppet.runtime

total seconds spent executing a specific resource

type: object

example: {"schedule": {"seconds": 0.25}}

custom

metrics.ttfb.seconds

Time to first byte in seconds.

type: half_float

custom

Network Fields

The network is defined as the communication path over which a host or network event happens.

The network.* fields should be populated with details about the network activity associated with an event.

Field Description Level

network.application

A name given to an application level protocol. This can be arbitrarily assigned for things like microservices, but also apply to things like skype, icq, facebook, twitter. This would be used in situations where the vendor or service can be decoded such as from the source/dest IP owners, ports, or wire format.

The field value must be normalized to lowercase for querying. See the documentation section "Implementing ECS".

type: keyword

example: aim

extended

network.bytes

Total bytes transferred in both directions.

If source.bytes and destination.bytes are known, network.bytes is their sum.

type: long

example: 368

core

network.community_id

A hash of source and destination IPs and ports, as well as the protocol used in a communication. This is a tool-agnostic standard to identify flows.

Learn more at https://github.com/corelight/community-id-spec.

type: keyword

example: 1:hO+sN4H+MG5MY/8hIrXPqc4ZQz0=

extended

network.direction

Direction of the network traffic.

Recommended values are:

* ingress

* egress

* inbound

* outbound

* internal

* external

* unknown

When mapping events from a host-based monitoring context, populate this field from the host’s point of view, using the values "ingress" or "egress".

When mapping events from a network or perimeter-based monitoring context, populate this field from the point of view of the network perimeter, using the values "inbound", "outbound", "internal" or "external".

Note that "internal" is not crossing perimeter boundaries, and is meant to describe communication between two hosts within the perimeter. Note also that "external" is meant to describe traffic between two hosts that are external to the perimeter. This could for example be useful for ISPs or VPN service providers.

type: keyword

example: inbound

core

network.forwarded_ip

Host IP address when the source IP address is the proxy.

type: ip

example: 192.1.1.2

core

network.iana_number

IANA Protocol Number (https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml). Standardized list of protocols. This aligns well with NetFlow and sFlow related logs which use the IANA Protocol Number.

type: keyword

example: 6

extended

network.inner

Network.inner fields are added in addition to network.vlan fields to describe the innermost VLAN when q-in-q VLAN tagging is present. Allowed fields include vlan.id and vlan.name. Inner vlan fields are typically used when sending traffic with multiple 802.1q encapsulations to a network sensor (e.g. Zeek, Wireshark.)

type: object

extended

network.name

Name given by operators to sections of their network.

type: keyword

example: Guest Wifi

extended

network.packets

Total packets transferred in both directions.

If source.packets and destination.packets are known, network.packets is their sum.

type: long

example: 24

core

network.protocol

L7 Network protocol name. ex. http, lumberjack, transport protocol.

The field value must be normalized to lowercase for querying. See the documentation section "Implementing ECS".

type: keyword

example: http

core

network.tcp_flags

Array of TCP Flags found in the packet.

The field value must be normalized to lowercase for querying.

type: keyword

Note: this field should contain an array of values.

example: ["ack", "fin"]

custom

network.transport

Same as network.iana_number, but instead using the Keyword name of the transport layer (udp, tcp, ipv6-icmp, etc.)

The field value must be normalized to lowercase for querying. See the documentation section "Implementing ECS".

type: keyword

example: tcp

core

network.type

In the OSI Model this would be the Network Layer. ipv4, ipv6, ipsec, pim, etc

The field value must be normalized to lowercase for querying. See the documentation section "Implementing ECS".

type: keyword

example: ipv4

core
Location Field Set Description

network.inner.vlan.*

vlan

Fields to describe observed VLAN information.

network.vlan.*

vlan

Fields to describe observed VLAN information.

Normalized Fields

Fields generated and normalized by the logging pipeline, but normally hidden from view in the Kibana Discover app and saved search results in the Dashboard app.

This field is intended to contain fields that generally duplicate other fields and clutter the view.

This field is hidden superficially by an entry in the index pattern source filters.

Field Description Level

normalized.dropped.field_type_mismatch

Array of dropped field names due to a type conflict.

Values in this field are the equivalent of those in error type mapper_parsing_exception.

type: keyword

custom

normalized.dropped.no_such_field

Array of dropped field names due to no available field definition.

Values in this field would normally be dropped silently by ElasticSearch.

type: keyword

custom

normalized.message

The message field downcased, truncated to 256 characters, and indexed in an aggregatable form.

The downcasing and truncation is handled by the Logstash pipeline.

Ideally, this should be a multi-field of the message field, but lacking a truncation feature, this behavior is currently impossible. For more information, see: https://github.com/elastic/elasticsearch/issues/60329

type: keyword

example: Hello World

custom

Observer Fields

An observer is defined as a special network, security, or application device used to detect, observe, or create network, security, or application-related events and metrics.

This could be a custom hardware appliance or a server that has been configured to run special network, security, or application software. Examples include firewalls, web proxies, intrusion detection/prevention systems, network monitoring sensors, web application firewalls, data loss prevention systems, and APM servers. The observer.* fields shall be populated with details of the system, if any, that detects, observes and/or creates a network, security, or application event or metric. Message queues and ETL components used in processing events or metrics are not considered observers in ECS.

Field Description Level

observer.egress

Observer.egress holds information like interface number and name, vlan, and zone information to classify egress traffic. Single armed monitoring such as a network sensor on a span port should only use observer.ingress to categorize traffic.

type: object

extended

observer.egress.zone

Network zone of outbound traffic as reported by the observer to categorize the destination area of egress traffic, e.g. Internal, External, DMZ, HR, Legal, etc.

type: keyword

example: Public_Internet

extended

observer.hostname

Hostname of the observer.

type: keyword

core

observer.ingress

Observer.ingress holds information like interface number and name, vlan, and zone information to classify ingress traffic. Single armed monitoring such as a network sensor on a span port should only use observer.ingress to categorize traffic.

type: object

extended

observer.ingress.zone

Network zone of incoming traffic as reported by the observer to categorize the source area of ingress traffic. e.g. internal, External, DMZ, HR, Legal, etc.

type: keyword

example: DMZ

extended

observer.ip

IP addresses of the observer.

type: ip

Note: this field should contain an array of values.

core

observer.mac

MAC addresses of the observer.

The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen.

type: keyword

Note: this field should contain an array of values.

example: ["00-00-5E-00-53-23", "00-00-5E-00-53-24"]

core

observer.name

Custom name of the observer.

This is a name that can be given to an observer. This can be helpful for example if multiple firewalls of the same model are used in an organization.

If no custom name is needed, the field can be left empty.

type: keyword

example: 1_proxySG

extended

observer.product

The product name of the observer.

type: keyword

example: s200

extended

observer.serial_number

Observer serial number.

type: keyword

extended

observer.type

The type of the observer the data is coming from.

There is no predefined list of observer types. Some examples are forwarder, firewall, ids, ips, proxy, poller, sensor, APM server.

type: keyword

example: firewall

core

observer.vendor

Vendor name of the observer.

type: keyword

example: Symantec

core

observer.version

Observer version.

type: keyword

core
Location Field Set Description

observer.egress.interface.*

interface

Fields to describe observer interface information.

observer.egress.vlan.*

vlan

Fields to describe observed VLAN information.

observer.geo.*

geo

Fields describing a location.

observer.ingress.interface.*

interface

Fields to describe observer interface information.

observer.ingress.vlan.*

vlan

Fields to describe observed VLAN information.

observer.os.*

os

OS fields contain information about the operating system.

Orchestrator Fields

Fields that describe the resources which container orchestrators manage or act upon.

Field Description Level

orchestrator.api_version

API version being used to carry out the action

type: keyword

example: v1beta1

extended

orchestrator.cluster.name

Name of the cluster.

type: keyword

extended

orchestrator.cluster.url

URL of the API used to manage the cluster.

type: keyword

extended

orchestrator.cluster.version

The version of the cluster.

type: keyword

extended

orchestrator.namespace

Namespace in which the action is taking place.

type: keyword

example: kube-system

extended

orchestrator.organization

Organization affected by the event (for multi-tenant orchestrator setups).

type: keyword

example: elastic

extended

orchestrator.resource.annotation

The list of annotations added to the resource.

type: keyword

Note: this field should contain an array of values.

example: ['key1:value1', 'key2:value2', 'key3:value3']

extended

orchestrator.resource.id

Unique ID of the resource being acted upon.

type: keyword

extended

orchestrator.resource.label

The list of labels added to the resource.

type: keyword

Note: this field should contain an array of values.

example: ['key1:value1', 'key2:value2', 'key3:value3']

extended

orchestrator.resource.name

Name of the resource being acted upon.

type: keyword

example: test-pod-cdcws

extended

orchestrator.resource.type

Type of resource being acted upon.

type: keyword

example: service

extended

orchestrator.type

Orchestrator cluster type (e.g. kubernetes, nomad or cloudfoundry).

type: keyword

example: kubernetes

extended

Operating System Fields

The OS fields contain information about the operating system.

Field Description Level

os.family

OS family (such as redhat, debian, freebsd, windows).

type: keyword

example: debian

extended

os.full

Operating system name, including the version or code name.

type: keyword

Multi-fields:

* os.full.text (type: text)

example: Mac OS Mojave

extended

os.kernel

Operating system kernel version as a raw string.

type: keyword

example: 4.4.0-112-generic

extended

os.name

Operating system name, without the version.

type: keyword

Multi-fields:

* os.name.text (type: text)

example: Mac OS X

extended

os.platform

Operating system platform (such centos, ubuntu, windows).

type: keyword

example: darwin

extended

os.type

Use the os.type field to categorize the operating system into one of the broad commercial families.

One of these following values should be used (lowercase): linux, macos, unix, windows.

If the OS you’re dealing with is not in the list, the field should not be populated. Please let us know by opening an issue with ECS, to propose its addition.

type: keyword

example: macos

extended

os.version

Operating system version as a raw string.

type: keyword

example: 10.14.1

extended

The os fields are expected to be nested at:

  • host.os

  • observer.os

  • user_agent.os

Note also that the os fields are not expected to be used directly at the root of the events.

Package Fields

These fields contain information about an installed software package. It contains general information about a package, such as name, version or size. It also contains installation details, such as time or location.

Field Description Level

package.architecture

Package architecture.

type: keyword

example: x86_64

extended

package.build_version

Additional information about the build version of the installed package.

For example use the commit SHA of a non-released package.

type: keyword

example: 36f4f7e89dd61b0988b12ee000b98966867710cd

extended

package.checksum

Checksum of the installed package for verification.

type: keyword

example: 68b329da9893e34099c7d8ad5cb9c940

extended

package.description

Description of the package.

type: keyword

example: Open source programming language to build simple/reliable/efficient software.

extended

package.install_scope

Indicating how the package was installed, e.g. user-local, global.

type: keyword

example: global

extended

package.installed

Time when package was installed.

type: date

extended

package.license

License under which the package was released.

Use a short name, e.g. the license identifier from SPDX License List where possible (https://spdx.org/licenses/).

type: keyword

example: Apache License 2.0

extended

package.name

Package name

type: keyword

example: go

extended

package.path

Path where the package is installed.

type: keyword

example: /usr/local/Cellar/go/1.12.9/

extended

package.reference

Home page or reference URL of the software in this package, if available.

type: keyword

example: https://golang.org

extended

package.size

Package size in bytes.

type: long

example: 62231

extended

package.type

Type of package.

This should contain the package file type, rather than the package manager name. Examples: rpm, dpkg, brew, npm, gem, nupkg, jar.

type: keyword

example: rpm

extended

package.version

Package version

type: keyword

example: 1.12.9

extended

Process Fields

These fields contain information about a process.

These fields can help you correlate metrics information with a process id/name from a log message. The process.pid often stays in the metric itself and is copied to the global field for correlation.

Field Description Level

process.args

Array of process arguments, starting with the absolute path to the executable.

May be filtered to protect sensitive information.

type: keyword

Note: this field should contain an array of values.

example: ["/usr/bin/ssh", "-l", "user", "10.0.0.16"]

extended

process.args_count

Length of the process.args array.

This field can be useful for querying or performing bucket analysis on how many arguments were provided to start a process. More arguments may be an indication of suspicious activity.

type: long

example: 4

extended

process.command_line

Full command line that started the process, including the absolute path to the executable, and all arguments.

Some arguments may be filtered to protect sensitive information.

type: keyword

Multi-fields:

* process.command_line.text (type: text)

example: /usr/bin/ssh -l user 10.0.0.16

extended

process.entity_id

Unique identifier for the process.

The implementation of this is specified by the data source, but some examples of what could be used here are a process-generated UUID, Sysmon Process GUIDs, or a hash of some uniquely identifying components of a process.

Constructing a globally unique identifier is a common practice to mitigate PID reuse as well as to identify a specific process over time, across multiple monitored hosts.

type: keyword

example: c2c455d9f99375d

extended

process.executable

Absolute path to the process executable.

type: keyword

Multi-fields:

* process.executable.text (type: text)

example: /usr/bin/ssh

extended

process.exit_code

The exit code of the process, if this is a termination event.

The field should be absent if there is no exit code for the event (e.g. process start).

type: long

example: 137

extended

process.name

Process name.

Sometimes called program name or similar.

type: keyword

Multi-fields:

* process.name.text (type: text)

example: ssh

extended

process.pgid

Identifier of the group of processes the process belongs to.

type: long

extended

process.pid

Process id.

type: long

example: 4242

core

process.ppid

Parent process' pid.

type: long

example: 4241

extended

process.start

The time the process started.

type: date

example: 2016-05-23T08:05:34.853Z

extended

process.thread.id

Thread ID.

type: long

example: 4242

extended

process.thread.name

Thread name.

type: keyword

example: thread-0

extended

process.title

Process title.

The proctitle, some times the same as process name. Can also be different: for example a browser setting its title to the web page currently opened.

type: keyword

Multi-fields:

* process.title.text (type: text)

extended

process.uptime

Seconds the process has been up.

type: long

example: 1325

extended

process.working_directory

The working directory of the process.

type: keyword

Multi-fields:

* process.working_directory.text (type: text)

example: /home/alice

extended

The process fields are expected to be nested at:

  • process.parent

Note also that the process fields may be used directly at the root of the events.

Location Field Set Description

process.code_signature.*

code_signature

These fields contain information about binary code signatures.

process.elf.*

elf

beta:[ This field reuse is beta and subject to change.]

These fields contain Linux Executable Linkable Format (ELF) metadata.

process.hash.*

hash

Hashes, usually file hashes.

process.parent.*

process

Information about the parent process.

process.pe.*

pe

These fields contain Windows Portable Executable (PE) metadata.

Rule Fields

Rule fields are used to capture the specifics of any observer or agent rules that generate alerts or other notable events.

Examples of data sources that would populate the rule fields include: network admission control platforms, network or host IDS/IPS, network firewalls, web application firewalls, url filters, endpoint detection and response (EDR) systems, etc.

Field Description Level

rule.author

Name, organization, or pseudonym of the author or authors who created the rule used to generate this event.

type: keyword

Note: this field should contain an array of values.

example: ["Star-Lord"]

extended

rule.category

A categorization value keyword used by the entity using the rule for detection of this event.

type: keyword

example: Attempted Information Leak

extended

rule.description

The description of the rule generating the event.

type: keyword

example: Block requests to public DNS over HTTPS / TLS protocols

extended

rule.id

A rule ID that is unique within the scope of an agent, observer, or other entity using the rule for detection of this event.

type: keyword

example: 101

extended

rule.license

Name of the license under which the rule used to generate this event is made available.

type: keyword

example: Apache 2.0

extended

rule.name

The name of the rule or signature generating the event.

type: keyword

example: BLOCK_DNS_over_TLS

extended

rule.reference

Reference URL to additional information about the rule used to generate this event.

The URL can point to the vendor’s documentation about the rule. If that’s not available, it can also be a link to a more general page describing this type of alert.

type: keyword

example: https://en.wikipedia.org/wiki/DNS_over_TLS

extended

rule.ruleset

Name of the ruleset, policy, group, or parent category in which the rule used to generate this event is a member.

type: keyword

example: Standard_Protocol_Filters

extended

rule.uuid

A rule ID that is unique within the scope of a set or group of agents, observers, or other entities using the rule for detection of this event.

type: keyword

example: 1100110011

extended

rule.version

The version / revision of the rule being used for analysis.

type: keyword

example: 1.1

extended

Server Fields

A Server is defined as the responder in a network connection for events regarding sessions, connections, or bidirectional flow records.

For TCP events, the server is the receiver of the initial SYN packet(s) of the TCP connection. For other protocols, the server is generally the responder in the network transaction. Some systems actually use the term "responder" to refer the server in TCP connections. The server fields describe details about the system acting as the server in the network event. Server fields are usually populated in conjunction with client fields. Server fields are generally not populated for packet-level events.

Client / server representations can add semantic context to an exchange, which is helpful to visualize the data in certain situations. If your context falls in that category, you should still ensure that source and destination are filled appropriately.

Field Description Level

server.address

Some event server addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field.

Then it should be duplicated to .ip or .domain, depending on which one it is.

type: keyword

extended

server.bytes

Bytes sent from the server to the client.

type: long

example: 184

core

server.domain

Server domain.

type: keyword

core

server.ip

IP address of the server (IPv4 or IPv6).

type: ip

core

server.mac

MAC address of the server.

The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen.

type: keyword

example: 00-00-5E-00-53-23

core

server.nat.ip

Translated ip of destination based NAT sessions (e.g. internet to private DMZ)

Typically used with load balancers, firewalls, or routers.

type: ip

extended

server.nat.port

Translated port of destination based NAT sessions (e.g. internet to private DMZ)

Typically used with load balancers, firewalls, or routers.

type: long

extended

server.packets

Packets sent from the server to the client.

type: long

example: 12

core

server.port

Port of the server.

type: long

core

server.registered_domain

The highest registered server domain, stripped of the subdomain.

For example, the registered domain for "foo.example.com" is "example.com".

This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last two labels will not work well for TLDs such as "co.uk".

type: keyword

example: example.com

extended

server.subdomain

The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain. In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain.

For example the subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.

type: keyword

example: east

extended

server.top_level_domain

The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com".

This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".

type: keyword

example: co.uk

extended
Location Field Set Description

server.as.*

as

Fields describing an Autonomous System (Internet routing prefix).

server.geo.*

geo

Fields describing a location.

server.user.*

user

Fields to describe the user relevant to the event.

Service Fields

The service fields describe the service for or from which the data was collected.

These fields help you find and correlate logs for a specific service and version.

Field Description Level

service.ephemeral_id

Ephemeral identifier of this service (if one exists).

This id normally changes across restarts, but service.id does not.

type: keyword

example: 8a4f500f

extended

service.id

Unique identifier of the running service. If the service is comprised of many nodes, the service.id should be the same for all nodes.

This id should uniquely identify the service. This makes it possible to correlate logs and metrics for one specific service, no matter which particular node emitted the event.

Note that if you need to see the events from one specific host of the service, you should filter on that host.name or host.id instead.

type: keyword

example: d37e5ebfe0ae6c4972dbe9f0174a1637bb8247f6

core

service.name

Name of the service data is collected from.

The name of the service is normally user given. This allows for distributed services that run on multiple hosts to correlate the related instances based on the name.

In the case of Elasticsearch the service.name could contain the cluster name. For Beats the service.name is by default a copy of the service.type field if no name is specified.

type: keyword

example: elasticsearch-metrics

core

service.node.name

Name of a service node.

This allows for two nodes of the same service running on the same host to be differentiated. Therefore, service.node.name should typically be unique across nodes of a given service.

In the case of Elasticsearch, the service.node.name could contain the unique node name within the Elasticsearch cluster. In cases where the service doesn’t have the concept of a node name, the host name or container name can be used to distinguish running instances that make up this service. If those do not provide uniqueness (e.g. multiple instances of the service running on the same host) - the node name can be manually set.

type: keyword

example: instance-0000000016

extended

service.state

Current state of the service.

type: keyword

core

service.type

The type of the service data is collected from.

The type can be used to group and correlate logs and metrics from one service type.

Example: If logs or metrics are collected from Elasticsearch, service.type would be elasticsearch.

type: keyword

example: elasticsearch

core

service.version

Version of the service the data was collected from.

This allows to look at a data set only for a specific version of a service.

type: keyword

example: 3.2.4

core

Source Fields

Source fields capture details about the sender of a network exchange/packet. These fields are populated from a network event, packet, or other event containing details of a network transaction.

Source fields are usually populated in conjunction with destination fields. The source and destination fields are considered the baseline and should always be filled if an event contains source and destination details from a network transaction. If the event also contains identification of the client and server roles, then the client and server fields should also be populated.

Field Description Level

source.address

Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field.

Then it should be duplicated to .ip or .domain, depending on which one it is.

type: keyword

extended

source.bytes

Bytes sent from the source to the destination.

type: long

example: 184

core

source.domain

Source domain.

type: keyword

core

source.ip

IP address of the source (IPv4 or IPv6).

type: ip

core

source.mac

MAC address of the source.

The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen.

type: keyword

example: 00-00-5E-00-53-23

core

source.nat.ip

Translated ip of source based NAT sessions (e.g. internal client to internet)

Typically connections traversing load balancers, firewalls, or routers.

type: ip

extended

source.nat.port

Translated port of source based NAT sessions. (e.g. internal client to internet)

Typically used with load balancers, firewalls, or routers.

type: long

extended

source.packets

Packets sent from the source to the destination.

type: long

example: 12

core

source.port

Port of the source.

type: long

core

source.registered_domain

The highest registered source domain, stripped of the subdomain.

For example, the registered domain for "foo.example.com" is "example.com".

This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last two labels will not work well for TLDs such as "co.uk".

type: keyword

example: example.com

extended

source.subdomain

The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain. In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain.

For example the subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.

type: keyword

example: east

extended

source.top_level_domain

The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com".

This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".

type: keyword

example: co.uk

extended
Location Field Set Description

source.as.*

as

Fields describing an Autonomous System (Internet routing prefix).

source.geo.*

geo

Fields describing a location.

source.user.*

user

Fields to describe the user relevant to the event.

TLS Fields

Fields related to a TLS connection. These fields focus on the TLS protocol itself and intentionally avoids in-depth analysis of the related x.509 certificate files.

Field Description Level

tls.cipher

String indicating the cipher used during the current connection.

type: keyword

example: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

extended

tls.client.certificate

PEM-encoded stand-alone certificate offered by the client. This is usually mutually-exclusive of client.certificate_chain since this value also exists in that list.

type: keyword

example: MII...

extended

tls.client.certificate_chain

Array of PEM-encoded certificates that make up the certificate chain offered by the client. This is usually mutually-exclusive of client.certificate since that value should be the first certificate in the chain.

type: keyword

Note: this field should contain an array of values.

example: ["MII...", "MII..."]

extended

tls.client.hash.md5

Certificate fingerprint using the MD5 digest of DER-encoded version of certificate offered by the client. For consistency with other hash values, this value should be formatted as an uppercase hash.

type: keyword

example: 0F76C7F2C55BFD7D8E8B8F4BFBF0C9EC

extended

tls.client.hash.sha1

Certificate fingerprint using the SHA1 digest of DER-encoded version of certificate offered by the client. For consistency with other hash values, this value should be formatted as an uppercase hash.

type: keyword

example: 9E393D93138888D288266C2D915214D1D1CCEB2A

extended

tls.client.hash.sha256

Certificate fingerprint using the SHA256 digest of DER-encoded version of certificate offered by the client. For consistency with other hash values, this value should be formatted as an uppercase hash.

type: keyword

example: 0687F666A054EF17A08E2F2162EAB4CBC0D265E1D7875BE74BF3C712CA92DAF0

extended

tls.client.issuer

Distinguished name of subject of the issuer of the x.509 certificate presented by the client.

type: keyword

example: CN=Example Root CA, OU=Infrastructure Team, DC=example, DC=com

extended

tls.client.ja3

A hash that identifies clients based on how they perform an SSL/TLS handshake.

type: keyword

example: d4e5b18d6b55c71272893221c96ba240

extended

tls.client.not_after

Date/Time indicating when client certificate is no longer considered valid.

type: date

example: 2021-01-01T00:00:00.000Z

extended

tls.client.not_before

Date/Time indicating when client certificate is first considered valid.

type: date

example: 1970-01-01T00:00:00.000Z

extended

tls.client.server_name

Also called an SNI, this tells the server which hostname to which the client is attempting to connect to. When this value is available, it should get copied to destination.domain.

type: keyword

example: www.elastic.co

extended

tls.client.subject

Distinguished name of subject of the x.509 certificate presented by the client.

type: keyword

example: CN=myclient, OU=Documentation Team, DC=example, DC=com

extended

tls.client.supported_ciphers

Array of ciphers offered by the client during the client hello.

type: keyword

Note: this field should contain an array of values.

example: ["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "..."]

extended

tls.curve

String indicating the curve used for the given cipher, when applicable.

type: keyword

example: secp256r1

extended

tls.established

Boolean flag indicating if the TLS negotiation was successful and transitioned to an encrypted tunnel.

type: boolean

extended

tls.next_protocol

String indicating the protocol being tunneled. Per the values in the IANA registry (https://www.iana.org/assignments/tls-extensiontype-values/tls-extensiontype-values.xhtml#alpn-protocol-ids), this string should be lower case.

type: keyword

example: http/1.1

extended

tls.resumed

Boolean flag indicating if this TLS connection was resumed from an existing TLS negotiation.

type: boolean

extended

tls.server.certificate

PEM-encoded stand-alone certificate offered by the server. This is usually mutually-exclusive of server.certificate_chain since this value also exists in that list.

type: keyword

example: MII...

extended

tls.server.certificate_chain

Array of PEM-encoded certificates that make up the certificate chain offered by the server. This is usually mutually-exclusive of server.certificate since that value should be the first certificate in the chain.

type: keyword

Note: this field should contain an array of values.

example: ["MII...", "MII..."]

extended

tls.server.hash.md5

Certificate fingerprint using the MD5 digest of DER-encoded version of certificate offered by the server. For consistency with other hash values, this value should be formatted as an uppercase hash.

type: keyword

example: 0F76C7F2C55BFD7D8E8B8F4BFBF0C9EC

extended

tls.server.hash.sha1

Certificate fingerprint using the SHA1 digest of DER-encoded version of certificate offered by the server. For consistency with other hash values, this value should be formatted as an uppercase hash.

type: keyword

example: 9E393D93138888D288266C2D915214D1D1CCEB2A

extended

tls.server.hash.sha256

Certificate fingerprint using the SHA256 digest of DER-encoded version of certificate offered by the server. For consistency with other hash values, this value should be formatted as an uppercase hash.

type: keyword

example: 0687F666A054EF17A08E2F2162EAB4CBC0D265E1D7875BE74BF3C712CA92DAF0

extended

tls.server.issuer

Subject of the issuer of the x.509 certificate presented by the server.

type: keyword

example: CN=Example Root CA, OU=Infrastructure Team, DC=example, DC=com

extended

tls.server.ja3s

A hash that identifies servers based on how they perform an SSL/TLS handshake.

type: keyword

example: 394441ab65754e2207b1e1b457b3641d

extended

tls.server.not_after

Timestamp indicating when server certificate is no longer considered valid.

type: date

example: 2021-01-01T00:00:00.000Z

extended

tls.server.not_before

Timestamp indicating when server certificate is first considered valid.

type: date

example: 1970-01-01T00:00:00.000Z

extended

tls.server.subject

Subject of the x.509 certificate presented by the server.

type: keyword

example: CN=www.example.com, OU=Infrastructure Team, DC=example, DC=com

extended

tls.version

Numeric part of the version parsed from the original string.

type: keyword

example: 1.2

extended

tls.version_protocol

Normalized lowercase protocol name parsed from original string.

type: keyword

example: tls

extended
Location Field Set Description

tls.client.x509.*

x509

These fields contain x509 certificate metadata.

tls.server.x509.*

x509

These fields contain x509 certificate metadata.

Tracing Fields

Distributed tracing makes it possible to analyze performance throughout a microservice architecture all in one view. This is accomplished by tracing all of the requests - from the initial web request in the front-end service - to queries made through multiple back-end services.

Unlike most field sets in ECS, the tracing fields are not nested under the field set name. In other words, the correct field name is trace.id, not tracing.trace.id, and so on.

Field Description Level

span.id

Unique identifier of the span within the scope of its trace.

A span represents an operation within a transaction, such as a request to another service, or a database query.

type: keyword

example: 3ff9a8981b7ccd5a

extended

trace.id

Unique identifier of the trace.

A trace groups multiple events like transactions that belong together. For example, a user request handled by multiple inter-connected services.

type: keyword

example: 4bf92f3577b34da6a3ce929d0e0e4736

extended

transaction.id

Unique identifier of the transaction within the scope of its trace.

A transaction is the highest level of work measured within a service, such as a request to a server.

type: keyword

example: 00f067aa0ba902b7

extended

URL Fields

URL fields provide support for complete or partial URLs, and supports the breaking down into scheme, domain, path, and so on.

Field Description Level

url.domain

Domain of the url, such as "www.elastic.co".

In some cases a URL may refer to an IP and/or port directly, without a domain name. In this case, the IP address would go to the domain field.

If the URL contains a literal IPv6 address enclosed by [ and ] (IETF RFC 2732), the [ and ] characters should also be captured in the domain field.

type: keyword

example: www.elastic.co

extended

url.extension

The field contains the file extension from the original request url, excluding the leading dot.

The file extension is only set if it exists, as not every url has a file extension.

The leading period must not be included. For example, the value must be "png", not ".png".

Note that when the file name has multiple extensions (example.tar.gz), only the last one should be captured ("gz", not "tar.gz").

type: keyword

example: png

extended

url.fragment

Portion of the url after the #, such as "top".

The # is not part of the fragment.

type: keyword

extended

url.full

If full URLs are important to your use case, they should be stored in url.full, whether this field is reconstructed or present in the event source.

type: keyword

Multi-fields:

* url.full.text (type: text)

example: https://www.elastic.co:443/search?q=elasticsearch#top

extended

url.original

Unmodified original url as seen in the event source.

Note that in network monitoring, the observed URL may be a full URL, whereas in access logs, the URL is often just represented as a path.

This field is meant to represent the URL as it was observed, complete or not.

type: keyword

Multi-fields:

* url.original.text (type: text)

example: https://www.elastic.co:443/search?q=elasticsearch#top or /search?q=elasticsearch

extended

url.password

Password of the request.

type: keyword

extended

url.path

Path of the request, such as "/search".

type: keyword

extended

url.port

Port of the request, such as 443.

type: long

example: 443

extended

url.query

The query field describes the query string of the request, such as "q=elasticsearch".

The ? is excluded from the query string. If a URL contains no ?, there is no query field. If there is a ? but no query, the query field exists with an empty string. The exists query can be used to differentiate between the two cases.

type: keyword

extended

url.registered_domain

The highest registered url domain, stripped of the subdomain.

For example, the registered domain for "foo.example.com" is "example.com".

This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last two labels will not work well for TLDs such as "co.uk".

type: keyword

example: example.com

extended

url.scheme

Scheme of the request, such as "https".

Note: The : is not part of the scheme.

type: keyword

example: https

extended

url.subdomain

The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain. In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain.

For example the subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.

type: keyword

example: east

extended

url.top_level_domain

The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com".

This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".

type: keyword

example: co.uk

extended

url.username

Username of the request.

type: keyword

extended

The url fields are expected to be nested at:

  • threat.enrichments.indicator.url

  • threat.indicator.url

Note also that the url fields may be used directly at the root of the events.

User Fields

The user fields describe information about the user that is relevant to the event.

Fields can have one entry or multiple entries. If a user has more than one id, provide an array that includes all of them.

Field Description Level

user.domain

Name of the directory the user is a member of.

For example, an LDAP or Active Directory domain name.

type: keyword

extended

user.email

User email address.

type: keyword

extended

user.extra

Additional information in key/value pairs.

For additional information provided by the authenticator to events.

Note: Not searchable or filterable.

type: object

example: {"authenticator": "ldap", "cipher": "argon2"}

custom

user.full_name

User’s full name, if available.

type: keyword

Multi-fields:

* user.full_name.text (type: text)

example: Albert Einstein

extended

user.hash

Unique user hash to correlate information for a user in anonymized form.

Useful if user.id or user.name contain confidential information and cannot be used.

type: keyword

extended

user.id

Unique identifier of the user.

type: keyword

core

user.name

Short name or login of the user.

type: keyword

Multi-fields:

* user.name.text (type: text)

example: albert

core

user.roles

Array of user roles at the time of the event.

type: keyword

Note: this field should contain an array of values.

example: ["kibana_admin", "reporting_user"]

extended

The user fields are expected to be nested at:

  • client.user

  • destination.user

  • host.user

  • server.user

  • source.user

  • user.changes

  • user.effective

  • user.target

Note also that the user fields may be used directly at the root of the events.

Location Field Set Description

user.changes.*

user

Captures changes made to a user.

user.effective.*

user

User whose privileges were assumed.

user.group.*

group

User’s group relevant to the event.

user.target.*

user

Targeted user of action taken.

For usage and examples of the user fields, please see the User Fields Usage and Examples section.

User Fields Usage and Examples

Here are the subjects covered in this page.

User fields can be present in any kind of event, without affecting the event’s categorization.

However when the event is about IAM (Identity and Account Management), it should be categorized as follows. In this section we’ll cover specifically event.category and event.type with regards to IAM activity. Make sure to read the Categorization section to see all allowed values, and read more about event.kind and event.outcome.

Note
 IAM activity is a bit particular in that events are expected to be assigned 2 event types. One of them indicates the type of activity (creation, deletion, change, etc.), and the other indicates whether a user or a group is the target of the management activity.

Many sections of the examples below are elided, in order to focus on the categorization of the events.

Creation of group "test-group":

```JSON { "event": { "kind": "event", "category": ["iam"], <1> "type": ["group", "creation"], <2> "outcome": "success" }, "group": { "name": "test-group", … }, "user": { … }, "related": { "user": [ … ] } } ``` <1> Category "iam" <2> Both relevant event types to a group creation

Adding "test-user" to "test-group":

```JSON { "event": { "kind": "event", "category": ["iam"], <1> "type": ["user", "change"], <2> "action": "user added to group", <3> "outcome": "success" }, "user": { … "target": { <4> "name": "test-user", "group": { "name": "test-group" } } }, "related": { "user": [ … ] } } ``` <1> Category "iam" <2> Both relevant event types to a user modification <3> event.action is not a categorization field, and has no mandated value. It can be populated based on source event details or by a pipeline, to ensure the event captures all subtleties of what’s happening. <4> How to use all possible user fields is detailed below.

Different systems use different values for user identifiers. Here are a few pointers to help normalize some simple cases.

  • When a system provides a composite value for the user name (e.g. DOMAINNAME\username), capture the domain name in user.domain and the user name (without the domain) in user.name.

  • When a system uses an email address as the main identifier, populate both user.id and user.email with it.

The user fields can be reused (or appear) in many places across ECS. This makes it possible to capture many users relevant to a single event.

Here’s the full list of places where the user fields can appear:

  • user.*

  • user.effective.*

  • user.target.*

  • user.changes.*

  • source.user.*

  • destination.user.*

  • client.user.*

  • server.user.*

  • host.user.* (deprecated)

Let’s go over the meaning of each.

The examples below will only populate user.name and sometimes user.id inside the various user nestings, for readability. However in implementations, unless otherwise noted, all user fields that can reasonably be populated in each location should be populated.

====== User fields at the Root of an Event

The user fields at the root of an event are used to capture the user performing the main action described by the event. This is especially important when there’s more than one user present on the event. user.* fields at the root of the event represent the user performing the action.

In many cases, events that only mention one user should populate the user fields at the root of the event, even if the user is not the one performing the action.

In cases where a purpose-specific user field such as url.username is populated, user.name should also be populated with the same user name.

{
  "url": { "username": "alice" }, <1>
  "user": { "name": "alice" }, <2>
  "related": { "user": ["alice"] }
}

  1. Purpose-specific username field

  2. Username copied to user.name to establish user.name as a reliable baseline.

====== Remote Logons

When users are crossing host boundaries, the users are captured at source.user and destination.user.

Examples of data sources where this is applicable:

  • Remote logons via ssh, kerberos

  • Firewalls observing network traffic

In order to align with ECS' design of having user at the root of the event as the user performing the action, all source.user fields should be copied to user at the root.

Here’s an example where user "alice" logs on to another host as user "deus":

{
  "user": {
    "name": "alice"
  },
  "source": {
    "user": {
      "name": "alice"
    },
    "ip": "10.42.42.42"
  },
  "destination": {
    "user": {
      "name": "deus"
    },
    "ip": "10.42.42.43"
  },
  "related": { "user": ["alice", "deus"] }
}

Whenever an event source populates the client and server fields in addition to source and destination, the user fields should be copied accordingly as well. You can review [ecs-mapping-network-events] to learn more about mapping network events.

====== Privilege Changes

The user.effective fields are relevant when there’s a privilege escalation or demotion and it’s possible to determine the user requesting/performing the escalation.

Use the user fields at the root to capture who is requesting the privilege change, and user.effective to capture the requested privilege level, whether or not the privilege change was successful.

Here are examples where this is applicable:

  • A user changing identity on a host.

    • Examples: sudo, su, Run as.

  • Running a program as a different user. Examples:

    • A trusted user runs a specific admin command as root via a mechanism such as the Posix setuid/setgid.

    • A service manager with administrator privileges starts child processes as limited users, for security purposes (e.g. root runs Apache HTTPD as user "apache")

In cases where the event source only gives information about the effective user and not who requested different privileges, the user fields at the root of the event should be used instead of user.effective.

Here’s an example of user "alice" running a command as root via sudo:

{
  "user": {
    "name": "alice",
    "id": "1001",
    "effective": {
      "name": "root",
      "id": "1"
    }
  },
  "related": { "user": ["alice", "root"] }
}

When it’s not possible (or it’s prohibitive) to determine which user is requesting different privilege levels, it’s acceptable to capture the effective user at the root of the event. Typically a privilege change event will already have happened, for example: bob "su" as root; and subsequent events will show the root user performing the actions.

====== Identity and Access Management

Whenever a user is performing an action that affects another user — typically in IAM scenarios — the user affected by the action is captured at user.target. The user performing the IAM activity is captured at the root of the event.

Examples of IAM activity include:

  • user-a creates or deletes user-b

  • user-a modifies user-b

In the create/delete scenarios, there’s either no prior state (user creation) or no post state (user deletion). In these cases, only user at the root and user.target must be populated.

Example where "root" creates user "bob":

{
  "user": {
    "name": "root",
    "id": "1",
    "target": {
      "name": "bob",
      "id": "1002",
      ...
    }
  }
  "related": { "user": ["bob", "root"] }
}

When there’s a change of state to an existing user, user.target must be used to capture the prior state of the user, and user.changes should list only the changes that were performed.

Example where "root" renames user "bob" to "bob.barker":

{
  "user": {
    "name": "root",
    "id": "1",
    "target": {
      "name": "bob",
      "id": "1002"
    },
    "changes": {
      "name": "bob.barker"
    }
  },
  "related": { "user": ["bob", "bob.barker", "root"] }
}

You’ll note in the example above that unmodified attributes like the user ID are not repeated under user.changes.*, since they didn’t change.

====== Combining IAM and Privilege Change

We’ve covered above how user.target and user.changes can be used at the same time. If privilege escalation is also present in the same IAM event, user.effective should of course be used as well.

Here’s the "rename" example from the IAM section above. In the following example, we know "alice" is escalating privileges to "root", in order to modify user "bob":

{
  "user": {
    "name": "alice",
    "id": "1001",
    "effective": {
      "name": "root",
      "id": "1"
    },
    "target": {
      "name": "bob",
      "id": "1002"
    },
    "changes": {
      "name": "bob.barker"
    }
  },
  "related": { "user": ["alice", "bob", "bob.barker", "root"] }
}

====== Subtleties around field reuse

Most cases of field reuse in ECS are reusing a field set inside a different field set. Two examples of this are:

  • reusing group in user, resulting in the user.group.* fields, or

  • reusing user in destination, resulting in the destination.user.* fields, which also include destination.user.group.*.

The user fields can also be reused within user as different names, representing the role of each relevant user. Examples are the user.target.* or user.effective.* fields.

However, it’s important to note that user fields reused within user are not carried around anywhere else. Let’s illustrate the various permutations of what’s valid and what is not.

Field Validity Notes

user.group.*

Valid

Normal reuse.

destination.user.group.*

Valid

The group reuse gets carried around when user is reused elsewhere. Populate only if relevant to the event.

user.target.group.*, user.effective.group.*, user.changes.group.*

Valid

The group reuse gets carried around even when user is reused within itself. Populate only if relevant to the event.

destination.user.target.*, destination.user.effective.*, destination.user.changes.*

Invalid

The user fields reused within user are not carried around anywhere else. The same rule applies when user is reused under source, client and server.

In all events in this page, we’ve populated the related.user fields.

Any event that has users in it should always populate the array field related.user with all usernames seen in the event; including event names that appear in custom fields. Note that this field is not a nesting of all user fields, it’s a flat array meant to contain user identifiers.

Taking the example from user.changes again, we can see that no matter the role of the each user (before/after privilege escalation, affected user, username after rename), they are all present in related.user:

{
  "user": {
    "name": "alice",
    "id": "1001",
    "effective": {
      "name": "root",
      "id": "1"
    },
    "target": {
      "name": "bob",
      "id": "1002"
    },
    "changes": {
      "name": "bob.barker"
    }
  },
  "related": { "user": ["alice", "root", "bob", "bob.barker"] }
}

Like the other fields in the related field set, related.user is meant to facilitate pivoting. For example, if you have a suspicion about user "bob.barker", searching for this name in related.user will give you all events related to this user, whether it’s the creation / rename of the user, or events where this user was active in a system.

For examples of mapping events from various sources, you can look at RFC 0007 in section Source Data.

As of ECS 1.8, host.user.* fields are deprecated and will be removed at the next major version of ECS.

User agent Fields

The user_agent fields normally come from a browser request.

They often show up in web service logs coming from the parsed user agent string.

Field Description Level

user_agent.device.name

Name of the device.

type: keyword

example: iPhone

extended

user_agent.name

Name of the user agent.

type: keyword

example: Safari

extended

user_agent.original

Unparsed user_agent string.

type: keyword

Multi-fields:

* user_agent.original.text (type: text)

example: Mozilla/5.0 (iPhone; CPU iPhone OS 12_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1

extended

user_agent.version

Version of the user agent.

type: keyword

example: 12.0

extended
Location Field Set Description

user_agent.os.*

os

OS fields contain information about the operating system.

ECS Categorization Fields

At a high level, ECS provides fields to classify events in two different ways: "Where it’s from" (e.g., event.module, event.dataset, agent.type, observer.type, etc.), and "What it is." The categorization fields hold the "What it is" information, independent of the source of the events.

ECS defines four categorization fields for this purpose, each of which falls under the event.* field set.

Categorization Fields

Note
 If your events don’t match any of these categorization values, you should leave the fields empty. This will ensure you can start populating the fields once the appropriate categorization values are published, in a later release.

Categorization Usage

Using the categorization fields contains examples combining the categorization fields to classify different types of events.

ECS Categorization Field: event.kind

This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy.

event.kind gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events.

The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not.

Allowed Values

alert

This value indicates an event such as an alert or notable event, triggered by a detection rule executing externally to the Elastic Stack.

event.kind:alert is often populated for events coming from firewalls, intrusion detection systems, endpoint detection and response systems, and so on.

This value is not used by Elastic solutions for alert documents that are created by rules executing within the Kibana alerting framework.

enrichment

The enrichment value indicates an event collected to provide additional context, often to other events.

An example is collecting indicators of compromise (IOCs) from a threat intelligence provider with the intent to use those values to enrich other events. The IOC events from the intelligence provider should be categorized as event.kind:enrichment.

event

This value is the most general and most common value for this field. It is used to represent events that indicate that something happened.

metric

This value is used to indicate that this event describes a numeric measurement taken at given point in time.

Examples include CPU utilization, memory usage, or device temperature.

Metric events are often collected on a predictable frequency, such as once every few seconds, or once a minute, but can also be used to describe ad-hoc numeric metric queries.

state

The state value is similar to metric, indicating that this event describes a measurement taken at given point in time, except that the measurement does not result in a numeric value, but rather one of a fixed set of categorical values that represent conditions or states.

Examples include periodic events reporting Elasticsearch cluster state (green/yellow/red), the state of a TCP connection (open, closed, fin_wait, etc.), the state of a host with respect to a software vulnerability (vulnerable, not vulnerable), and the state of a system regarding compliance with a regulatory standard (compliant, not compliant).

Note that an event that describes a change of state would not use event.kind:state, but instead would use event.kind:event since a state change fits the more general event definition of something that happened.

State events are often collected on a predictable frequency, such as once every few seconds, once a minute, once an hour, or once a day, but can also be used to describe ad-hoc state queries.

pipeline_error

This value indicates that an error occurred during the ingestion of this event, and that event data may be missing, inconsistent, or incorrect. event.kind:pipeline_error is often associated with parsing errors.

signal

This value is used by Elastic solutions (e.g., Security, Observability) for alert documents that are created by rules executing within the Kibana alerting framework.

Usage of this value is reserved, and data ingestion pipelines must not populate event.kind with the value "signal".

ECS Categorization Field: event.category

This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy.

event.category represents the "big buckets" of ECS categories. For example, filtering on event.category:process yields all events relating to process activity. This field is closely related to event.type, which is used as a subcategory.

This field is an array. This will allow proper categorization of some events that fall in multiple categories.

Allowed Values

authentication

Events in this category are related to the challenge and response process in which credentials are supplied and verified to allow the creation of a session. Common sources for these logs are Windows event logs and ssh logs. Visualize and analyze events in this category to look for failed logins, and other authentication-related activity.

Expected event types for category authentication:

start, end, info

configuration

Events in the configuration category have to deal with creating, modifying, or deleting the settings or parameters of an application, process, or system.

Example sources include security policy change logs, configuration auditing logging, and system integrity monitoring.

Expected event types for category configuration:

access, change, creation, deletion, info

database

The database category denotes events and metrics relating to a data storage and retrieval system. Note that use of this category is not limited to relational database systems. Examples include event logs from MS SQL, MySQL, Elasticsearch, MongoDB, etc. Use this category to visualize and analyze database activity such as accesses and changes.

Expected event types for category database:

access, change, info, error

driver

Events in the driver category have to do with operating system device drivers and similar software entities such as Windows drivers, kernel extensions, kernel modules, etc.

Use events and metrics in this category to visualize and analyze driver-related activity and status on hosts.

Expected event types for category driver:

change, end, info, start

file

Relating to a set of information that has been created on, or has existed on a filesystem. Use this category of events to visualize and analyze the creation, access, and deletions of files. Events in this category can come from both host-based and network-based sources. An example source of a network-based detection of a file transfer would be the Zeek file.log.

Expected event types for category file:

change, creation, deletion, info

host

Use this category to visualize and analyze information such as host inventory or host lifecycle events.

Most of the events in this category can usually be observed from the outside, such as from a hypervisor or a control plane’s point of view. Some can also be seen from within, such as "start" or "end".

Note that this category is for information about hosts themselves; it is not meant to capture activity "happening on a host".

Expected event types for category host:

access, change, end, info, start

iam

Identity and access management (IAM) events relating to users, groups, and administration. Use this category to visualize and analyze IAM-related logs and data from active directory, LDAP, Okta, Duo, and other IAM systems.

Expected event types for category iam:

admin, change, creation, deletion, group, info, user

intrusion_detection

Relating to intrusion detections from IDS/IPS systems and functions, both network and host-based. Use this category to visualize and analyze intrusion detection alerts from systems such as Snort, Suricata, and Palo Alto threat detections.

Expected event types for category intrusion_detection:

allowed, denied, info

malware

Malware detection events and alerts. Use this category to visualize and analyze malware detections from EDR/EPP systems such as Elastic Endpoint Security, Symantec Endpoint Protection, Crowdstrike, and network IDS/IPS systems such as Suricata, or other sources of malware-related events such as Palo Alto Networks threat logs and Wildfire logs.

Expected event types for category malware:

info

network

Relating to all network activity, including network connection lifecycle, network traffic, and essentially any event that includes an IP address. Many events containing decoded network protocol transactions fit into this category. Use events in this category to visualize or analyze counts of network ports, protocols, addresses, geolocation information, etc.

Expected event types for category network:

access, allowed, connection, denied, end, info, protocol, start

package

Relating to software packages installed on hosts. Use this category to visualize and analyze inventory of software installed on various hosts, or to determine host vulnerability in the absence of vulnerability scan data.

Expected event types for category package:

access, change, deletion, info, installation, start

process

Use this category of events to visualize and analyze process-specific information such as lifecycle events or process ancestry.

Expected event types for category process:

access, change, end, info, start

registry

Having to do with settings and assets stored in the Windows registry. Use this category to visualize and analyze activity such as registry access and modifications.

Expected event types for category registry:

access, change, creation, deletion

session

The session category is applied to events and metrics regarding logical persistent connections to hosts and services. Use this category to visualize and analyze interactive or automated persistent connections between assets. Data for this category may come from Windows Event logs, SSH logs, or stateless sessions such as HTTP cookie-based sessions, etc.

Expected event types for category session:

start, end, info

threat

Use this category to visualize and analyze events describing threat actors' targets, motives, or behaviors.

Expected event types for category threat:

indicator

web

Relating to web server access. Use this category to create a dashboard of web server/proxy activity from apache, IIS, nginx web servers, etc. Note: events from network observers such as Zeek http log may also be included in this category.

Expected event types for category web:

access, error, info

ECS Categorization Field: event.type

This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy.

event.type represents a categorization "sub-bucket" that, when used along with the event.category field values, enables filtering events down to a level appropriate for single visualization.

This field is an array. This will allow proper categorization of some events that fall in multiple event types.

Allowed Values

access

The access event type is used for the subset of events within a category that indicate that something was accessed. Common examples include event.category:database AND event.type:access, or event.category:file AND event.type:access. Note for file access, both directory listings and file opens should be included in this subcategory. You can further distinguish access operations using the ECS event.action field.

admin

The admin event type is used for the subset of events within a category that are related to admin objects. For example, administrative changes within an IAM framework that do not specifically affect a user or group (e.g., adding new applications to a federation solution or connecting discrete forests in Active Directory) would fall into this subcategory. Common example: event.category:iam AND event.type:change AND event.type:admin. You can further distinguish admin operations using the ECS event.action field.

allowed

The allowed event type is used for the subset of events within a category that indicate that something was allowed. Common examples include event.category:network AND event.type:connection AND event.type:allowed (to indicate a network firewall event for which the firewall disposition was to allow the connection to complete) and event.category:intrusion_detection AND event.type:allowed (to indicate a network intrusion prevention system event for which the IPS disposition was to allow the connection to complete). You can further distinguish allowed operations using the ECS event.action field, populating with values of your choosing, such as "allow", "detect", or "pass".

change

The change event type is used for the subset of events within a category that indicate that something has changed. If semantics best describe an event as modified, then include them in this subcategory. Common examples include event.category:process AND event.type:change, and event.category:file AND event.type:change. You can further distinguish change operations using the ECS event.action field.

connection

Used primarily with event.category:network this value is used for the subset of network traffic that includes sufficient information for the event to be included in flow or connection analysis. Events in this subcategory will contain at least source and destination IP addresses, source and destination TCP/UDP ports, and will usually contain counts of bytes and/or packets transferred. Events in this subcategory may contain unidirectional or bidirectional information, including summary information. Use this subcategory to visualize and analyze network connections. Flow analysis, including Netflow, IPFIX, and other flow-related events fit in this subcategory. Note that firewall events from many Next-Generation Firewall (NGFW) devices will also fit into this subcategory. A common filter for flow/connection information would be event.category:network AND event.type:connection AND event.type:end (to view or analyze all completed network connections, ignoring mid-flow reports). You can further distinguish connection events using the ECS event.action field, populating with values of your choosing, such as "timeout", or "reset".

creation

The "creation" event type is used for the subset of events within a category that indicate that something was created. A common example is event.category:file AND event.type:creation.

deletion

The deletion event type is used for the subset of events within a category that indicate that something was deleted. A common example is event.category:file AND event.type:deletion to indicate that a file has been deleted.

denied

The denied event type is used for the subset of events within a category that indicate that something was denied. Common examples include event.category:network AND event.type:denied (to indicate a network firewall event for which the firewall disposition was to deny the connection) and event.category:intrusion_detection AND event.type:denied (to indicate a network intrusion prevention system event for which the IPS disposition was to deny the connection to complete). You can further distinguish denied operations using the ECS event.action field, populating with values of your choosing, such as "blocked", "dropped", or "quarantined".

end

The end event type is used for the subset of events within a category that indicate something has ended. A common example is event.category:process AND event.type:end.

error

The error event type is used for the subset of events within a category that indicate or describe an error. A common example is event.category:database AND event.type:error. Note that pipeline errors that occur during the event ingestion process should not use this event.type value. Instead, they should use event.kind:pipeline_error.

group

The group event type is used for the subset of events within a category that are related to group objects. Common example: event.category:iam AND event.type:creation AND event.type:group. You can further distinguish group operations using the ECS event.action field.

indicator

The indicator event type is used for the subset of events within a category that contain details about indicators of compromise (IOCs).

A common example is event.category:threat AND event.type:indicator.

info

The info event type is used for the subset of events within a category that indicate that they are purely informational, and don’t report a state change, or any type of action. For example, an initial run of a file integrity monitoring system (FIM), where an agent reports all files under management, would fall into the "info" subcategory. Similarly, an event containing a dump of all currently running processes (as opposed to reporting that a process started/ended) would fall into the "info" subcategory. An additional common examples is event.category:intrusion_detection AND event.type:info.

installation

The installation event type is used for the subset of events within a category that indicate that something was installed. A common example is event.category:package AND event.type:installation.

protocol

The protocol event type is used for the subset of events within a category that indicate that they contain protocol details or analysis, beyond simply identifying the protocol. Generally, network events that contain specific protocol details will fall into this subcategory. A common example is event.category:network AND event.type:protocol AND event.type:connection AND event.type:end (to indicate that the event is a network connection event sent at the end of a connection that also includes a protocol detail breakdown). Note that events that only indicate the name or id of the protocol should not use the protocol value. Further note that when the protocol subcategory is used, the identified protocol is populated in the ECS network.protocol field.

start

The start event type is used for the subset of events within a category that indicate something has started. A common example is event.category:process AND event.type:start.

user

The user event type is used for the subset of events within a category that are related to user objects. Common example: event.category:iam AND event.type:deletion AND event.type:user. You can further distinguish user operations using the ECS event.action field.

ECS Categorization Field: event.outcome

This is one of four ECS Categorization Fields, and indicates the lowest level in the ECS category hierarchy.

event.outcome simply denotes whether the event represents a success or a failure from the perspective of the entity that produced the event.

Note that when a single transaction is described in multiple events, each event may populate different values of event.outcome, according to their perspective.

Also note that in the case of a compound event (a single event that contains multiple logical events), this field should be populated with the value that best captures the overall success or failure from the perspective of the event producer.

Further note that not all events will have an associated outcome. For example, this field is generally not populated for metric events, events with event.type:info, or any events for which an outcome does not make logical sense.

Allowed Values

failure

Indicates that this event describes a failed result. A common example is event.category:file AND event.type:access AND event.outcome:failure to indicate that a file access was attempted, but was not successful.

success

Indicates that this event describes a successful result. A common example is event.category:file AND event.type:create AND event.outcome:success to indicate that a file was successfully created.

unknown

Indicates that this event describes only an attempt for which the result is unknown from the perspective of the event producer. For example, if the event contains information only about the request side of a transaction that results in a response, populating event.outcome:unknown in the request event is appropriate. The unknown value should not be used when an outcome doesn’t make logical sense for the event. In such cases event.outcome should not be populated.

Using the Categorization Fields

The event categorization fields work together to identify and group similar events from multiple data sources.

These general principles can help guide the categorization process:

  • Events from multiple data sources that are similar enough to be viewed or analyzed together, should fall into the same event.category field.

  • Both event.category and event.type are arrays and may be populated with multiple allowed values, if the event can be reasonably classified into more than one category and/or type.

  • event.kind, event.category, event.type and event.outcome all have allowed values. This is to normalize these fields. Values that aren’t in the list of allowed values should not be used.

  • Values of event.outcome are a very limited set to indicate success or failure. Domain-specific actions, such as deny and allow, that could be considered outcomes are not captured in the event.outcome field, but rather in the event.type and/or event.action fields.

  • Values of event.category, event.type, and event.outcome are consistent across all values of event.kind.

  • When a specific event doesn’t fit into any of the defined allowed categorization values, the field should be left empty.

The following examples detail populating the categorization fields and provides some context for the classification decisions.

Firewall blocking a network connection

This event from a firewall describes a successfully blocked network connection:

...
  {
    "source": {
      "address": "10.42.42.42",
      "ip": "10.42.42.42",
      "port": 38842
    },
    "destination": {
      "address": "10.42.42.1",
      "ip": "10.42.42.1",
      "port": 443
    },
    "rule": {
      "name": "wan-lan",
      "id": "default"
    },
    ...
    "event": {
      "kind": "event", <1>
      "category": [ <2>
        "network"
      ],
      "type": [ <3>
        "connection",
        "denied"
      ],
      "outcome": "success", <4>
      "action": "dropped" <5>
    }
  }
...

  1. Classifying as an event.

  2. event.category categorizes this event as network activity.

  3. The event was both an attempted network connection and was denied.

  4. The blocking of this connection is expected. The outcome is a success from the perspective of the firewall emitting the event.

  5. The firewall classifies this denied connection as dropped, and this value is captured in event.action.

A "denied" network connection could fall under different action values: "blocked", "dropped", "quarantined", etc. The event.action field captures the action taken as described by the source, and populating event.type:denied provides an independent, normalized value.

A single query will return all denied network connections which have been normalized with the same categorization values:

event.category:network AND event.type:denied

Failed attempt to create a user account

User alice attempts to add a user account, bob, into a directory service, but the action fails:

{
  "user": {
    "name": "alice",
    "target": {
      "name": "bob"
    }
  },
  "event": {
    "kind": "event", <1>
    "category": [ <2>
      "iam"
    ],
    "type": [ <3>
      "user",
      "creation"
    ],
    "outcome": "failure" <4>
  }
}

  1. Again classifying as an event.

  2. Categorized using iam for an event user account activity.

  3. Both user and creation

  4. The creation of a user account was attempted, but it was not successful.

Informational listing of a file

A utility, such as a file integrity monitoring (FIM) application, takes inventory of a file but does not access or modify the file:

{
  "file": {
    "name": "example.png",
    "owner": "alice",
    "path": "/home/alice/example.png",
    "type": "file"
  },
  "event": {
    "kind": "event", <1>
    "category": [ <2>
      "file"
    ],
    "type": [ <3>
      "info"
    ]
  }
}

  1. Classifying as event.

  2. The event is reporting on a file.

  3. The info type categorizes purely informational events. The target file here was not accessed or modified.

The source data didn’t include any context around the event’s outcome, so event.outcome should not be populated.

Security application failed to block a network connection

An intrusion detection system (IDS) attempts to block a connection but fails. The event emitted by the IDS is considered an alert:

{
  "source": {
      "address": "10.42.42.42",
      "ip": "10.42.42.42",
      "port": 38842
    },
  "destination": {
      "address": "10.42.42.1",
      "ip": "10.42.42.1",
      "port": 443
  },
  ...
  "event": {
    "kind": "alert", <1>
    "category": [ <2>
      "intrusion_detection",
      "network"
    ],
    "type": [ <3>
      "connection",
      "denied"
    ],
    "outcome": "failure" <4>
  }
}

  1. The IDS emitted this event when a detection rule generated an alert. The event.kind is set to alert.

  2. With the event emitted from a network IDS device, the event is categorized both as network and intrusion_detection.

  3. The alert event is a ‘connection` that was denied by the IDS’ configuration.

  4. The IDS experienced an issue when attempting to deny the connection. Since the action taken by the IDS failed, the outcome is set as failure.

Migrating to ECS

There are multiple ways to reap the benefit of ECS. The simplest is to use [ecs-products-solutions].

If you have a custom pipeline or application you would like to convert to ECS, please have a look at [ecs-converting].

Products and Solutions that Support ECS

The following Elastic products support ECS out of the box, as of version 7.0:

Converting a Custom Implementation

A common schema helps you correlate and use data from various sources.

Fields for most Elastic modules and solutions (version 7.0 and later) are mapped to the Elastic Common Schema. You may want to map data from other implementations to ECS to help you correlate data across all of your products and solutions.

Before you start a conversion, be sure that you understand the basics below.

Core and extended levels

Make sure you understand the distinction between Core and Extended fields, as explained in the [ecs-guidelines].

Core and Extended fields are documented in the [ecs-field-reference] or, for a single page representation of all fields, please see the /generated/csv/fields.csv[generated CSV of fields].

An approach to mapping an existing implementation

Here’s the recommended approach for converting an existing implementation to ECS.

  1. Review each field in your original event and map it to the relevant ECS field.

    • Start by mapping your field to the relevant ECS Core field.

    • If a relevant ECS Core field does not exist, map your field to the relevant ECS Extended field.

    • If no relevant ECS Extended field exists, consider keeping your field with its original details, or possibly renaming it using ECS naming guidelines and attempt to map one or more of your original event fields to it.

  2. Review each ECS Core field, and attempt to populate it.

    • Review your original event data again

    • Consider populating the field based on additional meta-data such as static information (e.g. add event.category:authentication even if your auth events don’t mention the word "authentication")

    • Consider capturing additional environment meta-data, such as information about the host, container or cloud instance.

  3. Review other extended fields from any field set you are already using, and attempt to populate it as well.

  4. Set ecs.version to the version of the schema you are conforming to. This will allow you to upgrade your sources, pipelines and content (like dashboards) smoothly in the future.

Using a spreadsheet to plan your migration

Using a spreadsheet to plan the migration from pre-existing source fields to ECS is a common practice. It’s a good way to address each of your fields methodically among colleagues.

If the data source is either a structured log, or if you already have a pipeline producing events with these non-ECS field names, the tool ECS Mapper may help you get started in performing all of these field renames.

After exporting your mapping spreadsheet to CSV, ECS Mapper will convert your field mapping to equivalent pipelines for Beats, Elasticsearch, and Logstash. Learn more at ECS Mapper.

Additional Information

Questions and Answers

What are the benefits of using ECS?

The benefits to a user adopting these fields and names in their clusters are:

  • Data correlation. Ability to easily correlate data from the same or different sources, including:

    • data from metrics, logs, and application performance management (apm) tools

    • data from the same machines/hosts

    • data from the same service

  • Ease of recall. Improved ability to remember commonly used field names (because there is a single set, not a set per data source)

  • Ease of deduction. Improved ability to deduce field names (because the field naming follows a small number of rules with few exceptions)

  • Reuse. Ability to re-use analysis content (searches, visualizations, dashboards, alerts, reports, and machine learning jobs) across multiple data sources

  • Future proofing. Ability to use any future Elastic-provided analysis content in your environment without modifications

What if I have fields that conflict with ECS?

The processor] can help you resolve field conflicts. For example, imagine that you already have a field called "user," but ECS employs user as an object. You can use the rename processor on ingest time to rename your field to the matching ECS field. If your field does not match ECS, you can rename your field to user.value instead.

What if my events have additional fields?

Events may contain fields in addition to ECS fields. These fields can follow the ECS naming and writing rules, but this is not a requirement.

Why does ECS use a dot notation instead of an underline notation?

There are two common key formats for ingesting data into Elasticsearch:

  • Dot notation: user.firstname: Nicolas, user.lastname: Ruflin

  • Underline notation: user_firstname: Nicolas, user_lastname: Ruflin

ECS uses the dot notation to represent nested objects.

What is the difference between the two notations?

Ingesting user.firstname: Nicolas and user.lastname: Ruflin is identical to ingesting the following JSON:

"user": {
  "firstname": "Nicolas",
  "lastname": "Ruflin"
}

datatype]. In the case of the underline notation, both are just

Advantages of dot notation

With dot notation, each prefix in Elasticsearch is an object. Each object can have that control how fields inside the object are treated. In the context of ECS, for example, these parameters would allow you to disable dynamic property creation for certain prefixes.

Individual objects give you more flexibility on both the ingest and the event sides. In Elasticsearch, for example, you can use the remove processor to drop complete objects instead of selecting each key inside. You don’t have to know ahead of time which keys will be in an object.

In Beats, you can simplify the creation of events. For example, you can treat each object as an object (or struct in Golang), which makes constructing and modifying each part of the final event easier.

Disadvantage of dot notation

In Elasticsearch, each key can have only one type. For example, if user is an object, you can’t use it as a keyword type in the same index, like {"user": "nicolas ruflin"}. This restriction can be an issue in certain datasets. For the ECS data itself, this is not an issue because all fields are predefined.

What if I already use the underline notation?

As long as there are no conflicts, underline notation and ECS dot notation can coexist in the same document.

Glossary of ECS Terms

ECS

Elastic Common Schema. The Elastic Common Schema (ECS) is a document schema for Elasticsearch, for use cases such as logging and metrics. ECS defines a common set of fields, their datatype, and gives guidance on their correct usage. ECS is used to improve uniformity of event data coming from different sources.

Contributing to ECS

All information related to ECS is versioned in the elastic/ecs repository. All changes to ECS happen through Pull Requests submitted through Git.

See the Contribution Guidelines.

Generated Artifacts

ECS maintains a collection of artifacts which are generated based on the schema. Examples include Elasticsearch index templates, CSV, and Beats field mappings. The maintained artifacts can be found in the /generated#artifacts-generated-from-ecs[ECS Github repo].

Users can generate custom versions of these artifacts using the ECS project’s tooling. See the tooling /USAGE.md[usage documentation] for more detail.

Downloads