org.wikimedia.eventutilities.monitoring

Class CanaryEventProducer

java.lang.Object

org.wikimedia.eventutilities.monitoring.CanaryEventProducer

public class CanaryEventProducer
extends Object

Uses an EventStreamFactory to create and POST Wikimedia canary events to Wikimedia event services. Canary events are constructed from the first entry in the event schema's examples field. It is expected that event schemas have the fields listed at https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Guidelines#Required_fields, and also that the receiving event intake service (e.g. EventGate) will set meta.dt if it is not present in the event.

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	CANARY_DOMAIN Will be used as the value of meta.domain when building canary events.
protected static List<String>	DATACENTERS List of data center names that will be used to look up event service urls.
protected EventStreamFactory	eventStreamFactory EventStreamFactory instance used when constructing EventStreams.
protected static com.fasterxml.jackson.databind.ObjectMapper	OBJECT_MAPPER Used for serializing JsonNode events to Strings.

Constructor Summary

Constructors

Constructor and Description
CanaryEventProducer(EventSchemaLoader eventSchemaLoader, EventStreamConfig eventStreamConfig, BasicHttpClient httpClient) Constructs a new instance of CanaryEventProducer with a new instance of EventStreamFactory from eventSchemaLoader and eventStreamConfig.
CanaryEventProducer(EventStreamFactory eventStreamFactory, BasicHttpClient client) Constructs a new CanaryEventProducer using the provided EventStreamFactory.

Method Summary

Modifier and Type	Method and Description
com.fasterxml.jackson.databind.node.ObjectNode	canaryEvent(EventStream es) Given an EventStream, gets its schema and uses the JSONSchema examples to make a canary event.
com.fasterxml.jackson.databind.node.ObjectNode	canaryEvent(EventStream es, org.joda.time.DateTime timestamp) Given an EventStream, gets its schema and uses the JSONSchema examples to make a canary event.
com.fasterxml.jackson.databind.node.ObjectNode	canaryEvent(String streamName) Given a streamName, gets its schema and uses the JSONSchema examples to make a canary event.
com.fasterxml.jackson.databind.node.ObjectNode	canaryEvent(String streamName, org.joda.time.DateTime timestamp)
static com.fasterxml.jackson.databind.node.ArrayNode	eventsToArrayNode(List<com.fasterxml.jackson.databind.node.ObjectNode> events) Given a List of ObjectNodes, returns an ArrayNode of those ObjectNodes.
Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>>	getAllCanaryEventsToPost() Gets canary events to POST for all streams that EventStreamConfig knows about.
Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>>	getCanaryEventsToPost(List<String> streamNames) Gets canary events to POST for a List of stream names.
Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>>	getCanaryEventsToPost(List<String> streamNames, org.joda.time.DateTime timestamp) Gets canary events to POST for a List of stream names.
Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>>	getCanaryEventsToPost(String streamName) Gets canary events to POST for a single stream.
Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>>	getCanaryEventsToPost(String streamName, org.joda.time.DateTime timestamp) Gets canary events to POST for a single stream.
Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>>	getCanaryEventsToPostForStreams(List<EventStream> eventStreams, org.joda.time.DateTime timestamp) Given a list of streams and a timestamp, this will return a map of datacenter specific event service URIs to a list of canary events that should be POSTed to that event service.
EventStreamFactory	getEventStreamFactory() Returns the EventStreamFactory this CanaryEventProducer is using.
protected static com.fasterxml.jackson.databind.node.ObjectNode	makeCanaryEvent(String streamName, com.fasterxml.jackson.databind.node.ObjectNode example)
protected static com.fasterxml.jackson.databind.node.ObjectNode	makeCanaryEvent(String streamName, com.fasterxml.jackson.databind.node.ObjectNode example, org.joda.time.DateTime timestamp) Creates a canary event from an example event for a stream.
Map<URI,BasicHttpResult>	postAllCanaryEvents() POSTs canary events for all known streams.
Map<URI,BasicHttpResult>	postCanaryEvents(List<String> streamNames) Posts canary events for each named event stream.
Map<URI,BasicHttpResult>	postCanaryEvents(String streamName) Posts canary events for a single streamName.
Map<URI,BasicHttpResult>	postCanaryEventsForStreams(List<EventStream> eventStreams) Gets canary events for each eventStream, POSTs them to the appropriate event service url(s), and collects the results of each POST into a Map of event service url to result ObjectNode.
BasicHttpResult	postEvents(URI eventServiceUri, List<com.fasterxml.jackson.databind.node.ObjectNode> events) POSTs the given list of events to the eventServiceUri.
Map<URI,BasicHttpResult>	postEventsToUris(Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>> uriToEvents) Iterates over the Map of URI to events and posts events to the URI.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

eventStreamFactory

protected final EventStreamFactory eventStreamFactory

EventStreamFactory instance used when constructing EventStreams.

DATACENTERS

protected static final List<String> DATACENTERS

List of data center names that will be used to look up event service urls.

OBJECT_MAPPER

protected static final com.fasterxml.jackson.databind.ObjectMapper OBJECT_MAPPER

Used for serializing JsonNode events to Strings.

CANARY_DOMAIN

protected static final String CANARY_DOMAIN

Will be used as the value of meta.domain when building canary events. It does not matter what this is, but it should be consistent.

See Also:

Constant Field Values

Constructor Detail

CanaryEventProducer

public CanaryEventProducer(EventSchemaLoader eventSchemaLoader,
                           EventStreamConfig eventStreamConfig,
                           BasicHttpClient httpClient)

Constructs a new instance of CanaryEventProducer with a new instance of EventStreamFactory from eventSchemaLoader and eventStreamConfig.

CanaryEventProducer

public CanaryEventProducer(EventStreamFactory eventStreamFactory,
                           BasicHttpClient client)

Constructs a new CanaryEventProducer using the provided EventStreamFactory.

Method Detail

getEventStreamFactory

@Nonnull
public EventStreamFactory getEventStreamFactory()

Returns the EventStreamFactory this CanaryEventProducer is using.

canaryEvent

@Nonnull
public com.fasterxml.jackson.databind.node.ObjectNode canaryEvent(String streamName)

Given a streamName, gets its schema and uses the JSONSchema examples to make a canary event.

canaryEvent

@Nonnull
public com.fasterxml.jackson.databind.node.ObjectNode canaryEvent(String streamName,
                                                                           org.joda.time.DateTime timestamp)

canaryEvent

@Nonnull
public com.fasterxml.jackson.databind.node.ObjectNode canaryEvent(EventStream es)

Given an EventStream, gets its schema and uses the JSONSchema examples to make a canary event.

canaryEvent

@Nonnull
public com.fasterxml.jackson.databind.node.ObjectNode canaryEvent(EventStream es,
                                                                           org.joda.time.DateTime timestamp)

Given an EventStream, gets its schema and uses the JSONSchema examples to make a canary event.

makeCanaryEvent

@Nonnull
protected static com.fasterxml.jackson.databind.node.ObjectNode makeCanaryEvent(String streamName,
                                                                                         com.fasterxml.jackson.databind.node.ObjectNode example,
                                                                                         org.joda.time.DateTime timestamp)

Creates a canary event from an example event for a stream.

makeCanaryEvent

protected static com.fasterxml.jackson.databind.node.ObjectNode makeCanaryEvent(String streamName,
                                                                                com.fasterxml.jackson.databind.node.ObjectNode example)

getAllCanaryEventsToPost

@Nonnull
public Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>> getAllCanaryEventsToPost()

Gets canary events to POST for all streams that EventStreamConfig knows about. Refer to docs for getCanaryEventsToPostForStreams(eventStreams).

getCanaryEventsToPost

@Nonnull
public Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>> getCanaryEventsToPost(String streamName)

Gets canary events to POST for a single stream. Refer to docs for getCanaryEventsToPostForStreams(eventStreams).

getCanaryEventsToPost

@Nonnull
public Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>> getCanaryEventsToPost(String streamName,
                                                                                                    org.joda.time.DateTime timestamp)

Gets canary events to POST for a single stream. Refer to docs for getCanaryEventsToPostForStreams(eventStreams).

getCanaryEventsToPost

@Nonnull
public Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>> getCanaryEventsToPost(List<String> streamNames)

Gets canary events to POST for a List of stream names. Refer to docs for getCanaryEventsToPostForStreams(eventStreams).

getCanaryEventsToPost

@Nonnull
public Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>> getCanaryEventsToPost(List<String> streamNames,
                                                                                                    org.joda.time.DateTime timestamp)

Gets canary events to POST for a List of stream names. Refer to docs for getCanaryEventsToPostForStreams(eventStreams).

getCanaryEventsToPostForStreams

@Nonnull
public Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>> getCanaryEventsToPostForStreams(List<EventStream> eventStreams,
                                                                                                              org.joda.time.DateTime timestamp)

Given a list of streams and a timestamp, this will return a map of datacenter specific event service URIs to a list of canary events that should be POSTed to that event service. These can then be iterated through and posted to each event service URI to post expected canary events for each stream. If timestamp is null, the meta.dt field of canary events is removed to be set by EventGate, otherwise it is set to the provided value.

postAllCanaryEvents

@Nonnull
public Map<URI,BasicHttpResult> postAllCanaryEvents()

POSTs canary events for all known streams. Refer to docs for postCanaryEVents(streamNames). Refer to docs for postCanaryEventsForStreams(eventStreams).

postCanaryEvents

@Nonnull
public Map<URI,BasicHttpResult> postCanaryEvents(String streamName)

Posts canary events for a single streamName. Refer to docs for postCanaryEventsForStreams(eventStreams).

postCanaryEvents

@Nonnull
public Map<URI,BasicHttpResult> postCanaryEvents(List<String> streamNames)

Posts canary events for each named event stream. Refer to docs for postCanaryEventsForStreams(eventStreams).

postCanaryEventsForStreams

@Nonnull
public Map<URI,BasicHttpResult> postCanaryEventsForStreams(List<EventStream> eventStreams)

Gets canary events for each eventStream, POSTs them to the appropriate event service url(s), and collects the results of each POST into a Map of event service url to result ObjectNode. We want to attempt every POST we are supposed to do without bailing when an error is encountered. This is why the results are collected in this way The results should be examined after this method returns to check for any failures.

eventsToArrayNode

@Nonnull
public static com.fasterxml.jackson.databind.node.ArrayNode eventsToArrayNode(List<com.fasterxml.jackson.databind.node.ObjectNode> events)

Given a List of ObjectNodes, returns an ArrayNode of those ObjectNodes.

postEventsToUris

@Nonnull
public Map<URI,BasicHttpResult> postEventsToUris(Map<URI,List<com.fasterxml.jackson.databind.node.ObjectNode>> uriToEvents)

Iterates over the Map of URI to events and posts events to the URI.

postEvents

@Nonnull
public BasicHttpResult postEvents(URI eventServiceUri,
                                           List<com.fasterxml.jackson.databind.node.ObjectNode> events)

POSTs the given list of events to the eventServiceUri. Expects that eventServiceUri returns a JSON response. EventGate returns 201 if guaranteed success, 202 if hasty success, and 207 if partial success (some events were accepted, others were not). We want to only consider 201 and 202 as full success so we pass httpPostJson a custom isSuccess function to determine this. https://github.com/wikimedia/eventgate/blob/master/spec.yaml#L72 The returned BasicHttpResult will look like: success: true, status 201, message: "HTTP response message", body: response body if any } If ANY events failed POSTing, success will be false, and the reasons for the failure will be in message and body. If there is a local exception during POSTing, success will be false and the Exception message will be in message, and in the exception field will have the original Exception.