org.wikimedia.eventutilities.flink

Class EventRowTypeInfo

java.lang.Object

org.apache.flink.api.common.typeinfo.TypeInformation<T>

org.apache.flink.api.common.typeutils.CompositeType<T>

org.apache.flink.api.java.typeutils.TupleTypeInfoBase<org.apache.flink.types.Row>

org.apache.flink.api.java.typeutils.RowTypeInfo

org.wikimedia.eventutilities.flink.EventRowTypeInfo

All Implemented Interfaces:

Serializable

public class EventRowTypeInfo
extends org.apache.flink.api.java.typeutils.RowTypeInfo

Simple subclasss of RowTypeInfo that provides support for:

• WMF Event Platform rules
• flink application state schema evolution for the Row type (following WMF guidelines)

NOTE: the serialization format is not compatible with the flink RowSerialization and thus RowTypeInfo and EventRowTypeInfo should not be mixed up.
If for some reasons you have to work with plain RowTypeInfo (pyflink) you can obtain the corresponding RowTypeInfo using toRowTypeInfo().

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.api.common.typeutils.CompositeType

org.apache.flink.api.common.typeutils.CompositeType.FlatFieldDescriptor, org.apache.flink.api.common.typeutils.CompositeType.InvalidFieldReferenceException, org.apache.flink.api.common.typeutils.CompositeType.TypeComparatorBuilder<T>

Field Summary

Fields

Modifier and Type	Field and Description
static Set<String>	DEFAULT_NON_PROJECTED_FIELDS List of fields that should not be projected by default when projecting a source event to a target event.

Fields inherited from class org.apache.flink.api.java.typeutils.RowTypeInfo

fieldNames

Fields inherited from class org.apache.flink.api.java.typeutils.TupleTypeInfoBase

types

Constructor Summary

Constructors

Constructor and Description
EventRowTypeInfo(org.apache.flink.api.common.typeinfo.TypeInformation<?>[] types, String[] fieldNames)
EventRowTypeInfo(org.apache.flink.api.common.typeinfo.TypeInformation<?>[] types, String[] fieldNames, Map<String,String> aliasedRowKey)

Method Summary

Modifier and Type	Method and Description
boolean	canEqual(Object obj)
static EventRowTypeInfo	create(Map<String,String> aliasedRowKey, String[] fieldNames, org.apache.flink.api.common.typeinfo.TypeInformation<?>... types)
static EventRowTypeInfo	create(org.apache.flink.api.java.typeutils.RowTypeInfo info)
static EventRowTypeInfo	create(org.apache.flink.api.java.typeutils.RowTypeInfo info, Map<String,String> aliasedRowKey)
static EventRowTypeInfo	create(String[] fieldNames, org.apache.flink.api.common.typeinfo.TypeInformation<?>... types)
org.apache.flink.types.Row	createEmptyRow() Creates a new empty Row matching this RowTypeInfo.
org.apache.flink.types.Row	createEmptySubRow(String fieldPath) Creates a new empty instance of a sub Row identified by fieldPath.
org.apache.flink.api.common.typeutils.TypeSerializer<org.apache.flink.types.Row>	createSerializer(org.apache.flink.api.common.ExecutionConfig config) Deprecated.
org.apache.flink.api.common.typeutils.TypeSerializer<org.apache.flink.types.Row>	createSerializer(org.apache.flink.api.common.serialization.SerializerConfig config)
boolean	equals(Object o)
org.apache.flink.types.Row	extractKey(org.apache.flink.types.Row row) Extract a key for a given input Row.
Instant	getEventTime(org.apache.flink.types.Row element) Read the event time field.
Instant	getIngestionTime(org.apache.flink.types.Row element) Get the ingestion time of this event if set.
int	hashCode()
boolean	hasKey() Check if the EventRowTypeInformation instance has a partition key.
EventRowTypeInfo	keyTypeInfo() Return a TypeInfo representing the schema of Row keys.
org.apache.flink.types.Row	projectFrom(org.apache.flink.types.Row row, boolean intersect) Project an input Row to a new Row matching the TypeInformation of this class.
org.apache.flink.types.Row	projectFrom(org.apache.flink.types.Row row, boolean intersect, Set<String> ignoredFields) Project an input Row to a new Row matching the TypeInformation of this class.
void	setEventTime(org.apache.flink.types.Row element, Instant eventTime) Set the event time (dt) as per WMF event platform rules.
void	setIngestionTime(org.apache.flink.types.Row element, Instant ingestionTime) Set the ingestion time (meta.dt) as per WMF event platform rules.
org.apache.flink.api.java.typeutils.RowTypeInfo	toRowTypeInfo() Obtain the equivalent RowTypeInfo.

Methods inherited from class org.apache.flink.api.java.typeutils.RowTypeInfo

createComparator, createLegacySerializer, createTypeComparatorBuilder, getFieldIndex, getFieldNames, getFlatFields, getTypeAt, projectFields, schemaEquals, toString

Methods inherited from class org.apache.flink.api.java.typeutils.TupleTypeInfoBase

getArity, getFieldTypes, getTotalFields, getTypeAt, hasDeterministicFieldOrder, isBasicType, isCaseClass, isTupleType

Methods inherited from class org.apache.flink.api.common.typeutils.CompositeType

getFlatFields, getTypeClass, hasField, isKeyType, isSortKeyType

Methods inherited from class org.apache.flink.api.common.typeinfo.TypeInformation

getGenericParameters, of, of

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_NON_PROJECTED_FIELDS

public static final Set<String> DEFAULT_NON_PROJECTED_FIELDS

List of fields that should not be projected by default when projecting a source event to a target event.

Constructor Detail

EventRowTypeInfo

public EventRowTypeInfo(org.apache.flink.api.common.typeinfo.TypeInformation<?>[] types,
                        String[] fieldNames)

EventRowTypeInfo

public EventRowTypeInfo(org.apache.flink.api.common.typeinfo.TypeInformation<?>[] types,
                        String[] fieldNames,
                        @Nullable
                        Map<String,String> aliasedRowKey)

Method Detail

create

public static EventRowTypeInfo create(String[] fieldNames,
                                      org.apache.flink.api.common.typeinfo.TypeInformation<?>... types)

create

public static EventRowTypeInfo create(Map<String,String> aliasedRowKey,
                                      String[] fieldNames,
                                      org.apache.flink.api.common.typeinfo.TypeInformation<?>... types)

create

public static EventRowTypeInfo create(org.apache.flink.api.java.typeutils.RowTypeInfo info)

create

public static EventRowTypeInfo create(org.apache.flink.api.java.typeutils.RowTypeInfo info,
                                      Map<String,String> aliasedRowKey)

toRowTypeInfo

public org.apache.flink.api.java.typeutils.RowTypeInfo toRowTypeInfo()

Obtain the equivalent RowTypeInfo. Useful for APIs that expects to work on a plain RowTypeInfo.

canEqual

public boolean canEqual(Object obj)

Overrides:

canEqual in class org.apache.flink.api.java.typeutils.RowTypeInfo

createEmptyRow

public org.apache.flink.types.Row createEmptyRow()

Creates a new empty Row matching this RowTypeInfo.

createEmptySubRow

public org.apache.flink.types.Row createEmptySubRow(String fieldPath)

Creates a new empty instance of a sub Row identified by fieldPath. This path supports dotted notation to access nested subfields. For arrays or maps of Row simply pass the name of the field of the array or map.

Throws:

IllegalArgumentException - if the path leads to a non-existent field or a field that is not a Row.

projectFrom

public org.apache.flink.types.Row projectFrom(org.apache.flink.types.Row row,
                                              boolean intersect,
                                              Set<String> ignoredFields)

Project an input Row to a new Row matching the TypeInformation of this class. The passed Row must:

• if intersect is false be compatible with this TypeInformation and must have all the fields declared by it
• support named based access

If intersect is true only the fields declared by both row and this TypeInformation will be copied over. The use-case of this method is to support WMF schema downgrades in case a pipeline reads events in schema V2 but wants to write them as a previous version (only new fields are allowed during schema evolution). The param ignoredFields can be used to explicitly ignore some field from the projection. The content must include full-path using a dot as a path separator (e.g. "meta.dt" to exclude the nested "dt" field). Edge-cases: - when intersect is true the resulting Row might be emtpy if no common fields are present - if the intersection results in an empty composite field (map, row, arrays) this field will not be projected.

projectFrom

public org.apache.flink.types.Row projectFrom(org.apache.flink.types.Row row,
                                              boolean intersect)

Project an input Row to a new Row matching the TypeInformation of this class. The following data will be ignored:

• all the meta field content except the request_id, domain fields
• the $schema field

See Also:

projectFrom(Row, boolean, Set), DEFAULT_NON_PROJECTED_FIELDS

setIngestionTime

public void setIngestionTime(org.apache.flink.types.Row element,
                             Instant ingestionTime)

Set the ingestion time (meta.dt) as per WMF event platform rules.

Throws:

IllegalArgumentException - if the type information does not declare the meta field.

See Also:

meta.dt rules

setEventTime

public void setEventTime(org.apache.flink.types.Row element,
                         Instant eventTime)

Set the event time (dt) as per WMF event platform rules.

Throws:

IllegalArgumentException - if the type information does not declare the dt field

getEventTime

@Nullable
public Instant getEventTime(org.apache.flink.types.Row element)

Read the event time field. May return null if the field is not set.

getIngestionTime

public Instant getIngestionTime(org.apache.flink.types.Row element)

Get the ingestion time of this event if set.

createSerializer

@Deprecated
public org.apache.flink.api.common.typeutils.TypeSerializer<org.apache.flink.types.Row> createSerializer(org.apache.flink.api.common.ExecutionConfig config)

Deprecated.

Overrides:

createSerializer in class org.apache.flink.api.java.typeutils.RowTypeInfo

createSerializer

public org.apache.flink.api.common.typeutils.TypeSerializer<org.apache.flink.types.Row> createSerializer(org.apache.flink.api.common.serialization.SerializerConfig config)

Overrides:

createSerializer in class org.apache.flink.api.java.typeutils.RowTypeInfo

hasKey

public boolean hasKey()

Check if the EventRowTypeInformation instance has a partition key.

Returns:

extractKey

@Nullable
public org.apache.flink.types.Row extractKey(org.apache.flink.types.Row row)

Extract a key for a given input Row. Returns null is EventRowTypeInfo has no key.

Parameters:

row -

Returns:

keyTypeInfo

@Nullable
public EventRowTypeInfo keyTypeInfo()

Return a TypeInfo representing the schema of Row keys. Returns null if EventRowTypeInfo has no key.

Returns:

equals

public boolean equals(Object o)

Overrides:

equals in class org.apache.flink.api.java.typeutils.RowTypeInfo

hashCode

public int hashCode()

Overrides:

hashCode in class org.apache.flink.api.java.typeutils.RowTypeInfo