org.wikimedia.eventutilities.flink.stream

Class LineStreamFormat<T>

java.lang.Object

org.apache.flink.connector.file.src.reader.SimpleStreamFormat<T>

org.wikimedia.eventutilities.flink.stream.LineStreamFormat<T>

All Implemented Interfaces:

Serializable, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>, org.apache.flink.connector.file.src.reader.StreamFormat<T>

public class LineStreamFormat<T>
extends org.apache.flink.connector.file.src.reader.SimpleStreamFormat<T>

A reader format that reads text lines from a file, and uses the provided DeserializationSchema to deserialize them.

The reader uses Java's built-in InputStreamReader to decode the byte stream using StandardCharsets.UTF_8. NOTE: This class is based on Flink 1.15.0's TextLineInputFormat.

This format does not support optimized recovery from checkpoints. On recovery, it will re-read and discard the number of lines that were processed before the last checkpoint. That is due to the fact that the offsets of lines in the file cannot be tracked through the charset decoders with their internal buffering of stream input and charset decoder state.

See Also:

Serialized Form

Field Summary

Fields inherited from interface org.apache.flink.connector.file.src.reader.StreamFormat

FETCH_IO_SIZE

Constructor Summary

Constructors

Constructor and Description
LineStreamFormat(org.apache.flink.api.common.serialization.DeserializationSchema<T> deserializationSchema)

Method Summary

Modifier and Type	Method and Description
org.wikimedia.eventutilities.flink.stream.LineStreamFormat.Reader<T>	createReader(org.apache.flink.configuration.Configuration config, org.apache.flink.core.fs.FSDataInputStream stream)
org.apache.flink.api.common.typeinfo.TypeInformation<T>	getProducedType()

Methods inherited from class org.apache.flink.connector.file.src.reader.SimpleStreamFormat

createReader, isSplittable, restoreReader

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LineStreamFormat

public LineStreamFormat(org.apache.flink.api.common.serialization.DeserializationSchema<T> deserializationSchema)

Method Detail

createReader

public org.wikimedia.eventutilities.flink.stream.LineStreamFormat.Reader<T> createReader(org.apache.flink.configuration.Configuration config,
                                                                                         org.apache.flink.core.fs.FSDataInputStream stream)

Specified by:

createReader in class org.apache.flink.connector.file.src.reader.SimpleStreamFormat<T>

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<T> getProducedType()

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

Specified by:

getProducedType in interface org.apache.flink.connector.file.src.reader.StreamFormat<T>

Specified by:

getProducedType in class org.apache.flink.connector.file.src.reader.SimpleStreamFormat<T>