Flume can modify and drop events using Interceptors, which can be attached to any Flume source. The MorphlineInterceptor runs morphline transformations on intercepted events. For example, a morphline can ignore events or alter or insert certain event headers using regular expression-based pattern matching, or it can auto-detect and set a MIME type using Apache Tika. Interceptors can be used for content-based routing in a Flume topology.

Flume also supports multiplexing event flows by defining a flow multiplexer that can replicate or selectively route an event to specific channels. The Flume User Guide example demonstrates a source from agent foo fanning out the flow to three different channels. This can be a replicating or multiplexingfan-out. In a replicating fan-out, each event is sent to all three channels. In a multiplexing fan-out, an event is sent to a subset of available channels when an event attribute matches a specified value. For example, if an event attribute called stream.type is set to application/pdf, it goes to channel1 and channel3. If the attribute is set to avro/binary, it goes to channel2. If a channel is unavailable, then an exception is thrown, and the event is replayed after the channel becomes available again. You can configure this mapping in the flume.conf file.

The MorphlineInterceptor supports the following configuration options (required options in bold):

agent.sources.avroSrc.interceptors = morphlineinterceptor
agent.sources.avroSrc.interceptors.morphlineinterceptor.type = org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
agent.sources.avroSrc.interceptors.morphlineinterceptor.morphlineFile = /etc/flume-ng/conf/morphline.conf
agent.sources.avroSrc.interceptors.morphlineinterceptor.morphlineId = morphline1

  Note: A morphline interceptor cannot generate more than one output record for each input event.