I. Definition of Flume

Flume is a highly available, highly reliable, distributed massive log collection, aggregation and transmission system provided by Cloudera.Flume is based on a streaming architecture, which is flexible and simple.

The main function of Flume is to read the server in real timeLocal disk data, write data to HDFS.

Second, Flume infrastructure

The composition of Flume is shown in the following figure.

1. Agent

Agent is a JVM process that sends data from source to destination in the form of events.Agent mainly consists of 3 parts, Source, Channel and Sink.

2. Source

Source is the component responsible for receiving data to Flume Agent.The Source component can handle log data of various types and formats, including avro, thrift, exec, jms, spooling directory, netcat, taildir, sequence generator, syslog, http, legacy.

3. Sink

Sink continuously polls the Channel for events and removes them in batches, and writes these events in batches to a storage or indexing system, or is sent to another Flume Agent.Sink component destinations include hdfs, logger, avro, thrift, ipc, file, HBase, solr, custom.

4. Channel

Channel is a buffer between Source and Sink.Therefore, Channel allows Source and Sink to operate at different rates.Channel is thread-safe and can handle write operations from several sources and read operations from several sinks at the same time.

Flume comes with two channels: Memory Channel and File Channel.

Memory Channel is an in-memory queue.Memory Channel is suitable for scenarios where data loss is not a concern.If you need to care about data loss, then Memory Channel should not be used, because program death, machine downtime or restart will cause data loss.

File Channel writes all events to disk.So no data is lost in case of program shutdown or machine downtime.

5. Event

Transmission unit, the basic unit of Flume data transmission, sends data from source to destination in the form of Event.Event consists of Header and Body. Header is used to store some attributes of the event, which is a K-V structure, and Body is used to store the data in the form of a byte array.