Logstash ~ how logstash works

LogStash The event processing pipeline has three stages ：input->filter->output.input Generate an event ,filter Modify them ,output Send them elsewhere .input and output Support codec , It can encode or decode data when it enters or exits the pipeline , Instead of using a separate filter

1.input

input Official documentation of plug-ins

Use input to enter data LogStash, Some common inputs include ：

• file： Read from a file on the file system
• syslog： stay 514 Listening on the port syslog News and according to RFC3164 Format
• beats： Handle Beats Events sent

2.filter

filter Official documents of

filter yes LogStash Intermediate processing equipment in the pipeline . If the event meets certain conditions , You can combine filters with conditions to perform actions on events . Some common filters include ：

• grok： Parsing and constructing arbitrary text .Grok At present, it is LogStash The best way to parse unstructured log data into structured and queryable
• mutate： Perform a general conversion on the event field . You can rename 、 Delete 、 Replace and modify fields in events
• drop： Completely delete the event , For example, debug events
• clone： Copy an event , Fields may be added or removed
• geoip： Add something about IP The information system of address and geographical location

3.output

output Official documents of

output yes LogStash The final stage of . An event can pass through multiple outputs , But once all the output processing is complete , The event completes its execution . Some common outputs include ：

• elasticsearch： Send event data to ElasticSearch.
• file： Write event data to a file on disk
• graphite： Send event data to Graphite, Used to store and draw indicators
• statsd： Send event data to statsd, This is a “ Monitoring statistics , Such as counters and timers , adopt UDP Send and send aggregation to one or more pluggable back-end services ” Service for

4.codec

codec Official documents of

codec Is the basic flow filter , It can be operated as part of the input or output . The encoder can separate the transmission of messages from the serialization process .

• json： With JSON Format encodes or decodes data
• multiline： Merge multiple lines of text events into a single event