Logstash Input plug-ins for data processing services Input Common types and basic usage

1.Input Common types of plug-ins

logstash The plug-ins usually have stdin（ For debugging ）、file、redis、kafka、beats（filebeat） type .

2.Logstash Input Common configuration parameters

Input Some of the common configurations in are automatically , These fields can be in Redis、kakfa、filebeat、file And so on, used in various types of input plug-ins .

add_field： Add a field to an event , Put it at the top of the event , It is generally used to mark the log source . For example, which project does it belong to , Which application

tags： Add any number of tags , Other attributes used to mark logs , For example, it indicates whether the access log or the error log

type： Add a field for all inputs , For example, indicate the log type

These three configuration fields are added to the collected log data , You only need to select one of these three fields for configuration , stay outpost The output plug-in can match according to fields and values , Store the log data of different fields in different locations ES In the index library .

1. To configure logstash
[root@elkstack-1 ~]# vim /data/elk/logstash/conf.d/test.conf 
input {
    file {
        path => "/var/log/test/*.log"
        exclude => "error.log"
        start_position =>  "beginning"
        tags => "web"					# Add a label 
        type => "nginx"					# Add a field type, The value is nginx
        add_field => {					# Add field information at the front of the log data ,project by web,app by nginx
            "project" => "web"
            "app" => "nginx"
        }
    }
}

filter {

}

output {
    file {
        path => "/tmp/log.txt"
    }
}

2. Overload configuration 
[root@elkstack-1 ~]# ps aux | grep logstash | grep -v grep | awk '{print $2}' |xargs kill -HUPUP

Observe the collected log data information , Added some field properties we configured

{"message":"web field","app":"nginx","path":"/var/log/test/access.log","tags":["web"],"host":"elkstack-1","project":"web","@version":"1","@timestamp":"2021-12-09T09:18:58.735Z","type":"nginx"}

The collected log data is json Format , Can be placed json Displayed in parser

3.Input To configure File Type of input plug-in

With /var/log/messages For example , adopt logstash take message Log output from to /tmp/log.txt In file .

3.1. Classic case

1. Writing configuration files 
[root@elkstack-1 logstash]# cat conf.d/test.conf 
input {						# Define the source of the log 
	file {							# The type is file, Read log data from file 
		path => "/var/log/message"				# Log path 
	}
}

filter {			# filter 

}

output {					# Define the storage path of log output 
	file {
		path => "/tmp/log.txt"			# Log path 
	}
}

2. heavy load logstash
[root@elkstack-1 logstash]# ps aux | grep logstash | grep -v grep | awk '{print $2}' |xargs kill -HUP
 perhaps 
[root@elkstack-1 logstash]# systemctl restart logstash

Observe /tmp/log.txt Is there any in the file logstash Data content written .

1. towards /var/log/messages Insert log data into 
[root@elkstack-1 ~]# echo "123456" >> /var/log/messages

3. track /tmp/log.txt Observation logs are output 
[root@elkstack-1 ~]# tail -f /tmp/log.txt
{"path":"/var/log/messages","host":"elkstack-1","@version":"1","@timestamp":"2021-12-09T08:12:43.656Z","message":"123456"}

3.2.File Common fields of type

path： Log storage path , Support wildcard matching .

exclude： Exclude log files that do not need to be collected .

start_position： Specify where to start reading log files , The default is to read from the end , Specifies that the parameter value is beginning Indicates reading from the beginning of the file .

Example ： Read /var/log/test All log files in the directory , But exclude error.log This file .

1.logstash Profile contents 
[root@elkstack-1 ~]# vim /data/elk/logstash/conf.d/test.conf 
input {
	file {
		path => "/var/log/test/*.log"		# Read test All log files in the directory 
		exclude => "error.log"					# Don't collect error Log files 
		start_position =>  "beginning"				# Read from the beginning of the file 
	}
}

filter {

}

output {
	file {
		path => "/tmp/log.txt"
	}
}

2. heavy load logstash
[root@elkstack-1 ~]# ps aux | grep logstash | grep -v grep | awk '{print $2}' |xargs kill -HUP

Observe the effect of log collection

1. towards access Write data to the log 
[root@elkstack-1 ~]# echo access12332 > /var/log/test/access.log 
[root@elkstack-1 ~]# echo access1 >> /var/log/test/access.log 

2. Logs are logstash Normal collection of 
[root@elkstack-1 ~]# tail -f /tmp/log.txt 
{"path":"/var/log/test/access.log","message":"2","@version":"1","host":"elkstack-1","@timestamp":"2021-12-09T08:53:54.953Z"}
{"path":"/var/log/test/access.log","message":"access1","@version":"1","host":"elkstack-1","@timestamp":"2021-12-09T08:53:59.969Z"}

3. write in error Logs will not be collected 
[root@elkstack-1 ~]# echo abc >> /var/log/test/access.log

4.logstash beats Type of plug-in

Beats The plug-in is used to receive data from Beats The data sent by the data collector , Common data collectors are Filebeat、Metricbeat、Packetbeat、Winlogbeat、Auditbeat、Heartbeat、Functionbeat wait .

Examples are as follows ：

 1. To configure logstash
 [root@elkstack-1 ~]# vim /data/elk/logstash/conf.d/test.conf 
 input {
     beats {
         host => "0.0.0.0"
         port => 5044
     }
 }
 
 filter {
 
 }
 
 output {
     file {
         path => "/tmp/log.txt"
     }
 }