filebeat、logstash配置安装

1.下载安装包

https://www.elastic.co/cn/downloads/past-releases/filebeat-7-6-1
https://www.elastic.co/cn/downloads/past-releases/logstash-7-6-1

 2.logstash配置文件

在logstash config目录下新建logstash.conf文件

logstash.conf配置如下：

input {
    beats {
    port => 5044
    }
}

filter {
    grok {
    match => { 
    "message" => "%{TIMESTAMP_ISO8601:timestamp}\s*\[%{DATA:jetty}\]\s*%{LOGLEVEL:log_level}\s*%{NUMBER:number}\s*\TID:%{DATA:TID}\s*\---\s*\[%{DATA:thread}\]\s*(?m)(?<msg>.*|\s)" 
    }
}
mutate {
    enable_metric => "false"
    remove_field => ["message", "log", "tags",  "input", "agent", "host", "ecs", "@version"]
}
date {
    match => ["date","dd/MMM/yyyy:HH:mm:ss Z","yyyy-MM-dd HH:mm:ss"]
    target => "date"
    }
}

output {
elasticsearch {
   hosts => ["12.0.0.1:9200"]
   index => "web_log_%{+YYYY-MM}"
  }
}

2022-04-21 08:48:20.277 [wkb-api-biz]  INFO 3389 TID:760669fd04d54b0188f4ac533499b57f.116.16505021002761001 --- [XNIO-1 task-1] a.LoginUserHandlerMethodArgumentResolver 

grok正则表达式对应自己的日志文件：

 %{TIMESTAMP_ISO8601:timestamp}\s*\[%{DATA:jetty}\]\s*%{LOGLEVEL:log_level}\s*%{NUMBER:number}\s*\TID:%{DATA:TID}\s*\---\s*\[%{DATA:thread}\]\s*(?m)(?<msg>.*|\s)

3.进入logstash 按照目录

 bin/logstash -f config/logstash.conf --config.reload.automatic &

3.filebeat配置文件

在filebeat目录下新建filebeat.yml、input-jetty.yml两个文件

filebeat.yml配置如下：

filebeat.config:
  inputs:
    enabled: true
    path: input-jetty.yml
    reload.enabled: true
    reload.period: 10s

output.logstash:
  enabled: true
  hosts: ["127.0.0.1:5044"]
  escape_html: true
  index: 'wkb-jetty'

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat.log
  rotateeverybytes: 104857600
  keepfiles: 7
  permissions: 0644

input-jetty.yml配置如下

- type: log
  enabled: true
  tags: ["jetty"]
  paths:
    - /data/log/jetty/*jetty-custom.log
  multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

4. 启动filebeat

 ./filebeat -e -c filebeat.yml