Summary of Logstash log data write exception troubleshooting

Background and problem analysis

Write the logs of multiple services to Logstash through Filebeat, where Filebeat is deployed in a SideCar mode container, and now it is necessary to write data from multiple sources into an ES index by means of tags.
My steps:
[1] Modify the configMap configuration corresponding to fileBeat on K8S
[2] Modify the logstash configuration, first copy a configuration backup, and then add additional tagsLabel.

Question one

logstash[]: java.util.concurrent.RejectedExecutionException: event executor terminated

This problem occurs after startup. The reason for the above problem is that I cp a configuration file in the logstash configuration directory, which eventually leads to multiple conf files in the configuration directory.This needs to be noted that multiple configuration files are not allowed in the configuration directory. If you need to backup, you can put them in an external directory.
Problem 2
After solving this problem, I found another problem, only the log data of one of the services was seen in the ES index, the data of other services was not stored in the database, and the service log data that could have been stored in the database was alsoNot in stock.

Check the FileBeat service, there is no problem with normal operation.

Check Logstash service and observe its log data

[logstash.outputs.elasticsearch] Could not indexevent to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-info-2022.07.13", :_type=>"_doc", :routing=>nil},#], :response=>{"index"=>{"_index"=>"logstash-info-2022.07.13", "_type"=>"_doc", "_id"=>"LzhC9oEB0ca6UQlyyblY", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [message] of type [text] in document with id 'LzhC9oEB0ca6UQlyyblY'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=span>>"Can't get text on a START_OBJECT at 1:378"}}}}}

Some logs can be output normally, but some log data have warnings as shown in the figure above.It can be seen that the reason for the error is that the type of message does not match when it is mapped to the ES index.Considering the normal operation of the previous log, compare the currently inserted index record with the previous index record,

The above picture is a comparison of the index records that can be stored in yesterday and currently, and it can be found that the storage structure of the two messages is different.Because the data types of the indexed fields are inconsistent, two logs with different JSON formats cannot be mapped to one index.

Because the log of the current application is customized and needs to be written in a unified format, and this problematic service is an old project before, and it is not written in a unified log format, so such a mess will beThere is a problem with the logs being written together.

Resolution reference

[1] Back up the current log index data, then delete the index data and recreate it, and cancel the services that are not output according to the unified format log into the current index. After processing, the problem is solved, and the log falls into the index normally.It is very rude to delete the index here, because this is only half a day's log data, non-business information, and has been backed up.
[2] If the index cannot be rebuilt, then only the log data record of the corresponding service can be deleted, and then the log is extracted.
[3] Another reason for this problem is that the data from logstash to ES index is dynamically mapped to generate an index (table) structure. If it is data with strict requirements, we can set the index template for our project.The storage of personnel information in the database uses the index template method, which restricts the data format from the index (table) structure.