当前位置：网站首页>How can flume solve the problem of too many small files when collecting

How can flume solve the problem of too many small files when collecting

2022-04-22 08:50:00 【The figure under the stars】

  1. HDFS Store a lot of small files , What's the impact? ？
    (1) At the metadata level ： Every little file has a metadata , This includes the file path 、 file name 、 owner 、 Subordinate to the group 、 jurisdiction 、 Creation time, etc , This information is stored in namenode in . therefore , Too many small files , Will occupy namenode The server has a lot of memory , influence namenode Performance and service life of ;
    (2) At the computational level ： By default ,MR Will start a for each small file map Task calculation , It greatly affects the computing performance ; It also affects the disk addressing time .
  2. How to solve the problem of too many small files
    stay flume Set three parameters in the configuration ：
    (1) hdfs.rollInterval： When the file is created for more than a few seconds, it will scroll to generate a new file
    (2) hdfs.rollSize： When the file reaches how many bytes, it will scroll to generate a new file
    (3) hdfs.rollCount： When event When the number reaches how many, it will scroll to generate new files

版权声明
本文为[The figure under the stars]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/04/202204220750015418.html

边栏推荐

猜你喜欢

随机推荐