The Tips

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.I already have installed many flume systems to collect streaming log data. But I found some problems when we used flume. I write this blog to record the problems and solutions. and anybody else will avoid such problem.

The running environment

• CDH version 5.8.0+
• Flume 1.6.0+
• Java 1.7.0+
• Linux 2.6.32-573.el6.x86_6
• Centos 6.6+

Tips one :rotating invalid

flume by using the following configuration uploads and rotates files to hadoop ...

View comments.

The Problem

Today, My colleague found an abnormal problem and asked me the reason. I recorded the following analysis steps.

This problem is that linux du command does not print the results, and at the same time, the process of du command occupies 100% CPU usage. Result of top command is the following:

# top
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                    
36343 root      20   0  110m  13m 1780 R 99.8  0.0   1:37.66 du                                                                                                         
36370 root      20   0  110m  13m 1780 R 94.2  0.0   1:20.98 du
   86 root ...

View comments.

The Problem

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.I already have installed many flume systems to collect streaming log data. But I found one problem that flume process was occuping more virtual memories and fewer physical memories. Why is this situation occurred? This blog explains and provides the methods to solve this problem.

The running environment

• CDH version 5.8.0+
• Flume 1.6.0+
• Java 1.7.0+
• Linux 2.6.32-573.el6.x86_6
• Centos 6.6+

The Top command Result of flume process

26963 root ...

View comments.