메뉴 건너뛰기

검색창 닫기

Cloudera, BigData, Semantic IoT, Hadoop, NoSQL

Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.


flume 동일서버에서 LA와 LC동시에 기동하여 테스트

총관리자 2014.04.01 18:34 조회 수 : 1105

1. 로그를 받을 agent설정 파일(flume-conf.properties)

agent.sources = avroGenSrc
agent.channels = memoryChannel
agent.sinks = fileSink

# For each one of the sources, the type is defined
agent.sources.avroGenSrc.type = avro
agent.sources.avroGenSrc.bind = localhost
agent.sources.avroGenSrc.port = 3333

# The channel can be defined as follows.
agent.sources.avroGenSrc.channels = memoryChannel

# Each sink's type must be defined
agent.sinks.fileSink.type = file_roll
agent.sinks.fileSink.sink.directory=/home/hadoop/saved_data
agent.sinks.fileSink.sink.rollInterval = 10
agent.sinks.fileSink.sink.batchSize = 10

#Specify the channel the sink should use
agent.sinks.fileSink.channel = memoryChannel

# Each channel's type is defined.
agent.channels.memoryChannel.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 10000
agent.channels.memoryChannel.transctionCapacity = 10000


2. 로그를 전송하는 agent(flume-conf-agent01.properties)


agent01.sources = execGenSrc
agent01.channels = memoryChannel
agent01.sinks = avroSink

# For each one of the sources, the type is defined
agent01.sources.execGenSrc.type = exec
agent01.sources.execGenSrc.command = tail -f /home/hadoop/log_data/log1.log
agent01.sources.execGenSrc.batchSize = 10

# The channel can be defined as follows.
agent01.sources.execGenSrc.channels = memoryChannel

# Each sink's type must be defined
agent01.sinks.avroSink.type = avro
agent01.sinks.avroSink.hostname=localhost
agent01.sinks.avroSink.port=3333
agent01.sinks.avroSink.batch-size = 10

#Specify the channel the sink should use
agent01.sinks.avroSink.channel = memoryChannel

# Each channel's type is defined.
agent01.channels.memoryChannel.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent01.channels.memoryChannel.capacity = 10000
agent01.channels.memoryChannel.transctionCapacity = 10000


3. log를 받을 폴더 생성및 전송할 데이타 생성

   가. 받을 폴더 : mkdir /home/hadoop/flume/saved_data

   나. 보낼 폴더 : mkdir /home/hadoop/flume/log_data

   다. 보낼파일 :  touch /home/hadoop/flume/log1.log

   라. 폴더로 이동 : cd /home/hadoop/flume

   마. log1.log에 간단하게 로그 추가 : echo "aaaaabbbbbcccc" >> log1.log

 

4. agent실행

  가. 로그를 받을 agent : flume-ng agent --conf-file ./conf/flume-conf.properties --name agent

  나. 로그를 전송할 agent : flume-ng agent --conf-file ./conf/flume-conf-agent01.properties --name agent01

이 게시물을
이 글의 추천인 목록 목록
쓰기 태그
위로