Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다.
문의사항은 gooper@gooper.com로 메일을
보내주세요.
버젼차이로 인한 프로토콜이 맞지 않아서 서로 엉뚱하게 보내고 엉뚱하게 처리하고 있다.
0.9에서 0.10으로 넘어가면서 api명칭및 패키지 구조, 특히 Properties지정하는 항목이 변경되거나 필수항목이 추가되는 등의 많은 변화가 발생했다.
api는
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
등을 0.9버젼에서 사용했는데... 0.10이상에서는 deplecated되고 org.apache로 시작하는 패키지를 사용한다.
-------------producer쪽 로그-----------------
16/08/18 20:00:17 INFO VerifiableProperties: Verifying properties
16/08/18 20:00:17 INFO VerifiableProperties: Property metadata.broker.list is overridden to sda1:9092,sda2:9092,sda3:9092
16/08/18 20:00:17 INFO VerifiableProperties: Property partitioner.class is overridden to kafka.producer.DefaultPartitioner
16/08/18 20:00:17 INFO VerifiableProperties: Property request.required.acks is overridden to 1
16/08/18 20:00:17 INFO VerifiableProperties: Property serializer.class is overridden to kafka.serializer.DefaultEncoder
Send start (OneM2M)......................
16/08/18 20:00:18 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,sda2,9092) with correlation id 0 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:00:18 INFO SyncProducer: Connected to sda2:9092 for producing
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from sda2:9092
16/08/18 20:00:18 WARN BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic COL_ONEM2M ->
No partition metadata for topic COL_ONEM2M due to org.apache.kafka.common.errors.UnknownTopicOrPartitionException}] for topic [COL_ONEM2M]: class org.apache.kafka.common.errors.UnknownTopicOrPartitionException
16/08/18 20:00:18 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,sda3,9092) with correlation id 1 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:00:18 INFO SyncProducer: Connected to sda3:9092 for producing
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from sda3:9092
16/08/18 20:00:18 WARN BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic COL_ONEM2M ->
No partition metadata for topic COL_ONEM2M due to org.apache.kafka.common.errors.UnknownTopicOrPartitionException}] for topic [COL_ONEM2M]: class org.apache.kafka.common.errors.UnknownTopicOrPartitionException
16/08/18 20:00:18 ERROR DefaultEventHandler: Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: COL_ONEM2M
16/08/18 20:00:18 INFO DefaultEventHandler: Back off for 100 ms before retrying send. Remaining retries = 3
16/08/18 20:00:18 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(0,sda1,9092) with correlation id 2 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:00:18 INFO SyncProducer: Connected to sda1:9092 for producing
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from sda1:9092
16/08/18 20:00:18 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:00:18 WARN DefaultEventHandler: Failed to send producer request with correlation id 4 to broker 3 with data for partitions [COL_ONEM2M,0]
java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:506)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:412)
at kafka.api.ProducerResponse$$anonfun$1$$anonfun$apply$1.apply(ProducerResponse.scala:40)
at kafka.api.ProducerResponse$$anonfun$1$$anonfun$apply$1.apply(ProducerResponse.scala:36)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.api.ProducerResponse$$anonfun$1.apply(ProducerResponse.scala:36)
at kafka.api.ProducerResponse$$anonfun$1.apply(ProducerResponse.scala:33)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at kafka.api.ProducerResponse$.readFrom(ProducerResponse.scala:33)
at kafka.producer.SyncProducer.send(SyncProducer.scala:114)
at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:275)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:113)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:105)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:105)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)
at kafka.producer.Producer.send(Producer.scala:78)
at kafka.javaapi.producer.Producer.send(Producer.scala:35)
at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.send(AvroOneM2MEmitter.java:59)
at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.main(AvroOneM2MEmitter.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/08/18 20:00:18 INFO DefaultEventHandler: Back off for 100 ms before retrying send. Remaining retries = 2
16/08/18 20:00:18 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,sda3,9092) with correlation id 5 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:00:18 INFO SyncProducer: Connected to sda3:9092 for producing
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from sda3:9092
16/08/18 20:00:18 WARN BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic COL_ONEM2M ->
No partition metadata for topic COL_ONEM2M due to org.apache.kafka.common.errors.UnknownTopicOrPartitionException}] for topic [COL_ONEM2M]: class org.apache.kafka.common.errors.UnknownTopicOrPartitionException
16/08/18 20:00:18 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:00:18 WARN DefaultEventHandler: Failed to send producer request with correlation id 7 to broker 2 with data for partitions [COL_ONEM2M,2]
java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:506)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:412)
at kafka.api.ProducerResponse$$anonfun$1$$anonfun$apply$1.apply(ProducerResponse.scala:40)
at kafka.api.ProducerResponse$$anonfun$1$$anonfun$apply$1.apply(ProducerResponse.scala:36)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.api.ProducerResponse$$anonfun$1.apply(ProducerResponse.scala:36)
at kafka.api.ProducerResponse$$anonfun$1.apply(ProducerResponse.scala:33)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at kafka.api.ProducerResponse$.readFrom(ProducerResponse.scala:33)
at kafka.producer.SyncProducer.send(SyncProducer.scala:114)
at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:275)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:113)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:105)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:105)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)
at kafka.producer.Producer.send(Producer.scala:78)
at kafka.javaapi.producer.Producer.send(Producer.scala:35)
at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.send(AvroOneM2MEmitter.java:59)
at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.main(AvroOneM2MEmitter.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/08/18 20:00:18 INFO DefaultEventHandler: Back off for 100 ms before retrying send. Remaining retries = 1
16/08/18 20:00:18 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,sda2,9092) with correlation id 8 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:00:18 INFO SyncProducer: Connected to sda2:9092 for producing
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from sda2:9092
16/08/18 20:00:18 WARN BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic COL_ONEM2M ->
No partition metadata for topic COL_ONEM2M due to org.apache.kafka.common.errors.UnknownTopicOrPartitionException}] for topic [COL_ONEM2M]: class org.apache.kafka.common.errors.UnknownTopicOrPartitionException
16/08/18 20:00:18 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:00:18 WARN DefaultEventHandler: Failed to send producer request with correlation id 10 to broker 1 with data for partitions [COL_ONEM2M,1]
java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:506)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:412)
at kafka.api.ProducerResponse$$anonfun$1$$anonfun$apply$1.apply(ProducerResponse.scala:40)
at kafka.api.ProducerResponse$$anonfun$1$$anonfun$apply$1.apply(ProducerResponse.scala:36)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.api.ProducerResponse$$anonfun$1.apply(ProducerResponse.scala:36)
at kafka.api.ProducerResponse$$anonfun$1.apply(ProducerResponse.scala:33)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at kafka.api.ProducerResponse$.readFrom(ProducerResponse.scala:33)
at kafka.producer.SyncProducer.send(SyncProducer.scala:114)
at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:275)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:113)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:105)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:105)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)
at kafka.producer.Producer.send(Producer.scala:78)
at kafka.javaapi.producer.Producer.send(Producer.scala:35)
at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.send(AvroOneM2MEmitter.java:59)
at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.main(AvroOneM2MEmitter.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/08/18 20:00:18 INFO DefaultEventHandler: Back off for 100 ms before retrying send. Remaining retries = 0
16/08/18 20:00:18 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,sda3,9092) with correlation id 11 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:00:18 INFO SyncProducer: Connected to sda3:9092 for producing
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from sda3:9092
16/08/18 20:00:18 WARN BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic COL_ONEM2M ->
No partition metadata for topic COL_ONEM2M due to org.apache.kafka.common.errors.UnknownTopicOrPartitionException}] for topic [COL_ONEM2M]: class org.apache.kafka.common.errors.UnknownTopicOrPartitionException
16/08/18 20:00:18 ERROR DefaultEventHandler: Failed to send requests for topics COL_ONEM2M with correlation ids in [0,11]
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:96)
at kafka.producer.Producer.send(Producer.scala:78)
at kafka.javaapi.producer.Producer.send(Producer.scala:35)
at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.send(AvroOneM2MEmitter.java:59)
at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.main(AvroOneM2MEmitter.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Send end(OneM2M)......................
16/08/18 20:00:18 INFO Producer: Shutting down producer
16/08/18 20:00:18 INFO ProducerPool: Closing all sync producers
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:00:18 INFO Producer: Producer shutdown completed in 7 ms
----------------------------------consumer로그-----------------
16/08/18 20:04:10 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:10 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:10 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:10 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:10 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(3,localhost,9092) with correlation id 1614 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:10 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:10 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:10 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:10 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:10 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:10 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:10 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:10 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:10 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:10 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(3,localhost,9092) with correlation id 1615 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:10 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:10 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:10 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:10 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:10 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:10 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:10 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:10 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:10 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:10 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,localhost,9092) with correlation id 1616 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:10 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:10 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:10 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:10 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:10 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:11 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:11 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:11 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:11 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,localhost,9092) with correlation id 1617 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:11 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:11 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:11 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:11 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:11 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:11 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:11 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(3,localhost,9092) with correlation id 1618 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:11 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:11 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:11 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:11 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:11 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:11 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:11 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,localhost,9092) with correlation id 1619 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:11 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:11 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:11 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:11 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:11 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:11 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:11 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,localhost,9092) with correlation id 1620 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:11 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:11 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:11 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:11 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:11 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:11 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:11 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,localhost,9092) with correlation id 1621 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:11 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:11 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:11 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:12 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:12 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:12 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:12 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(3,localhost,9092) with correlation id 1622 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:12 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:12 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:12 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:12 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:12 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:12 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:12 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,localhost,9092) with correlation id 1623 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:12 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:12 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:12 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:12 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:12 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:12 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:12 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,localhost,9092) with correlation id 1624 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:12 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:12 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:12 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:12 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:12 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:12 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:12 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,localhost,9092) with correlation id 1625 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:12 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:12 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:12 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:12 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:12 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:12 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:12 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(3,localhost,9092) with correlation id 1626 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:12 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:12 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:12 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
^C16/08/18 20:04:13 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:13 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:13 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:13 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:13 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,localhost,9092) with correlation id 1627 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:13 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:13 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:13 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:13 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:13 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
댓글 0
번호 | 제목 | 글쓴이 | 날짜 | 조회 수 |
---|---|---|---|---|
39 | 동일서버에서 LA와 LC동시에 기동하여 테스트 | 총관리자 | 2014.04.01 | 936 |
38 | 다수의 로그 에이전트로 부터 로그를 받아 각각의 파일로 저장하는 방법(interceptor및 multiplexing) | 총관리자 | 2014.04.04 | 4095 |
37 | sqoop 1.4.4 설치및 테스트 | 총관리자 | 2014.04.21 | 3139 |
36 | sqoop export/import등을 할때 driver를 못찾는 오류가 발생하면... | 총관리자 | 2014.05.15 | 872 |
35 | sqoop에서 oracle관련 작업할때 테이블명, 사용자명, DB명은 모두 대문자로 사용할것 | 총관리자 | 2014.05.15 | 1536 |
34 | source의 type을 spooldir로 하는 경우 해당 경로에 파일이 들어오면 파일단위로 전송함 | 총관리자 | 2014.05.20 | 689 |
33 | import 혹은 export할때 hive파일의 default 구분자는 --input-fields-terminated-by " |