You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Foo Lim <fo...@vungle.com> on 2014/12/27 00:30:06 UTC
1.6 release date?
Hi all,
Happy holidays! Just wondering if there's any ETA on a 1.6 release.
Looking forward to the kafka sink plugin that I can't get to compile
independently. :-/
Thanks!
Re: 1.6 release date?
Posted by Foo Lim <fo...@vungle.com>.
Hi Frank,
Thanks for the reply. I have other topics in kafka that are working:
~/kafka_2.9.2-0.8.1.1$ bin/kafka-topics.sh --zookeeper localhost --describe
Topic:requests PartitionCount:1 ReplicationFactor:1 Configs:
Topic: requests Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
test
tst1
$ bin/kafka-console-consumer.sh --zookeeper localhost --topic test
--from-beginning
test
tst1
Am I supposed to create the 'custom-topic' with 2 partitions before
running 'mvn clean install'?
Let's try creating the 'custom-topic':
$ bin/kafka-topics.sh --create --partitions 2 --topic custom-topic
--zookeeper localhost --replication-factor 1
Created topic "custom-topic".
$ bin/kafka-topics.sh --zookeeper localhost --describe --topic custom-topic
Topic:custom-topic PartitionCount:2 ReplicationFactor:1 Configs:
Topic: custom-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: custom-topic Partition: 1 Leader: 0 Replicas: 0 Isr: 0
$ cd ~/flume-ng-kafka-sink
$ mvn clean install
[...]
[2014-12-30 02:34:38,520] INFO Fetching metadata from broker
id:0,host:vagrant-ubuntu-precise-64,port:51064 with correlation id 26
for 1 topic(s) Set(custom-topic) (kafka.client.ClientUtils$)
[2014-12-30 02:34:38,521] INFO Connected to
vagrant-ubuntu-precise-64:51064 for producing
(kafka.producer.SyncProducer)
[2014-12-30 02:34:38,525] INFO Disconnecting from
vagrant-ubuntu-precise-64:51064 (kafka.producer.SyncProducer)
[2014-12-30 02:34:38,526] INFO Closing socket connection to
/10.0.2.15. (kafka.network.Processor)
[2014-12-30 02:34:38,528] WARN [KafkaApi-0] Offset request with
correlation id 0 from client
group_1-ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419906871686-f7db9c60-0-0
on partition [custom-topic,1] failed due to Leader not local for
partition [custom-topic,1] on broker 0 (kafka.server.KafkaApis)
[2014-12-30 02:34:38,530] WARN
[group_1_vagrant-ubuntu-precise-64-1419906871686-f7db9c60-leader-finder-thread],
Failed to add leader for partitions [custom-topic,1],[custom-topic,0];
will retry (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.NotLeaderForPartitionException
at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:379)
at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:73)
at kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:160)
at kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
at kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
at kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at scala.collection.immutable.Map$Map2.foreach(Map.scala:130)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
at kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
at kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[2014-12-30 02:34:38,536] INFO
[ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419906871686-f7db9c60-0-0],
Shutting down (kafka.consumer.ConsumerFetcherThread)
[2014-12-30 02:34:38,528] INFO
[ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419906871686-f7db9c60-0-0],
Starting (kafka.consumer.ConsumerFetcherThread)
[2014-12-30 02:34:38,539] INFO
[ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419906871686-f7db9c60-0-0],
Shutdown completed (kafka.consumer.ConsumerFetcherThread)
On Mon, Dec 29, 2014 at 3:53 PM, Frank Yao <ba...@gmail.com> wrote:
> hi foo
>
> it seems your stack showed exception was caused by kafka itself
>
> Failed to add leader for partitions
>
>
> I have used kafka sink and source of flume 1.6 for several weeks and it
> works well.
>
> Could you please try to use kafka console producer first to test if the
> partitionis okay or not?
> Frank Yao
> @Vipshop, Shanghai
> from iPhone
>
> 在 2014年12月30日，04:21，Foo Lim <fo...@vungle.com> 写道：
>
> BTW, I followed the directions & ran
>
> ~/flume-ng-kafka-sink$ mvn clean install
>
> On Mon, Dec 29, 2014 at 12:10 PM, Foo Lim <fo...@vungle.com> wrote:
>
> Hi Gwen,
>
>
> Thanks for the reply.
>
>
> I'll try the CDH jar file. Where do I put it in the flume directory
> structure?
>
>
> I have kafka_2.9.2-0.8.1.1 running. Here's the test error (which keeps
>
> repeating) in the project
>
> git@github.com:thilinamb/flume-ng-kafka-sink.git
>
>
> [2014-12-29 20:02:34,028] INFO Verifying properties
>
> (kafka.utils.VerifiableProperties)
>
> [2014-12-29 20:02:34,029] INFO Property client.id is overridden to
>
> group_1 (kafka.utils.VerifiableProperties)
>
> [2014-12-29 20:02:34,030] INFO Property metadata.broker.list is
>
> overridden to vagrant-ubuntu-precise-64:50753
>
> (kafka.utils.VerifiableProperties)
>
> [2014-12-29 20:02:34,030] INFO Property request.timeout.ms is
>
> overridden to 30000 (kafka.utils.VerifiableProperties)
>
> [2014-12-29 20:02:34,031] INFO Fetching metadata from broker
>
> id:0,host:vagrant-ubuntu-precise-64,port:50753 with correlation id 18
>
> for 1 topic(s) Set(custom-topic) (kafka.client.ClientUtils$)
>
> [2014-12-29 20:02:34,032] INFO Connected to
>
> vagrant-ubuntu-precise-64:50753 for producing
>
> (kafka.producer.SyncProducer)
>
> [2014-12-29 20:02:34,035] INFO Disconnecting from
>
> vagrant-ubuntu-precise-64:50753 (kafka.producer.SyncProducer)
>
> [2014-12-29 20:02:34,036] INFO Closing socket connection to
>
> /10.0.2.15. (kafka.network.Processor)
>
> [2014-12-29 20:02:34,038] WARN [KafkaApi-0] Offset request with
>
> correlation id 0 from client
>
> group_1-ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0
>
> on partition [custom-topic,1] failed due to Leader not local for
>
> partition [custom-topic,1] on broker 0 (kafka.server.KafkaApis)
>
> [2014-12-29 20:02:34,040] WARN
>
> [group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-leader-finder-thread],
>
> Failed to add leader for partitions [custom-topic,1],[custom-topic,0];
>
> will retry (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
>
> kafka.common.NotLeaderForPartitionException
>
> at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>
> at java.lang.Class.newInstance(Class.java:379)
>
> at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:73)
>
> at
> kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:160)
>
> at
> kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
>
> at
> kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
>
> at
> kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
>
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:130)
>
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>
> at
> kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
>
> at
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
>
> at
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
>
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
>
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>
> at
> kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
>
> at
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
>
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>
> [2014-12-29 20:02:34,045] INFO
>
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>
> Shutting down (kafka.consumer.ConsumerFetcherThread)
>
> [2014-12-29 20:02:34,039] INFO
>
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>
> Starting (kafka.consumer.ConsumerFetcherThread)
>
> [2014-12-29 20:02:34,046] INFO
>
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>
> Shutdown completed (kafka.consumer.ConsumerFetcherThread)
>
> [2014-12-29 20:02:34,047] INFO Closing socket connection to
>
> /10.0.2.15. (kafka.network.Processor)
>
> [2014-12-29 20:02:34,048] INFO
>
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>
> Stopped (kafka.consumer.ConsumerFetcherThread)
>
>
> On Fri, Dec 26, 2014 at 4:06 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>
> I can't say when's the 1.6 release, but I have other solutions :)
>
>
> 1. The packages that are part of CDH5.3 release will contain that jar.
>
> Perhaps use this distro? Or even just get the RPM, unpackage and dig the jar
>
> out?
>
> 2. Let us know what's the compilation error, perhaps we can help there?
>
>
> On Fri, Dec 26, 2014 at 3:30 PM, Foo Lim <fo...@vungle.com> wrote:
>
>
> Hi all,
>
>
> Happy holidays! Just wondering if there's any ETA on a 1.6 release.
>
> Looking forward to the kafka sink plugin that I can't get to compile
>
> independently. :-/
>
>
> Thanks!
>
>
>
Re: 1.6 release date?
Posted by Gwen Shapira <gs...@cloudera.com>.
doesn't look like they will overlap. KafkaSinkTest should use:
<currentdir>/target/kafka-logs
On Mon, Jan 5, 2015 at 5:02 PM, Foo Lim <fo...@vungle.com> wrote:
> Hi Gwen,
>
> Kafka:
> log.dirs=/tmp/kafka-logs
> (the default)
>
> ZK:
> dataDir=/vagrant/zookeeper-3.4.6/data
> (changed from default)
>
> Thanks
>
>
> On Mon, Jan 5, 2015 at 4:39 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>>
>> mmmm... normally traces of Kafka or ZK are not an issue. Can you share
>> which directories does Kafka use for logs? (i.e. log.dir in Kafka
>> configuration)? And where does ZK store its data?
>>
>> Thanks,
>> Gwen
>>
>> On Mon, Jan 5, 2015 at 11:43 AM, Foo Lim <fo...@vungle.com> wrote:
>> > Thanks Gwen!
>> >
>> > -DskipTests worked! It built
>> > "dist/target/flume-kafka-sink-dist-0.5.0-bin.zip". I didn't know this
>> > was an
>> > option.
>> >
>> > The README said
>> > ## Dependency Versions
>> > - Apache Flume - 1.5.0
>> > - Apache Kafka - 0.8.1.1
>> >
>> > ## Prerequisites
>> > - Java 1.6 or higher
>> > - [Apache Maven 3](http://maven.apache.org)
>> > - An [Apache Flume](https://flume.apache.org) installation (See the
>> > dependent version above)
>> > - An [Apache Kafka](http://kafka.apache.org) installation (See the
>> > dependent
>> > version above)
>> >
>> > so I thought I had to have both running.
>> >
>> > Anyway, I stopped both kafka & zookeeper, but I'm still getting the same
>> > error
>> >
>> > kafka.common.NotLeaderForPartitionException
>> >
>> > Does my machine have to NOT have any traces of kafka or ZK?
>> >
>> >
>> > Now, on to testing if this sink works..
>> >
>> > Thanks again!
>> >
>> > Foo
>> >
>> >
>> > On Sun, Jan 4, 2015 at 5:19 PM, Gwen Shapira <gs...@cloudera.com>
>> > wrote:
>> >>
>> >> I think the only thing that's failing in the build are the unit tests.
>> >>
>> >> 1. To build without unit tests: "mvn clean install -DskipTests"
>> >> 2. I suspect you are building Flume on a machine that has either Kafka
>> >> or Zookeeper installed, and we don't randomize ports properly in the
>> >> tests, therefore creating a mess. You can build on a dev machine not
>> >> running Kafka and Zookeeper. I'll create a patch making sure we don't
>> >> accidentally pick used ports for tests.
>> >>
>> >> Gwen
>> >>
>> >> On Mon, Dec 29, 2014 at 7:31 PM, Frank Yao <ba...@gmail.com> wrote:
>> >> > you can get source code from 1.6 repo and modify pom.xml to adjust
>> >> > code
>> >> > to
>> >> > be compitable with 1.5. I have done this once and it works well
>> >> > Frank Yao
>> >> > @Vipshop, Shanghai
>> >> > from iPhone
>> >> >
>> >> > 在 2014年12月30日，10:47，Foo Lim <fo...@vungle.com> 写道：
>> >> >
>> >> > Hi again Frank,
>> >> >
>> >> > You are using flume 1.6. I'm trying to get the sink running on 1.5.2
>> >> > for a production machine. Thanks..
>> >> >
>> >> > On Mon, Dec 29, 2014 at 3:53 PM, Frank Yao <ba...@gmail.com>
>> >> > wrote:
>> >> >
>> >> > hi foo
>> >> >
>> >> >
>> >> > it seems your stack showed exception was caused by kafka itself
>> >> >
>> >> >
>> >> > Failed to add leader for partitions
>> >> >
>> >> >
>> >> >
>> >> > I have used kafka sink and source of flume 1.6 for several weeks and
>> >> > it
>> >> >
>> >> > works well.
>> >> >
>> >> >
>> >> > Could you please try to use kafka console producer first to test if
>> >> > the
>> >> >
>> >> > partitionis okay or not?
>> >> >
>> >> > Frank Yao
>> >> >
>> >> > @Vipshop, Shanghai
>> >> >
>> >> > from iPhone
>> >> >
>> >> >
>> >> > 在 2014年12月30日，04:21，Foo Lim <fo...@vungle.com> 写道：
>> >> >
>> >> >
>> >> > BTW, I followed the directions & ran
>> >> >
>> >> >
>> >> > ~/flume-ng-kafka-sink$ mvn clean install
>> >> >
>> >> >
>> >> > On Mon, Dec 29, 2014 at 12:10 PM, Foo Lim <fo...@vungle.com> wrote:
>> >> >
>> >> >
>> >> > Hi Gwen,
>> >> >
>> >> >
>> >> >
>> >> > Thanks for the reply.
>> >> >
>> >> >
>> >> >
>> >> > I'll try the CDH jar file. Where do I put it in the flume directory
>> >> >
>> >> > structure?
>> >> >
>> >> >
>> >> >
>> >> > I have kafka_2.9.2-0.8.1.1 running. Here's the test error (which
>> >> > keeps
>> >> >
>> >> >
>> >> > repeating) in the project
>> >> >
>> >> >
>> >> > git@github.com:thilinamb/flume-ng-kafka-sink.git
>> >> >
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,028] INFO Verifying properties
>> >> >
>> >> >
>> >> > (kafka.utils.VerifiableProperties)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,029] INFO Property client.id is overridden to
>> >> >
>> >> >
>> >> > group_1 (kafka.utils.VerifiableProperties)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,030] INFO Property metadata.broker.list is
>> >> >
>> >> >
>> >> > overridden to vagrant-ubuntu-precise-64:50753
>> >> >
>> >> >
>> >> > (kafka.utils.VerifiableProperties)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,030] INFO Property request.timeout.ms is
>> >> >
>> >> >
>> >> > overridden to 30000 (kafka.utils.VerifiableProperties)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,031] INFO Fetching metadata from broker
>> >> >
>> >> >
>> >> > id:0,host:vagrant-ubuntu-precise-64,port:50753 with correlation id 18
>> >> >
>> >> >
>> >> > for 1 topic(s) Set(custom-topic) (kafka.client.ClientUtils$)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,032] INFO Connected to
>> >> >
>> >> >
>> >> > vagrant-ubuntu-precise-64:50753 for producing
>> >> >
>> >> >
>> >> > (kafka.producer.SyncProducer)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,035] INFO Disconnecting from
>> >> >
>> >> >
>> >> > vagrant-ubuntu-precise-64:50753 (kafka.producer.SyncProducer)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,036] INFO Closing socket connection to
>> >> >
>> >> >
>> >> > /10.0.2.15. (kafka.network.Processor)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,038] WARN [KafkaApi-0] Offset request with
>> >> >
>> >> >
>> >> > correlation id 0 from client
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > group_1-ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0
>> >> >
>> >> >
>> >> > on partition [custom-topic,1] failed due to Leader not local for
>> >> >
>> >> >
>> >> > partition [custom-topic,1] on broker 0 (kafka.server.KafkaApis)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,040] WARN
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > [group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-leader-finder-thread],
>> >> >
>> >> >
>> >> > Failed to add leader for partitions
>> >> > [custom-topic,1],[custom-topic,0];
>> >> >
>> >> >
>> >> > will retry (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
>> >> >
>> >> >
>> >> > kafka.common.NotLeaderForPartitionException
>> >> >
>> >> >
>> >> > at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown
>> >> > Source)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> >> >
>> >> >
>> >> > at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> >> >
>> >> >
>> >> > at java.lang.Class.newInstance(Class.java:379)
>> >> >
>> >> >
>> >> > at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:73)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:160)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>> >> >
>> >> >
>> >> > at scala.collection.immutable.Map$Map2.foreach(Map.scala:130)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>> >> >
>> >> >
>> >> > at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
>> >> >
>> >> >
>> >> > at
>> >> >
>> >> >
>> >> >
>> >> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
>> >> >
>> >> >
>> >> > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,045] INFO
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>> >> >
>> >> >
>> >> > Shutting down (kafka.consumer.ConsumerFetcherThread)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,039] INFO
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>> >> >
>> >> >
>> >> > Starting (kafka.consumer.ConsumerFetcherThread)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,046] INFO
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>> >> >
>> >> >
>> >> > Shutdown completed (kafka.consumer.ConsumerFetcherThread)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,047] INFO Closing socket connection to
>> >> >
>> >> >
>> >> > /10.0.2.15. (kafka.network.Processor)
>> >> >
>> >> >
>> >> > [2014-12-29 20:02:34,048] INFO
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>> >> >
>> >> >
>> >> > Stopped (kafka.consumer.ConsumerFetcherThread)
>> >> >
>> >> >
>> >> >
>> >> > On Fri, Dec 26, 2014 at 4:06 PM, Gwen Shapira <gs...@cloudera.com>
>> >> > wrote:
>> >> >
>> >> >
>> >> > I can't say when's the 1.6 release, but I have other solutions :)
>> >> >
>> >> >
>> >> >
>> >> > 1. The packages that are part of CDH5.3 release will contain that
>> >> > jar.
>> >> >
>> >> >
>> >> > Perhaps use this distro? Or even just get the RPM, unpackage and dig
>> >> > the
>> >> > jar
>> >> >
>> >> >
>> >> > out?
>> >> >
>> >> >
>> >> > 2. Let us know what's the compilation error, perhaps we can help
>> >> > there?
>> >> >
>> >> >
>> >> >
>> >> > On Fri, Dec 26, 2014 at 3:30 PM, Foo Lim <fo...@vungle.com> wrote:
>> >> >
>> >> >
>> >> >
>> >> > Hi all,
>> >> >
>> >> >
>> >> >
>> >> > Happy holidays! Just wondering if there's any ETA on a 1.6 release.
>> >> >
>> >> >
>> >> > Looking forward to the kafka sink plugin that I can't get to compile
>> >> >
>> >> >
>> >> > independently. :-/
>> >> >
>> >> >
>> >> >
>> >> > Thanks!
>> >> >
>> >> >
>> >> >
>> >> >
>> >
>> >
>
>
Re: 1.6 release date?
Posted by Foo Lim <fo...@vungle.com>.
Hi Gwen,
Kafka:
log.dirs=/tmp/kafka-logs
(the default)
ZK:
dataDir=/vagrant/zookeeper-3.4.6/data
(changed from default)
Thanks
On Mon, Jan 5, 2015 at 4:39 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> mmmm... normally traces of Kafka or ZK are not an issue. Can you share
> which directories does Kafka use for logs? (i.e. log.dir in Kafka
> configuration)? And where does ZK store its data?
>
> Thanks,
> Gwen
>
> On Mon, Jan 5, 2015 at 11:43 AM, Foo Lim <fo...@vungle.com> wrote:
> > Thanks Gwen!
> >
> > -DskipTests worked! It built
> > "dist/target/flume-kafka-sink-dist-0.5.0-bin.zip". I didn't know this
> was an
> > option.
> >
> > The README said
> > ## Dependency Versions
> > - Apache Flume - 1.5.0
> > - Apache Kafka - 0.8.1.1
> >
> > ## Prerequisites
> > - Java 1.6 or higher
> > - [Apache Maven 3](http://maven.apache.org)
> > - An [Apache Flume](https://flume.apache.org) installation (See the
> > dependent version above)
> > - An [Apache Kafka](http://kafka.apache.org) installation (See the
> dependent
> > version above)
> >
> > so I thought I had to have both running.
> >
> > Anyway, I stopped both kafka & zookeeper, but I'm still getting the same
> > error
> >
> > kafka.common.NotLeaderForPartitionException
> >
> > Does my machine have to NOT have any traces of kafka or ZK?
> >
> >
> > Now, on to testing if this sink works..
> >
> > Thanks again!
> >
> > Foo
> >
> >
> > On Sun, Jan 4, 2015 at 5:19 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
> >>
> >> I think the only thing that's failing in the build are the unit tests.
> >>
> >> 1. To build without unit tests: "mvn clean install -DskipTests"
> >> 2. I suspect you are building Flume on a machine that has either Kafka
> >> or Zookeeper installed, and we don't randomize ports properly in the
> >> tests, therefore creating a mess. You can build on a dev machine not
> >> running Kafka and Zookeeper. I'll create a patch making sure we don't
> >> accidentally pick used ports for tests.
> >>
> >> Gwen
> >>
> >> On Mon, Dec 29, 2014 at 7:31 PM, Frank Yao <ba...@gmail.com> wrote:
> >> > you can get source code from 1.6 repo and modify pom.xml to adjust
> code
> >> > to
> >> > be compitable with 1.5. I have done this once and it works well
> >> > Frank Yao
> >> > @Vipshop, Shanghai
> >> > from iPhone
> >> >
> >> > 在 2014年12月30日，10:47，Foo Lim <fo...@vungle.com> 写道：
> >> >
> >> > Hi again Frank,
> >> >
> >> > You are using flume 1.6. I'm trying to get the sink running on 1.5.2
> >> > for a production machine. Thanks..
> >> >
> >> > On Mon, Dec 29, 2014 at 3:53 PM, Frank Yao <ba...@gmail.com>
> wrote:
> >> >
> >> > hi foo
> >> >
> >> >
> >> > it seems your stack showed exception was caused by kafka itself
> >> >
> >> >
> >> > Failed to add leader for partitions
> >> >
> >> >
> >> >
> >> > I have used kafka sink and source of flume 1.6 for several weeks and
> it
> >> >
> >> > works well.
> >> >
> >> >
> >> > Could you please try to use kafka console producer first to test if
> the
> >> >
> >> > partitionis okay or not?
> >> >
> >> > Frank Yao
> >> >
> >> > @Vipshop, Shanghai
> >> >
> >> > from iPhone
> >> >
> >> >
> >> > 在 2014年12月30日，04:21，Foo Lim <fo...@vungle.com> 写道：
> >> >
> >> >
> >> > BTW, I followed the directions & ran
> >> >
> >> >
> >> > ~/flume-ng-kafka-sink$ mvn clean install
> >> >
> >> >
> >> > On Mon, Dec 29, 2014 at 12:10 PM, Foo Lim <fo...@vungle.com> wrote:
> >> >
> >> >
> >> > Hi Gwen,
> >> >
> >> >
> >> >
> >> > Thanks for the reply.
> >> >
> >> >
> >> >
> >> > I'll try the CDH jar file. Where do I put it in the flume directory
> >> >
> >> > structure?
> >> >
> >> >
> >> >
> >> > I have kafka_2.9.2-0.8.1.1 running. Here's the test error (which keeps
> >> >
> >> >
> >> > repeating) in the project
> >> >
> >> >
> >> > git@github.com:thilinamb/flume-ng-kafka-sink.git
> >> >
> >> >
> >> >
> >> > [2014-12-29 20:02:34,028] INFO Verifying properties
> >> >
> >> >
> >> > (kafka.utils.VerifiableProperties)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,029] INFO Property client.id is overridden to
> >> >
> >> >
> >> > group_1 (kafka.utils.VerifiableProperties)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,030] INFO Property metadata.broker.list is
> >> >
> >> >
> >> > overridden to vagrant-ubuntu-precise-64:50753
> >> >
> >> >
> >> > (kafka.utils.VerifiableProperties)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,030] INFO Property request.timeout.ms is
> >> >
> >> >
> >> > overridden to 30000 (kafka.utils.VerifiableProperties)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,031] INFO Fetching metadata from broker
> >> >
> >> >
> >> > id:0,host:vagrant-ubuntu-precise-64,port:50753 with correlation id 18
> >> >
> >> >
> >> > for 1 topic(s) Set(custom-topic) (kafka.client.ClientUtils$)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,032] INFO Connected to
> >> >
> >> >
> >> > vagrant-ubuntu-precise-64:50753 for producing
> >> >
> >> >
> >> > (kafka.producer.SyncProducer)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,035] INFO Disconnecting from
> >> >
> >> >
> >> > vagrant-ubuntu-precise-64:50753 (kafka.producer.SyncProducer)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,036] INFO Closing socket connection to
> >> >
> >> >
> >> > /10.0.2.15. (kafka.network.Processor)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,038] WARN [KafkaApi-0] Offset request with
> >> >
> >> >
> >> > correlation id 0 from client
> >> >
> >> >
> >> >
> >> >
> group_1-ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0
> >> >
> >> >
> >> > on partition [custom-topic,1] failed due to Leader not local for
> >> >
> >> >
> >> > partition [custom-topic,1] on broker 0 (kafka.server.KafkaApis)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,040] WARN
> >> >
> >> >
> >> >
> >> >
> [group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-leader-finder-thread],
> >> >
> >> >
> >> > Failed to add leader for partitions [custom-topic,1],[custom-topic,0];
> >> >
> >> >
> >> > will retry (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
> >> >
> >> >
> >> > kafka.common.NotLeaderForPartitionException
> >> >
> >> >
> >> > at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown
> Source)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> >> >
> >> >
> >> > at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> >> >
> >> >
> >> > at java.lang.Class.newInstance(Class.java:379)
> >> >
> >> >
> >> > at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:73)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:160)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> >> >
> >> >
> >> > at scala.collection.immutable.Map$Map2.foreach(Map.scala:130)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> >> >
> >> >
> >> > at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
> >> >
> >> >
> >> > at
> >> >
> >> >
> >> >
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
> >> >
> >> >
> >> > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,045] INFO
> >> >
> >> >
> >> >
> >> >
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
> >> >
> >> >
> >> > Shutting down (kafka.consumer.ConsumerFetcherThread)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,039] INFO
> >> >
> >> >
> >> >
> >> >
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
> >> >
> >> >
> >> > Starting (kafka.consumer.ConsumerFetcherThread)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,046] INFO
> >> >
> >> >
> >> >
> >> >
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
> >> >
> >> >
> >> > Shutdown completed (kafka.consumer.ConsumerFetcherThread)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,047] INFO Closing socket connection to
> >> >
> >> >
> >> > /10.0.2.15. (kafka.network.Processor)
> >> >
> >> >
> >> > [2014-12-29 20:02:34,048] INFO
> >> >
> >> >
> >> >
> >> >
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
> >> >
> >> >
> >> > Stopped (kafka.consumer.ConsumerFetcherThread)
> >> >
> >> >
> >> >
> >> > On Fri, Dec 26, 2014 at 4:06 PM, Gwen Shapira <gs...@cloudera.com>
> >> > wrote:
> >> >
> >> >
> >> > I can't say when's the 1.6 release, but I have other solutions :)
> >> >
> >> >
> >> >
> >> > 1. The packages that are part of CDH5.3 release will contain that jar.
> >> >
> >> >
> >> > Perhaps use this distro? Or even just get the RPM, unpackage and dig
> the
> >> > jar
> >> >
> >> >
> >> > out?
> >> >
> >> >
> >> > 2. Let us know what's the compilation error, perhaps we can help
> there?
> >> >
> >> >
> >> >
> >> > On Fri, Dec 26, 2014 at 3:30 PM, Foo Lim <fo...@vungle.com> wrote:
> >> >
> >> >
> >> >
> >> > Hi all,
> >> >
> >> >
> >> >
> >> > Happy holidays! Just wondering if there's any ETA on a 1.6 release.
> >> >
> >> >
> >> > Looking forward to the kafka sink plugin that I can't get to compile
> >> >
> >> >
> >> > independently. :-/
> >> >
> >> >
> >> >
> >> > Thanks!
> >> >
> >> >
> >> >
> >> >
> >
> >
>
Re: 1.6 release date?
Posted by Gwen Shapira <gs...@cloudera.com>.
mmmm... normally traces of Kafka or ZK are not an issue. Can you share
which directories does Kafka use for logs? (i.e. log.dir in Kafka
configuration)? And where does ZK store its data?
Thanks,
Gwen
On Mon, Jan 5, 2015 at 11:43 AM, Foo Lim <fo...@vungle.com> wrote:
> Thanks Gwen!
>
> -DskipTests worked! It built
> "dist/target/flume-kafka-sink-dist-0.5.0-bin.zip". I didn't know this was an
> option.
>
> The README said
> ## Dependency Versions
> - Apache Flume - 1.5.0
> - Apache Kafka - 0.8.1.1
>
> ## Prerequisites
> - Java 1.6 or higher
> - [Apache Maven 3](http://maven.apache.org)
> - An [Apache Flume](https://flume.apache.org) installation (See the
> dependent version above)
> - An [Apache Kafka](http://kafka.apache.org) installation (See the dependent
> version above)
>
> so I thought I had to have both running.
>
> Anyway, I stopped both kafka & zookeeper, but I'm still getting the same
> error
>
> kafka.common.NotLeaderForPartitionException
>
> Does my machine have to NOT have any traces of kafka or ZK?
>
>
> Now, on to testing if this sink works..
>
> Thanks again!
>
> Foo
>
>
> On Sun, Jan 4, 2015 at 5:19 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>>
>> I think the only thing that's failing in the build are the unit tests.
>>
>> 1. To build without unit tests: "mvn clean install -DskipTests"
>> 2. I suspect you are building Flume on a machine that has either Kafka
>> or Zookeeper installed, and we don't randomize ports properly in the
>> tests, therefore creating a mess. You can build on a dev machine not
>> running Kafka and Zookeeper. I'll create a patch making sure we don't
>> accidentally pick used ports for tests.
>>
>> Gwen
>>
>> On Mon, Dec 29, 2014 at 7:31 PM, Frank Yao <ba...@gmail.com> wrote:
>> > you can get source code from 1.6 repo and modify pom.xml to adjust code
>> > to
>> > be compitable with 1.5. I have done this once and it works well
>> > Frank Yao
>> > @Vipshop, Shanghai
>> > from iPhone
>> >
>> > 在 2014年12月30日，10:47，Foo Lim <fo...@vungle.com> 写道：
>> >
>> > Hi again Frank,
>> >
>> > You are using flume 1.6. I'm trying to get the sink running on 1.5.2
>> > for a production machine. Thanks..
>> >
>> > On Mon, Dec 29, 2014 at 3:53 PM, Frank Yao <ba...@gmail.com> wrote:
>> >
>> > hi foo
>> >
>> >
>> > it seems your stack showed exception was caused by kafka itself
>> >
>> >
>> > Failed to add leader for partitions
>> >
>> >
>> >
>> > I have used kafka sink and source of flume 1.6 for several weeks and it
>> >
>> > works well.
>> >
>> >
>> > Could you please try to use kafka console producer first to test if the
>> >
>> > partitionis okay or not?
>> >
>> > Frank Yao
>> >
>> > @Vipshop, Shanghai
>> >
>> > from iPhone
>> >
>> >
>> > 在 2014年12月30日，04:21，Foo Lim <fo...@vungle.com> 写道：
>> >
>> >
>> > BTW, I followed the directions & ran
>> >
>> >
>> > ~/flume-ng-kafka-sink$ mvn clean install
>> >
>> >
>> > On Mon, Dec 29, 2014 at 12:10 PM, Foo Lim <fo...@vungle.com> wrote:
>> >
>> >
>> > Hi Gwen,
>> >
>> >
>> >
>> > Thanks for the reply.
>> >
>> >
>> >
>> > I'll try the CDH jar file. Where do I put it in the flume directory
>> >
>> > structure?
>> >
>> >
>> >
>> > I have kafka_2.9.2-0.8.1.1 running. Here's the test error (which keeps
>> >
>> >
>> > repeating) in the project
>> >
>> >
>> > git@github.com:thilinamb/flume-ng-kafka-sink.git
>> >
>> >
>> >
>> > [2014-12-29 20:02:34,028] INFO Verifying properties
>> >
>> >
>> > (kafka.utils.VerifiableProperties)
>> >
>> >
>> > [2014-12-29 20:02:34,029] INFO Property client.id is overridden to
>> >
>> >
>> > group_1 (kafka.utils.VerifiableProperties)
>> >
>> >
>> > [2014-12-29 20:02:34,030] INFO Property metadata.broker.list is
>> >
>> >
>> > overridden to vagrant-ubuntu-precise-64:50753
>> >
>> >
>> > (kafka.utils.VerifiableProperties)
>> >
>> >
>> > [2014-12-29 20:02:34,030] INFO Property request.timeout.ms is
>> >
>> >
>> > overridden to 30000 (kafka.utils.VerifiableProperties)
>> >
>> >
>> > [2014-12-29 20:02:34,031] INFO Fetching metadata from broker
>> >
>> >
>> > id:0,host:vagrant-ubuntu-precise-64,port:50753 with correlation id 18
>> >
>> >
>> > for 1 topic(s) Set(custom-topic) (kafka.client.ClientUtils$)
>> >
>> >
>> > [2014-12-29 20:02:34,032] INFO Connected to
>> >
>> >
>> > vagrant-ubuntu-precise-64:50753 for producing
>> >
>> >
>> > (kafka.producer.SyncProducer)
>> >
>> >
>> > [2014-12-29 20:02:34,035] INFO Disconnecting from
>> >
>> >
>> > vagrant-ubuntu-precise-64:50753 (kafka.producer.SyncProducer)
>> >
>> >
>> > [2014-12-29 20:02:34,036] INFO Closing socket connection to
>> >
>> >
>> > /10.0.2.15. (kafka.network.Processor)
>> >
>> >
>> > [2014-12-29 20:02:34,038] WARN [KafkaApi-0] Offset request with
>> >
>> >
>> > correlation id 0 from client
>> >
>> >
>> >
>> > group_1-ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0
>> >
>> >
>> > on partition [custom-topic,1] failed due to Leader not local for
>> >
>> >
>> > partition [custom-topic,1] on broker 0 (kafka.server.KafkaApis)
>> >
>> >
>> > [2014-12-29 20:02:34,040] WARN
>> >
>> >
>> >
>> > [group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-leader-finder-thread],
>> >
>> >
>> > Failed to add leader for partitions [custom-topic,1],[custom-topic,0];
>> >
>> >
>> > will retry (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
>> >
>> >
>> > kafka.common.NotLeaderForPartitionException
>> >
>> >
>> > at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
>> >
>> >
>> > at
>> >
>> >
>> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> >
>> >
>> > at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> >
>> >
>> > at java.lang.Class.newInstance(Class.java:379)
>> >
>> >
>> > at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:73)
>> >
>> >
>> > at
>> >
>> >
>> > kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:160)
>> >
>> >
>> > at
>> >
>> >
>> > kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
>> >
>> >
>> > at
>> >
>> >
>> > kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
>> >
>> >
>> > at
>> >
>> >
>> > kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
>> >
>> >
>> > at
>> >
>> >
>> > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>> >
>> >
>> > at scala.collection.immutable.Map$Map2.foreach(Map.scala:130)
>> >
>> >
>> > at
>> >
>> >
>> > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>> >
>> >
>> > at
>> >
>> >
>> > kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
>> >
>> >
>> > at
>> >
>> >
>> > kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
>> >
>> >
>> > at
>> >
>> >
>> > kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
>> >
>> >
>> > at
>> >
>> >
>> > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>> >
>> >
>> > at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
>> >
>> >
>> > at
>> >
>> >
>> > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>> >
>> >
>> > at
>> >
>> >
>> > kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
>> >
>> >
>> > at
>> >
>> >
>> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
>> >
>> >
>> > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>> >
>> >
>> > [2014-12-29 20:02:34,045] INFO
>> >
>> >
>> >
>> > [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>> >
>> >
>> > Shutting down (kafka.consumer.ConsumerFetcherThread)
>> >
>> >
>> > [2014-12-29 20:02:34,039] INFO
>> >
>> >
>> >
>> > [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>> >
>> >
>> > Starting (kafka.consumer.ConsumerFetcherThread)
>> >
>> >
>> > [2014-12-29 20:02:34,046] INFO
>> >
>> >
>> >
>> > [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>> >
>> >
>> > Shutdown completed (kafka.consumer.ConsumerFetcherThread)
>> >
>> >
>> > [2014-12-29 20:02:34,047] INFO Closing socket connection to
>> >
>> >
>> > /10.0.2.15. (kafka.network.Processor)
>> >
>> >
>> > [2014-12-29 20:02:34,048] INFO
>> >
>> >
>> >
>> > [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>> >
>> >
>> > Stopped (kafka.consumer.ConsumerFetcherThread)
>> >
>> >
>> >
>> > On Fri, Dec 26, 2014 at 4:06 PM, Gwen Shapira <gs...@cloudera.com>
>> > wrote:
>> >
>> >
>> > I can't say when's the 1.6 release, but I have other solutions :)
>> >
>> >
>> >
>> > 1. The packages that are part of CDH5.3 release will contain that jar.
>> >
>> >
>> > Perhaps use this distro? Or even just get the RPM, unpackage and dig the
>> > jar
>> >
>> >
>> > out?
>> >
>> >
>> > 2. Let us know what's the compilation error, perhaps we can help there?
>> >
>> >
>> >
>> > On Fri, Dec 26, 2014 at 3:30 PM, Foo Lim <fo...@vungle.com> wrote:
>> >
>> >
>> >
>> > Hi all,
>> >
>> >
>> >
>> > Happy holidays! Just wondering if there's any ETA on a 1.6 release.
>> >
>> >
>> > Looking forward to the kafka sink plugin that I can't get to compile
>> >
>> >
>> > independently. :-/
>> >
>> >
>> >
>> > Thanks!
>> >
>> >
>> >
>> >
>
>
Re: 1.6 release date?
Posted by Foo Lim <fo...@vungle.com>.
Thanks Gwen!
-DskipTests worked! It built
"dist/target/flume-kafka-sink-dist-0.5.0-bin.zip". I didn't know this was
an option.
The README said
## Dependency Versions
- Apache Flume - 1.5.0
- Apache Kafka - 0.8.1.1
## Prerequisites
- Java 1.6 or higher
- [Apache Maven 3](http://maven.apache.org)
- An [Apache Flume](https://flume.apache.org) installation (See the
dependent version above)
- An [Apache Kafka](http://kafka.apache.org) installation (See the
dependent version above)
so I thought I had to have both running.
Anyway, I stopped both kafka & zookeeper, but I'm still getting the same
error
kafka.common.NotLeaderForPartitionException
Does my machine have to NOT have any traces of kafka or ZK?
Now, on to testing if this sink works..
Thanks again!
Foo
On Sun, Jan 4, 2015 at 5:19 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> I think the only thing that's failing in the build are the unit tests.
>
> 1. To build without unit tests: "mvn clean install -DskipTests"
> 2. I suspect you are building Flume on a machine that has either Kafka
> or Zookeeper installed, and we don't randomize ports properly in the
> tests, therefore creating a mess. You can build on a dev machine not
> running Kafka and Zookeeper. I'll create a patch making sure we don't
> accidentally pick used ports for tests.
>
> Gwen
>
> On Mon, Dec 29, 2014 at 7:31 PM, Frank Yao <ba...@gmail.com> wrote:
> > you can get source code from 1.6 repo and modify pom.xml to adjust code
> to
> > be compitable with 1.5. I have done this once and it works well
> > Frank Yao
> > @Vipshop, Shanghai
> > from iPhone
> >
> > 在 2014年12月30日，10:47，Foo Lim <fo...@vungle.com> 写道：
> >
> > Hi again Frank,
> >
> > You are using flume 1.6. I'm trying to get the sink running on 1.5.2
> > for a production machine. Thanks..
> >
> > On Mon, Dec 29, 2014 at 3:53 PM, Frank Yao <ba...@gmail.com> wrote:
> >
> > hi foo
> >
> >
> > it seems your stack showed exception was caused by kafka itself
> >
> >
> > Failed to add leader for partitions
> >
> >
> >
> > I have used kafka sink and source of flume 1.6 for several weeks and it
> >
> > works well.
> >
> >
> > Could you please try to use kafka console producer first to test if the
> >
> > partitionis okay or not?
> >
> > Frank Yao
> >
> > @Vipshop, Shanghai
> >
> > from iPhone
> >
> >
> > 在 2014年12月30日，04:21，Foo Lim <fo...@vungle.com> 写道：
> >
> >
> > BTW, I followed the directions & ran
> >
> >
> > ~/flume-ng-kafka-sink$ mvn clean install
> >
> >
> > On Mon, Dec 29, 2014 at 12:10 PM, Foo Lim <fo...@vungle.com> wrote:
> >
> >
> > Hi Gwen,
> >
> >
> >
> > Thanks for the reply.
> >
> >
> >
> > I'll try the CDH jar file. Where do I put it in the flume directory
> >
> > structure?
> >
> >
> >
> > I have kafka_2.9.2-0.8.1.1 running. Here's the test error (which keeps
> >
> >
> > repeating) in the project
> >
> >
> > git@github.com:thilinamb/flume-ng-kafka-sink.git
> >
> >
> >
> > [2014-12-29 20:02:34,028] INFO Verifying properties
> >
> >
> > (kafka.utils.VerifiableProperties)
> >
> >
> > [2014-12-29 20:02:34,029] INFO Property client.id is overridden to
> >
> >
> > group_1 (kafka.utils.VerifiableProperties)
> >
> >
> > [2014-12-29 20:02:34,030] INFO Property metadata.broker.list is
> >
> >
> > overridden to vagrant-ubuntu-precise-64:50753
> >
> >
> > (kafka.utils.VerifiableProperties)
> >
> >
> > [2014-12-29 20:02:34,030] INFO Property request.timeout.ms is
> >
> >
> > overridden to 30000 (kafka.utils.VerifiableProperties)
> >
> >
> > [2014-12-29 20:02:34,031] INFO Fetching metadata from broker
> >
> >
> > id:0,host:vagrant-ubuntu-precise-64,port:50753 with correlation id 18
> >
> >
> > for 1 topic(s) Set(custom-topic) (kafka.client.ClientUtils$)
> >
> >
> > [2014-12-29 20:02:34,032] INFO Connected to
> >
> >
> > vagrant-ubuntu-precise-64:50753 for producing
> >
> >
> > (kafka.producer.SyncProducer)
> >
> >
> > [2014-12-29 20:02:34,035] INFO Disconnecting from
> >
> >
> > vagrant-ubuntu-precise-64:50753 (kafka.producer.SyncProducer)
> >
> >
> > [2014-12-29 20:02:34,036] INFO Closing socket connection to
> >
> >
> > /10.0.2.15. (kafka.network.Processor)
> >
> >
> > [2014-12-29 20:02:34,038] WARN [KafkaApi-0] Offset request with
> >
> >
> > correlation id 0 from client
> >
> >
> >
> group_1-ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0
> >
> >
> > on partition [custom-topic,1] failed due to Leader not local for
> >
> >
> > partition [custom-topic,1] on broker 0 (kafka.server.KafkaApis)
> >
> >
> > [2014-12-29 20:02:34,040] WARN
> >
> >
> >
> [group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-leader-finder-thread],
> >
> >
> > Failed to add leader for partitions [custom-topic,1],[custom-topic,0];
> >
> >
> > will retry (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
> >
> >
> > kafka.common.NotLeaderForPartitionException
> >
> >
> > at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
> >
> >
> > at
> >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> >
> >
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> >
> >
> > at java.lang.Class.newInstance(Class.java:379)
> >
> >
> > at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:73)
> >
> >
> > at
> >
> >
> kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:160)
> >
> >
> > at
> >
> >
> kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
> >
> >
> > at
> >
> >
> kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
> >
> >
> > at
> >
> >
> kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
> >
> >
> > at
> >
> >
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> >
> >
> > at scala.collection.immutable.Map$Map2.foreach(Map.scala:130)
> >
> >
> > at
> >
> >
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> >
> >
> > at
> >
> >
> kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
> >
> >
> > at
> >
> >
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
> >
> >
> > at
> >
> >
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
> >
> >
> > at
> >
> >
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> >
> >
> > at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
> >
> >
> > at
> >
> >
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> >
> >
> > at
> >
> >
> kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
> >
> >
> > at
> >
> >
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
> >
> >
> > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> >
> >
> > [2014-12-29 20:02:34,045] INFO
> >
> >
> >
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
> >
> >
> > Shutting down (kafka.consumer.ConsumerFetcherThread)
> >
> >
> > [2014-12-29 20:02:34,039] INFO
> >
> >
> >
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
> >
> >
> > Starting (kafka.consumer.ConsumerFetcherThread)
> >
> >
> > [2014-12-29 20:02:34,046] INFO
> >
> >
> >
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
> >
> >
> > Shutdown completed (kafka.consumer.ConsumerFetcherThread)
> >
> >
> > [2014-12-29 20:02:34,047] INFO Closing socket connection to
> >
> >
> > /10.0.2.15. (kafka.network.Processor)
> >
> >
> > [2014-12-29 20:02:34,048] INFO
> >
> >
> >
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
> >
> >
> > Stopped (kafka.consumer.ConsumerFetcherThread)
> >
> >
> >
> > On Fri, Dec 26, 2014 at 4:06 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
> >
> >
> > I can't say when's the 1.6 release, but I have other solutions :)
> >
> >
> >
> > 1. The packages that are part of CDH5.3 release will contain that jar.
> >
> >
> > Perhaps use this distro? Or even just get the RPM, unpackage and dig the
> jar
> >
> >
> > out?
> >
> >
> > 2. Let us know what's the compilation error, perhaps we can help there?
> >
> >
> >
> > On Fri, Dec 26, 2014 at 3:30 PM, Foo Lim <fo...@vungle.com> wrote:
> >
> >
> >
> > Hi all,
> >
> >
> >
> > Happy holidays! Just wondering if there's any ETA on a 1.6 release.
> >
> >
> > Looking forward to the kafka sink plugin that I can't get to compile
> >
> >
> > independently. :-/
> >
> >
> >
> > Thanks!
> >
> >
> >
> >
>
Re: 1.6 release date?
Posted by Gwen Shapira <gs...@cloudera.com>.
I think the only thing that's failing in the build are the unit tests.
1. To build without unit tests: "mvn clean install -DskipTests"
2. I suspect you are building Flume on a machine that has either Kafka
or Zookeeper installed, and we don't randomize ports properly in the
tests, therefore creating a mess. You can build on a dev machine not
running Kafka and Zookeeper. I'll create a patch making sure we don't
accidentally pick used ports for tests.
Gwen
On Mon, Dec 29, 2014 at 7:31 PM, Frank Yao <ba...@gmail.com> wrote:
> you can get source code from 1.6 repo and modify pom.xml to adjust code to
> be compitable with 1.5. I have done this once and it works well
> Frank Yao
> @Vipshop, Shanghai
> from iPhone
>
> 在 2014年12月30日，10:47，Foo Lim <fo...@vungle.com> 写道：
>
> Hi again Frank,
>
> You are using flume 1.6. I'm trying to get the sink running on 1.5.2
> for a production machine. Thanks..
>
> On Mon, Dec 29, 2014 at 3:53 PM, Frank Yao <ba...@gmail.com> wrote:
>
> hi foo
>
>
> it seems your stack showed exception was caused by kafka itself
>
>
> Failed to add leader for partitions
>
>
>
> I have used kafka sink and source of flume 1.6 for several weeks and it
>
> works well.
>
>
> Could you please try to use kafka console producer first to test if the
>
> partitionis okay or not?
>
> Frank Yao
>
> @Vipshop, Shanghai
>
> from iPhone
>
>
> 在 2014年12月30日，04:21，Foo Lim <fo...@vungle.com> 写道：
>
>
> BTW, I followed the directions & ran
>
>
> ~/flume-ng-kafka-sink$ mvn clean install
>
>
> On Mon, Dec 29, 2014 at 12:10 PM, Foo Lim <fo...@vungle.com> wrote:
>
>
> Hi Gwen,
>
>
>
> Thanks for the reply.
>
>
>
> I'll try the CDH jar file. Where do I put it in the flume directory
>
> structure?
>
>
>
> I have kafka_2.9.2-0.8.1.1 running. Here's the test error (which keeps
>
>
> repeating) in the project
>
>
> git@github.com:thilinamb/flume-ng-kafka-sink.git
>
>
>
> [2014-12-29 20:02:34,028] INFO Verifying properties
>
>
> (kafka.utils.VerifiableProperties)
>
>
> [2014-12-29 20:02:34,029] INFO Property client.id is overridden to
>
>
> group_1 (kafka.utils.VerifiableProperties)
>
>
> [2014-12-29 20:02:34,030] INFO Property metadata.broker.list is
>
>
> overridden to vagrant-ubuntu-precise-64:50753
>
>
> (kafka.utils.VerifiableProperties)
>
>
> [2014-12-29 20:02:34,030] INFO Property request.timeout.ms is
>
>
> overridden to 30000 (kafka.utils.VerifiableProperties)
>
>
> [2014-12-29 20:02:34,031] INFO Fetching metadata from broker
>
>
> id:0,host:vagrant-ubuntu-precise-64,port:50753 with correlation id 18
>
>
> for 1 topic(s) Set(custom-topic) (kafka.client.ClientUtils$)
>
>
> [2014-12-29 20:02:34,032] INFO Connected to
>
>
> vagrant-ubuntu-precise-64:50753 for producing
>
>
> (kafka.producer.SyncProducer)
>
>
> [2014-12-29 20:02:34,035] INFO Disconnecting from
>
>
> vagrant-ubuntu-precise-64:50753 (kafka.producer.SyncProducer)
>
>
> [2014-12-29 20:02:34,036] INFO Closing socket connection to
>
>
> /10.0.2.15. (kafka.network.Processor)
>
>
> [2014-12-29 20:02:34,038] WARN [KafkaApi-0] Offset request with
>
>
> correlation id 0 from client
>
>
> group_1-ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0
>
>
> on partition [custom-topic,1] failed due to Leader not local for
>
>
> partition [custom-topic,1] on broker 0 (kafka.server.KafkaApis)
>
>
> [2014-12-29 20:02:34,040] WARN
>
>
> [group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-leader-finder-thread],
>
>
> Failed to add leader for partitions [custom-topic,1],[custom-topic,0];
>
>
> will retry (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
>
>
> kafka.common.NotLeaderForPartitionException
>
>
> at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
>
>
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>
>
> at java.lang.Class.newInstance(Class.java:379)
>
>
> at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:73)
>
>
> at
>
> kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:160)
>
>
> at
>
> kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
>
>
> at
>
> kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
>
>
> at
>
> kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
>
>
> at
>
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>
>
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:130)
>
>
> at
>
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>
>
> at
>
> kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
>
>
> at
>
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
>
>
> at
>
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
>
>
> at
>
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>
>
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
>
>
> at
>
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>
>
> at
>
> kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
>
>
> at
>
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
>
>
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>
>
> [2014-12-29 20:02:34,045] INFO
>
>
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>
>
> Shutting down (kafka.consumer.ConsumerFetcherThread)
>
>
> [2014-12-29 20:02:34,039] INFO
>
>
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>
>
> Starting (kafka.consumer.ConsumerFetcherThread)
>
>
> [2014-12-29 20:02:34,046] INFO
>
>
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>
>
> Shutdown completed (kafka.consumer.ConsumerFetcherThread)
>
>
> [2014-12-29 20:02:34,047] INFO Closing socket connection to
>
>
> /10.0.2.15. (kafka.network.Processor)
>
>
> [2014-12-29 20:02:34,048] INFO
>
>
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>
>
> Stopped (kafka.consumer.ConsumerFetcherThread)
>
>
>
> On Fri, Dec 26, 2014 at 4:06 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>
>
> I can't say when's the 1.6 release, but I have other solutions :)
>
>
>
> 1. The packages that are part of CDH5.3 release will contain that jar.
>
>
> Perhaps use this distro? Or even just get the RPM, unpackage and dig the jar
>
>
> out?
>
>
> 2. Let us know what's the compilation error, perhaps we can help there?
>
>
>
> On Fri, Dec 26, 2014 at 3:30 PM, Foo Lim <fo...@vungle.com> wrote:
>
>
>
> Hi all,
>
>
>
> Happy holidays! Just wondering if there's any ETA on a 1.6 release.
>
>
> Looking forward to the kafka sink plugin that I can't get to compile
>
>
> independently. :-/
>
>
>
> Thanks!
>
>
>
>
Re: 1.6 release date?
Posted by Frank Yao <ba...@gmail.com>.
you can get source code from 1.6 repo and modify pom.xml to adjust code to be compitable with 1.5. I have done this once and it works well
Frank Yao
@Vipshop, Shanghai
from iPhone
> �� 2014��12��30�գ�10:47��Foo Lim <fo...@vungle.com> д����
>
> Hi again Frank,
>
> You are using flume 1.6. I'm trying to get the sink running on 1.5.2
> for a production machine. Thanks..
>
>> On Mon, Dec 29, 2014 at 3:53 PM, Frank Yao <ba...@gmail.com> wrote:
>> hi foo
>>
>> it seems your stack showed exception was caused by kafka itself
>>
>> Failed to add leader for partitions
>>
>>
>> I have used kafka sink and source of flume 1.6 for several weeks and it
>> works well.
>>
>> Could you please try to use kafka console producer first to test if the
>> partitionis okay or not?
>> Frank Yao
>> @Vipshop, Shanghai
>> from iPhone
>>
>> �� 2014��12��30�գ�04:21��Foo Lim <fo...@vungle.com> д����
>>
>> BTW, I followed the directions & ran
>>
>> ~/flume-ng-kafka-sink$ mvn clean install
>>
>> On Mon, Dec 29, 2014 at 12:10 PM, Foo Lim <fo...@vungle.com> wrote:
>>
>> Hi Gwen,
>>
>>
>> Thanks for the reply.
>>
>>
>> I'll try the CDH jar file. Where do I put it in the flume directory
>> structure?
>>
>>
>> I have kafka_2.9.2-0.8.1.1 running. Here's the test error (which keeps
>>
>> repeating) in the project
>>
>> git@github.com:thilinamb/flume-ng-kafka-sink.git
>>
>>
>> [2014-12-29 20:02:34,028] INFO Verifying properties
>>
>> (kafka.utils.VerifiableProperties)
>>
>> [2014-12-29 20:02:34,029] INFO Property client.id is overridden to
>>
>> group_1 (kafka.utils.VerifiableProperties)
>>
>> [2014-12-29 20:02:34,030] INFO Property metadata.broker.list is
>>
>> overridden to vagrant-ubuntu-precise-64:50753
>>
>> (kafka.utils.VerifiableProperties)
>>
>> [2014-12-29 20:02:34,030] INFO Property request.timeout.ms is
>>
>> overridden to 30000 (kafka.utils.VerifiableProperties)
>>
>> [2014-12-29 20:02:34,031] INFO Fetching metadata from broker
>>
>> id:0,host:vagrant-ubuntu-precise-64,port:50753 with correlation id 18
>>
>> for 1 topic(s) Set(custom-topic) (kafka.client.ClientUtils$)
>>
>> [2014-12-29 20:02:34,032] INFO Connected to
>>
>> vagrant-ubuntu-precise-64:50753 for producing
>>
>> (kafka.producer.SyncProducer)
>>
>> [2014-12-29 20:02:34,035] INFO Disconnecting from
>>
>> vagrant-ubuntu-precise-64:50753 (kafka.producer.SyncProducer)
>>
>> [2014-12-29 20:02:34,036] INFO Closing socket connection to
>>
>> /10.0.2.15. (kafka.network.Processor)
>>
>> [2014-12-29 20:02:34,038] WARN [KafkaApi-0] Offset request with
>>
>> correlation id 0 from client
>>
>> group_1-ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0
>>
>> on partition [custom-topic,1] failed due to Leader not local for
>>
>> partition [custom-topic,1] on broker 0 (kafka.server.KafkaApis)
>>
>> [2014-12-29 20:02:34,040] WARN
>>
>> [group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-leader-finder-thread],
>>
>> Failed to add leader for partitions [custom-topic,1],[custom-topic,0];
>>
>> will retry (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
>>
>> kafka.common.NotLeaderForPartitionException
>>
>> at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
>>
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>
>> at java.lang.Class.newInstance(Class.java:379)
>>
>> at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:73)
>>
>> at
>> kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:160)
>>
>> at
>> kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
>>
>> at
>> kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
>>
>> at
>> kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
>>
>> at
>> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>>
>> at scala.collection.immutable.Map$Map2.foreach(Map.scala:130)
>>
>> at
>> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>>
>> at
>> kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
>>
>> at
>> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
>>
>> at
>> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
>>
>> at
>> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>>
>> at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
>>
>> at
>> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>>
>> at
>> kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
>>
>> at
>> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
>>
>> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>>
>> [2014-12-29 20:02:34,045] INFO
>>
>> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>>
>> Shutting down (kafka.consumer.ConsumerFetcherThread)
>>
>> [2014-12-29 20:02:34,039] INFO
>>
>> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>>
>> Starting (kafka.consumer.ConsumerFetcherThread)
>>
>> [2014-12-29 20:02:34,046] INFO
>>
>> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>>
>> Shutdown completed (kafka.consumer.ConsumerFetcherThread)
>>
>> [2014-12-29 20:02:34,047] INFO Closing socket connection to
>>
>> /10.0.2.15. (kafka.network.Processor)
>>
>> [2014-12-29 20:02:34,048] INFO
>>
>> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>>
>> Stopped (kafka.consumer.ConsumerFetcherThread)
>>
>>
>> On Fri, Dec 26, 2014 at 4:06 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>>
>> I can't say when's the 1.6 release, but I have other solutions :)
>>
>>
>> 1. The packages that are part of CDH5.3 release will contain that jar.
>>
>> Perhaps use this distro? Or even just get the RPM, unpackage and dig the jar
>>
>> out?
>>
>> 2. Let us know what's the compilation error, perhaps we can help there?
>>
>>
>> On Fri, Dec 26, 2014 at 3:30 PM, Foo Lim <fo...@vungle.com> wrote:
>>
>>
>> Hi all,
>>
>>
>> Happy holidays! Just wondering if there's any ETA on a 1.6 release.
>>
>> Looking forward to the kafka sink plugin that I can't get to compile
>>
>> independently. :-/
>>
>>
>> Thanks!
>>
>>
>>
Re: 1.6 release date?
Posted by Foo Lim <fo...@vungle.com>.
Hi again Frank,
You are using flume 1.6. I'm trying to get the sink running on 1.5.2
for a production machine. Thanks..
On Mon, Dec 29, 2014 at 3:53 PM, Frank Yao <ba...@gmail.com> wrote:
> hi foo
>
> it seems your stack showed exception was caused by kafka itself
>
> Failed to add leader for partitions
>
>
> I have used kafka sink and source of flume 1.6 for several weeks and it
> works well.
>
> Could you please try to use kafka console producer first to test if the
> partitionis okay or not?
> Frank Yao
> @Vipshop, Shanghai
> from iPhone
>
> 在 2014年12月30日，04:21，Foo Lim <fo...@vungle.com> 写道：
>
> BTW, I followed the directions & ran
>
> ~/flume-ng-kafka-sink$ mvn clean install
>
> On Mon, Dec 29, 2014 at 12:10 PM, Foo Lim <fo...@vungle.com> wrote:
>
> Hi Gwen,
>
>
> Thanks for the reply.
>
>
> I'll try the CDH jar file. Where do I put it in the flume directory
> structure?
>
>
> I have kafka_2.9.2-0.8.1.1 running. Here's the test error (which keeps
>
> repeating) in the project
>
> git@github.com:thilinamb/flume-ng-kafka-sink.git
>
>
> [2014-12-29 20:02:34,028] INFO Verifying properties
>
> (kafka.utils.VerifiableProperties)
>
> [2014-12-29 20:02:34,029] INFO Property client.id is overridden to
>
> group_1 (kafka.utils.VerifiableProperties)
>
> [2014-12-29 20:02:34,030] INFO Property metadata.broker.list is
>
> overridden to vagrant-ubuntu-precise-64:50753
>
> (kafka.utils.VerifiableProperties)
>
> [2014-12-29 20:02:34,030] INFO Property request.timeout.ms is
>
> overridden to 30000 (kafka.utils.VerifiableProperties)
>
> [2014-12-29 20:02:34,031] INFO Fetching metadata from broker
>
> id:0,host:vagrant-ubuntu-precise-64,port:50753 with correlation id 18
>
> for 1 topic(s) Set(custom-topic) (kafka.client.ClientUtils$)
>
> [2014-12-29 20:02:34,032] INFO Connected to
>
> vagrant-ubuntu-precise-64:50753 for producing
>
> (kafka.producer.SyncProducer)
>
> [2014-12-29 20:02:34,035] INFO Disconnecting from
>
> vagrant-ubuntu-precise-64:50753 (kafka.producer.SyncProducer)
>
> [2014-12-29 20:02:34,036] INFO Closing socket connection to
>
> /10.0.2.15. (kafka.network.Processor)
>
> [2014-12-29 20:02:34,038] WARN [KafkaApi-0] Offset request with
>
> correlation id 0 from client
>
> group_1-ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0
>
> on partition [custom-topic,1] failed due to Leader not local for
>
> partition [custom-topic,1] on broker 0 (kafka.server.KafkaApis)
>
> [2014-12-29 20:02:34,040] WARN
>
> [group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-leader-finder-thread],
>
> Failed to add leader for partitions [custom-topic,1],[custom-topic,0];
>
> will retry (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
>
> kafka.common.NotLeaderForPartitionException
>
> at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>
> at java.lang.Class.newInstance(Class.java:379)
>
> at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:73)
>
> at
> kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:160)
>
> at
> kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
>
> at
> kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
>
> at
> kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
>
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:130)
>
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>
> at
> kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
>
> at
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
>
> at
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
>
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
>
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>
> at
> kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
>
> at
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
>
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>
> [2014-12-29 20:02:34,045] INFO
>
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>
> Shutting down (kafka.consumer.ConsumerFetcherThread)
>
> [2014-12-29 20:02:34,039] INFO
>
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>
> Starting (kafka.consumer.ConsumerFetcherThread)
>
> [2014-12-29 20:02:34,046] INFO
>
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>
> Shutdown completed (kafka.consumer.ConsumerFetcherThread)
>
> [2014-12-29 20:02:34,047] INFO Closing socket connection to
>
> /10.0.2.15. (kafka.network.Processor)
>
> [2014-12-29 20:02:34,048] INFO
>
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>
> Stopped (kafka.consumer.ConsumerFetcherThread)
>
>
> On Fri, Dec 26, 2014 at 4:06 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>
> I can't say when's the 1.6 release, but I have other solutions :)
>
>
> 1. The packages that are part of CDH5.3 release will contain that jar.
>
> Perhaps use this distro? Or even just get the RPM, unpackage and dig the jar
>
> out?
>
> 2. Let us know what's the compilation error, perhaps we can help there?
>
>
> On Fri, Dec 26, 2014 at 3:30 PM, Foo Lim <fo...@vungle.com> wrote:
>
>
> Hi all,
>
>
> Happy holidays! Just wondering if there's any ETA on a 1.6 release.
>
> Looking forward to the kafka sink plugin that I can't get to compile
>
> independently. :-/
>
>
> Thanks!
>
>
>
Re: 1.6 release date?
Posted by Frank Yao <ba...@gmail.com>.
hi foo
it seems your stack showed exception was caused by kafka itself
> Failed to add leader for partitions
I have used kafka sink and source of flume 1.6 for several weeks and it works well.
Could you please try to use kafka console producer first to test if the partitionis okay or not?
Frank Yao
@Vipshop, Shanghai
from iPhone
> �� 2014��12��30�գ�04:21��Foo Lim <fo...@vungle.com> д����
>
> BTW, I followed the directions & ran
>
> ~/flume-ng-kafka-sink$ mvn clean install
>
>> On Mon, Dec 29, 2014 at 12:10 PM, Foo Lim <fo...@vungle.com> wrote:
>> Hi Gwen,
>>
>> Thanks for the reply.
>>
>> I'll try the CDH jar file. Where do I put it in the flume directory structure?
>>
>> I have kafka_2.9.2-0.8.1.1 running. Here's the test error (which keeps
>> repeating) in the project
>> git@github.com:thilinamb/flume-ng-kafka-sink.git
>>
>> [2014-12-29 20:02:34,028] INFO Verifying properties
>> (kafka.utils.VerifiableProperties)
>> [2014-12-29 20:02:34,029] INFO Property client.id is overridden to
>> group_1 (kafka.utils.VerifiableProperties)
>> [2014-12-29 20:02:34,030] INFO Property metadata.broker.list is
>> overridden to vagrant-ubuntu-precise-64:50753
>> (kafka.utils.VerifiableProperties)
>> [2014-12-29 20:02:34,030] INFO Property request.timeout.ms is
>> overridden to 30000 (kafka.utils.VerifiableProperties)
>> [2014-12-29 20:02:34,031] INFO Fetching metadata from broker
>> id:0,host:vagrant-ubuntu-precise-64,port:50753 with correlation id 18
>> for 1 topic(s) Set(custom-topic) (kafka.client.ClientUtils$)
>> [2014-12-29 20:02:34,032] INFO Connected to
>> vagrant-ubuntu-precise-64:50753 for producing
>> (kafka.producer.SyncProducer)
>> [2014-12-29 20:02:34,035] INFO Disconnecting from
>> vagrant-ubuntu-precise-64:50753 (kafka.producer.SyncProducer)
>> [2014-12-29 20:02:34,036] INFO Closing socket connection to
>> /10.0.2.15. (kafka.network.Processor)
>> [2014-12-29 20:02:34,038] WARN [KafkaApi-0] Offset request with
>> correlation id 0 from client
>> group_1-ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0
>> on partition [custom-topic,1] failed due to Leader not local for
>> partition [custom-topic,1] on broker 0 (kafka.server.KafkaApis)
>> [2014-12-29 20:02:34,040] WARN
>> [group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-leader-finder-thread],
>> Failed to add leader for partitions [custom-topic,1],[custom-topic,0];
>> will retry (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
>> kafka.common.NotLeaderForPartitionException
>> at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> at java.lang.Class.newInstance(Class.java:379)
>> at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:73)
>> at kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:160)
>> at kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
>> at kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
>> at kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
>> at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>> at scala.collection.immutable.Map$Map2.foreach(Map.scala:130)
>> at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>> at kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
>> at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
>> at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
>> at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>> at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
>> at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>> at kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
>> at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
>> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>> [2014-12-29 20:02:34,045] INFO
>> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>> Shutting down (kafka.consumer.ConsumerFetcherThread)
>> [2014-12-29 20:02:34,039] INFO
>> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>> Starting (kafka.consumer.ConsumerFetcherThread)
>> [2014-12-29 20:02:34,046] INFO
>> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>> Shutdown completed (kafka.consumer.ConsumerFetcherThread)
>> [2014-12-29 20:02:34,047] INFO Closing socket connection to
>> /10.0.2.15. (kafka.network.Processor)
>> [2014-12-29 20:02:34,048] INFO
>> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
>> Stopped (kafka.consumer.ConsumerFetcherThread)
>>
>>> On Fri, Dec 26, 2014 at 4:06 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>>> I can't say when's the 1.6 release, but I have other solutions :)
>>>
>>> 1. The packages that are part of CDH5.3 release will contain that jar.
>>> Perhaps use this distro? Or even just get the RPM, unpackage and dig the jar
>>> out?
>>> 2. Let us know what's the compilation error, perhaps we can help there?
>>>
>>>> On Fri, Dec 26, 2014 at 3:30 PM, Foo Lim <fo...@vungle.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> Happy holidays! Just wondering if there's any ETA on a 1.6 release.
>>>> Looking forward to the kafka sink plugin that I can't get to compile
>>>> independently. :-/
>>>>
>>>> Thanks!
>>>
>>>
Re: 1.6 release date?
Posted by Foo Lim <fo...@vungle.com>.
BTW, I followed the directions & ran
~/flume-ng-kafka-sink$ mvn clean install
On Mon, Dec 29, 2014 at 12:10 PM, Foo Lim <fo...@vungle.com> wrote:
> Hi Gwen,
>
> Thanks for the reply.
>
> I'll try the CDH jar file. Where do I put it in the flume directory structure?
>
> I have kafka_2.9.2-0.8.1.1 running. Here's the test error (which keeps
> repeating) in the project
> git@github.com:thilinamb/flume-ng-kafka-sink.git
>
> [2014-12-29 20:02:34,028] INFO Verifying properties
> (kafka.utils.VerifiableProperties)
> [2014-12-29 20:02:34,029] INFO Property client.id is overridden to
> group_1 (kafka.utils.VerifiableProperties)
> [2014-12-29 20:02:34,030] INFO Property metadata.broker.list is
> overridden to vagrant-ubuntu-precise-64:50753
> (kafka.utils.VerifiableProperties)
> [2014-12-29 20:02:34,030] INFO Property request.timeout.ms is
> overridden to 30000 (kafka.utils.VerifiableProperties)
> [2014-12-29 20:02:34,031] INFO Fetching metadata from broker
> id:0,host:vagrant-ubuntu-precise-64,port:50753 with correlation id 18
> for 1 topic(s) Set(custom-topic) (kafka.client.ClientUtils$)
> [2014-12-29 20:02:34,032] INFO Connected to
> vagrant-ubuntu-precise-64:50753 for producing
> (kafka.producer.SyncProducer)
> [2014-12-29 20:02:34,035] INFO Disconnecting from
> vagrant-ubuntu-precise-64:50753 (kafka.producer.SyncProducer)
> [2014-12-29 20:02:34,036] INFO Closing socket connection to
> /10.0.2.15. (kafka.network.Processor)
> [2014-12-29 20:02:34,038] WARN [KafkaApi-0] Offset request with
> correlation id 0 from client
> group_1-ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0
> on partition [custom-topic,1] failed due to Leader not local for
> partition [custom-topic,1] on broker 0 (kafka.server.KafkaApis)
> [2014-12-29 20:02:34,040] WARN
> [group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-leader-finder-thread],
> Failed to add leader for partitions [custom-topic,1],[custom-topic,0];
> will retry (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
> kafka.common.NotLeaderForPartitionException
> at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at java.lang.Class.newInstance(Class.java:379)
> at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:73)
> at kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:160)
> at kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
> at kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
> at kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
> at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:130)
> at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> at kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
> at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
> at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
> at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
> at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> at kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
> at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> [2014-12-29 20:02:34,045] INFO
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
> Shutting down (kafka.consumer.ConsumerFetcherThread)
> [2014-12-29 20:02:34,039] INFO
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
> Starting (kafka.consumer.ConsumerFetcherThread)
> [2014-12-29 20:02:34,046] INFO
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
> Shutdown completed (kafka.consumer.ConsumerFetcherThread)
> [2014-12-29 20:02:34,047] INFO Closing socket connection to
> /10.0.2.15. (kafka.network.Processor)
> [2014-12-29 20:02:34,048] INFO
> [ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
> Stopped (kafka.consumer.ConsumerFetcherThread)
>
> On Fri, Dec 26, 2014 at 4:06 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>> I can't say when's the 1.6 release, but I have other solutions :)
>>
>> 1. The packages that are part of CDH5.3 release will contain that jar.
>> Perhaps use this distro? Or even just get the RPM, unpackage and dig the jar
>> out?
>> 2. Let us know what's the compilation error, perhaps we can help there?
>>
>> On Fri, Dec 26, 2014 at 3:30 PM, Foo Lim <fo...@vungle.com> wrote:
>>>
>>> Hi all,
>>>
>>> Happy holidays! Just wondering if there's any ETA on a 1.6 release.
>>> Looking forward to the kafka sink plugin that I can't get to compile
>>> independently. :-/
>>>
>>> Thanks!
>>
>>
Re: 1.6 release date?
Posted by Foo Lim <fo...@vungle.com>.
Hi Gwen,
Thanks for the reply.
I'll try the CDH jar file. Where do I put it in the flume directory structure?
I have kafka_2.9.2-0.8.1.1 running. Here's the test error (which keeps
repeating) in the project
git@github.com:thilinamb/flume-ng-kafka-sink.git
[2014-12-29 20:02:34,028] INFO Verifying properties
(kafka.utils.VerifiableProperties)
[2014-12-29 20:02:34,029] INFO Property client.id is overridden to
group_1 (kafka.utils.VerifiableProperties)
[2014-12-29 20:02:34,030] INFO Property metadata.broker.list is
overridden to vagrant-ubuntu-precise-64:50753
(kafka.utils.VerifiableProperties)
[2014-12-29 20:02:34,030] INFO Property request.timeout.ms is
overridden to 30000 (kafka.utils.VerifiableProperties)
[2014-12-29 20:02:34,031] INFO Fetching metadata from broker
id:0,host:vagrant-ubuntu-precise-64,port:50753 with correlation id 18
for 1 topic(s) Set(custom-topic) (kafka.client.ClientUtils$)
[2014-12-29 20:02:34,032] INFO Connected to
vagrant-ubuntu-precise-64:50753 for producing
(kafka.producer.SyncProducer)
[2014-12-29 20:02:34,035] INFO Disconnecting from
vagrant-ubuntu-precise-64:50753 (kafka.producer.SyncProducer)
[2014-12-29 20:02:34,036] INFO Closing socket connection to
/10.0.2.15. (kafka.network.Processor)
[2014-12-29 20:02:34,038] WARN [KafkaApi-0] Offset request with
correlation id 0 from client
group_1-ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0
on partition [custom-topic,1] failed due to Leader not local for
partition [custom-topic,1] on broker 0 (kafka.server.KafkaApis)
[2014-12-29 20:02:34,040] WARN
[group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-leader-finder-thread],
Failed to add leader for partitions [custom-topic,1],[custom-topic,0];
will retry (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.NotLeaderForPartitionException
at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:379)
at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:73)
at kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:160)
at kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
at kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
at kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at scala.collection.immutable.Map$Map2.foreach(Map.scala:130)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
at kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
at kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[2014-12-29 20:02:34,045] INFO
[ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
Shutting down (kafka.consumer.ConsumerFetcherThread)
[2014-12-29 20:02:34,039] INFO
[ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
Starting (kafka.consumer.ConsumerFetcherThread)
[2014-12-29 20:02:34,046] INFO
[ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
Shutdown completed (kafka.consumer.ConsumerFetcherThread)
[2014-12-29 20:02:34,047] INFO Closing socket connection to
/10.0.2.15. (kafka.network.Processor)
[2014-12-29 20:02:34,048] INFO
[ConsumerFetcherThread-group_1_vagrant-ubuntu-precise-64-1419883349017-325c06d5-0-0],
Stopped (kafka.consumer.ConsumerFetcherThread)
On Fri, Dec 26, 2014 at 4:06 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> I can't say when's the 1.6 release, but I have other solutions :)
>
> 1. The packages that are part of CDH5.3 release will contain that jar.
> Perhaps use this distro? Or even just get the RPM, unpackage and dig the jar
> out?
> 2. Let us know what's the compilation error, perhaps we can help there?
>
> On Fri, Dec 26, 2014 at 3:30 PM, Foo Lim <fo...@vungle.com> wrote:
>>
>> Hi all,
>>
>> Happy holidays! Just wondering if there's any ETA on a 1.6 release.
>> Looking forward to the kafka sink plugin that I can't get to compile
>> independently. :-/
>>
>> Thanks!
>
>
Re: 1.6 release date?
Posted by Gwen Shapira <gs...@cloudera.com>.
I can't say when's the 1.6 release, but I have other solutions :)
1. The packages that are part of CDH5.3 release will contain that jar.
Perhaps use this distro? Or even just get the RPM, unpackage and dig the
jar out?
2. Let us know what's the compilation error, perhaps we can help there?
On Fri, Dec 26, 2014 at 3:30 PM, Foo Lim <fo...@vungle.com> wrote:
> Hi all,
>
> Happy holidays! Just wondering if there's any ETA on a 1.6 release.
> Looking forward to the kafka sink plugin that I can't get to compile
> independently. :-/
>
> Thanks!
>
Re: 1.6 release date?
Posted by Ashish <pa...@gmail.com>.
So, I shall hold my patches until the next release :)
When do we start?
On Wed, Jan 21, 2015 at 12:09 AM, Hari Shreedharan <
hshreedharan@cloudera.com> wrote:
> I am interested in kicking off the 1.6 release process soon. I will send
> out an email on another thread.
>
> Thanks,
> Hari
>
>
> On Tue, Jan 20, 2015 at 12:26 AM, Santiago Mola <sm...@stratio.com> wrote:
>
>> In the mean time, you can also use Stratio Ingestion builds:
>>
>> https://github.com/Stratio/flume-ingestion/releases
>>
>> In the releases page you can find tarball, RPM and DEB packages for the
>> 0.3.0 RC1 release, which is based on Apache Flume 1.6.
>>
>>
>> 2014-12-27 0:30 GMT+01:00 Foo Lim <fo...@vungle.com>:
>>
>>> Hi all,
>>>
>>> Happy holidays! Just wondering if there's any ETA on a 1.6 release.
>>> Looking forward to the kafka sink plugin that I can't get to compile
>>> independently. :-/
>>>
>>> Thanks!
>>>
>>
>>
>>
>> --
>>
>> Santiago M. Mola
>>
>>
>> <http://www.stratio.com/>
>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>> 28224 Pozuelo de Alarcón, Madrid
>> Tel: +34 91 828 6473 // www.stratio.com // *@stratiobd
>> <https://twitter.com/StratioBD>*
>>
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: 1.6 release date?
Posted by Hari Shreedharan <hs...@cloudera.com>.
I am interested in kicking off the 1.6 release process soon. I will send out an email on another thread.
Thanks, Hari
On Tue, Jan 20, 2015 at 12:26 AM, Santiago Mola <sm...@stratio.com> wrote:
> In the mean time, you can also use Stratio Ingestion builds:
> https://github.com/Stratio/flume-ingestion/releases
> In the releases page you can find tarball, RPM and DEB packages for the
> 0.3.0 RC1 release, which is based on Apache Flume 1.6.
> 2014-12-27 0:30 GMT+01:00 Foo Lim <fo...@vungle.com>:
>> Hi all,
>>
>> Happy holidays! Just wondering if there's any ETA on a 1.6 release.
>> Looking forward to the kafka sink plugin that I can't get to compile
>> independently. :-/
>>
>> Thanks!
>>
> --
> Santiago M. Mola
> <http://www.stratio.com/>
> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
> 28224 Pozuelo de Alarcón, Madrid
> Tel: +34 91 828 6473 // www.stratio.com // *@stratiobd
> <https://twitter.com/StratioBD>*
Re: 1.6 release date?
Posted by Santiago Mola <sm...@stratio.com>.
In the mean time, you can also use Stratio Ingestion builds:
https://github.com/Stratio/flume-ingestion/releases
In the releases page you can find tarball, RPM and DEB packages for the
0.3.0 RC1 release, which is based on Apache Flume 1.6.
2014-12-27 0:30 GMT+01:00 Foo Lim <fo...@vungle.com>:
> Hi all,
>
> Happy holidays! Just wondering if there's any ETA on a 1.6 release.
> Looking forward to the kafka sink plugin that I can't get to compile
> independently. :-/
>
> Thanks!
>
--
Santiago M. Mola
<http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 828 6473 // www.stratio.com // *@stratiobd
<https://twitter.com/StratioBD>*