You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Ewen Cheslack-Postava <ew...@confluent.io> on 2017/02/10 16:51:20 UTC
[VOTE] 0.10.2.0 RC1
Hello Kafka users, developers and client-developers,
This is RC1 for release of Apache Kafka 0.10.2.0.
This is a minor version release of Apache Kafka. It includes 19 new KIPs.
See the release notes and release plan (https://cwiki.apache.org/
confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A few
feature highlights: SASL-SCRAM support, improved client compatibility to
allow use of clients newer than the broker, session windows and global
tables in the Kafka Streams API, single message transforms in the Kafka
Connect framework.
Important note: in addition to the artifacts generated using JDK7 for Scala
2.10 and 2.11, this release also includes experimental artifacts built
using JDK8 for Scala 2.12.
Important code changes since RC0 (non-docs, non system tests):
* KAFKA-4728; KafkaConsumer#commitSync should copy its input
* KAFKA-4441; Monitoring incorrect during topic creation and deletion
* KAFKA-4734; Trim the time index on old segments
* KAFKA-4725; Stop leaking messages in produce request body when requests
are delayed
* KAFKA-4716: Fix case when controller cannot be reached
Release notes for the 0.10.2.0 release:
http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
*** Please download, test and vote by Monday, Feb 13, 5pm PT ***
Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS
* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/
* Javadoc:
http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
* Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
* Documentation:
http://kafka.apache.org/0102/documentation.html
* Protocol:
http://kafka.apache.org/0102/protocol.html
* Successful Jenkins builds for the 0.10.2 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-0.10.2-jdk7/74/
System tests: https://jenkins.confluent.io/job/system-test-kafka-0.10.2/25/
/**************************************
Thanks,
Ewen
Re: [VOTE] 0.10.2.0 RC1
Posted by Nina Hanzlikova <ni...@gmail.com>.
Hi,
I am a colleague of Ian's. We use the following processing pipeline in
stream app he mentions:
https://github.com/zalando-incubator/pipeline-backbone
The streams are built using:
object Run extends App {
// ...
private val latch = new CountDownLatch(1)
private val builder = {
val b = new KStreamBuilder()
TextPipeline.initializePipelineKStream(
b,
Serdes.serdeFrom(new TextPipelineSerializer(), new
TextPipelineDeserializer()),
Some(latch)
)
b
}
private val streams = {
val s = TextPipeline.createKStreams(builder, props)
s.setUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {
override def uncaughtException(t: Thread, e: Throwable): Unit = {
log.error(s"Unexpected Exception caught in thread [${t.getName}]:", e)
}
})
s
}
sys.addShutdownHook(shutdown())
try {
log.info("Starting...")
streams.start()
loopForever()
} catch {
case NonFatal(e) => log.error("Exception starting kafka streams", e)
} finally {
streams.close()
}
def shutdown(): Unit = {
log.warn("Shutting down the system")
streams.close()
}
}
TextPipeline.initializePipelineKStream(builder: KStreamBuilder,
tpdSerde: Serde[TextPipelineDatum], latch: Option[CountDownLatch]) = {
val coord = new KafkaStreamsBackboneCoordinator(backbone, latch)
// from the linked pipeline
val kstream = builder.stream(new StringSerde, tpdSerde, INPUT_TOPIC)
kstream.transformValues[Xor[TransformationPipelineFailure,
TextPipelineDatum]](coord)
.flatMapValues(new
KafkaStreamValidDatumValueMapper[TextPipelineDatum])
// from the linked pipeline
}
TextPipeline.createKStreams(builder: KStreamBuilder, props:
Properties): KafkaStreams =
new KafkaStreams(builder, new StreamsConfig(props))
The stream processing being done is quite slow and intensive (can take
of the order of minutes). We also have a large number of partitions on
our input topic (96), with replication factor 3. We have dropped our
max.poll.records to 10 and still see this problem.
Thanks for your help and best regards,
Nina
On 13 February 2017 at 08:09, Eno Thereska <en...@gmail.com> wrote:
> +1 (non binding)
>
> Checked streams. Verified that stream tests work and examples off confluentinc/examples/kafka-streams work.
>
> Thanks
> Eno
>
>> On 10 Feb 2017, at 16:51, Ewen Cheslack-Postava <ew...@confluent.io> wrote:
>>
>> Hello Kafka users, developers and client-developers,
>>
>> This is RC1 for release of Apache Kafka 0.10.2.0.
>>
>> This is a minor version release of Apache Kafka. It includes 19 new KIPs.
>> See the release notes and release plan (https://cwiki.apache.org/
>> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A few
>> feature highlights: SASL-SCRAM support, improved client compatibility to
>> allow use of clients newer than the broker, session windows and global
>> tables in the Kafka Streams API, single message transforms in the Kafka
>> Connect framework.
>>
>> Important note: in addition to the artifacts generated using JDK7 for Scala
>> 2.10 and 2.11, this release also includes experimental artifacts built
>> using JDK8 for Scala 2.12.
>>
>> Important code changes since RC0 (non-docs, non system tests):
>>
>> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
>> * KAFKA-4441; Monitoring incorrect during topic creation and deletion
>> * KAFKA-4734; Trim the time index on old segments
>> * KAFKA-4725; Stop leaking messages in produce request body when requests
>> are delayed
>> * KAFKA-4716: Fix case when controller cannot be reached
>>
>> Release notes for the 0.10.2.0 release:
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/
>>
>> * Javadoc:
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
>>
>> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
>>
>>
>> * Documentation:
>> http://kafka.apache.org/0102/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/0102/protocol.html
>>
>> * Successful Jenkins builds for the 0.10.2 branch:
>> Unit/integration tests: https://builds.apache.org/job/kafka-0.10.2-jdk7/74/
>> System tests: https://jenkins.confluent.io/job/system-test-kafka-0.10.2/25/
>>
>> /**************************************
>>
>> Thanks,
>> Ewen
>
--
By monitor glow
On conference paper due
A grad student naps.
-- PhD Haiku
Re: [VOTE] 0.10.2.0 RC1
Posted by Nina Hanzlikova <ni...@gmail.com>.
Hi,
I am a colleague of Ian's. We use the following processing pipeline in
stream app he mentions:
https://github.com/zalando-incubator/pipeline-backbone
The streams are built using:
object Run extends App {
// ...
private val latch = new CountDownLatch(1)
private val builder = {
val b = new KStreamBuilder()
TextPipeline.initializePipelineKStream(
b,
Serdes.serdeFrom(new TextPipelineSerializer(), new
TextPipelineDeserializer()),
Some(latch)
)
b
}
private val streams = {
val s = TextPipeline.createKStreams(builder, props)
s.setUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {
override def uncaughtException(t: Thread, e: Throwable): Unit = {
log.error(s"Unexpected Exception caught in thread [${t.getName}]:", e)
}
})
s
}
sys.addShutdownHook(shutdown())
try {
log.info("Starting...")
streams.start()
loopForever()
} catch {
case NonFatal(e) => log.error("Exception starting kafka streams", e)
} finally {
streams.close()
}
def shutdown(): Unit = {
log.warn("Shutting down the system")
streams.close()
}
}
TextPipeline.initializePipelineKStream(builder: KStreamBuilder,
tpdSerde: Serde[TextPipelineDatum], latch: Option[CountDownLatch]) = {
val coord = new KafkaStreamsBackboneCoordinator(backbone, latch)
// from the linked pipeline
val kstream = builder.stream(new StringSerde, tpdSerde, INPUT_TOPIC)
kstream.transformValues[Xor[TransformationPipelineFailure,
TextPipelineDatum]](coord)
.flatMapValues(new
KafkaStreamValidDatumValueMapper[TextPipelineDatum])
// from the linked pipeline
}
TextPipeline.createKStreams(builder: KStreamBuilder, props:
Properties): KafkaStreams =
new KafkaStreams(builder, new StreamsConfig(props))
The stream processing being done is quite slow and intensive (can take
of the order of minutes). We also have a large number of partitions on
our input topic (96), with replication factor 3. We have dropped our
max.poll.records to 10 and still see this problem.
Thanks for your help and best regards,
Nina
On 13 February 2017 at 08:09, Eno Thereska <en...@gmail.com> wrote:
> +1 (non binding)
>
> Checked streams. Verified that stream tests work and examples off confluentinc/examples/kafka-streams work.
>
> Thanks
> Eno
>
>> On 10 Feb 2017, at 16:51, Ewen Cheslack-Postava <ew...@confluent.io> wrote:
>>
>> Hello Kafka users, developers and client-developers,
>>
>> This is RC1 for release of Apache Kafka 0.10.2.0.
>>
>> This is a minor version release of Apache Kafka. It includes 19 new KIPs.
>> See the release notes and release plan (https://cwiki.apache.org/
>> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A few
>> feature highlights: SASL-SCRAM support, improved client compatibility to
>> allow use of clients newer than the broker, session windows and global
>> tables in the Kafka Streams API, single message transforms in the Kafka
>> Connect framework.
>>
>> Important note: in addition to the artifacts generated using JDK7 for Scala
>> 2.10 and 2.11, this release also includes experimental artifacts built
>> using JDK8 for Scala 2.12.
>>
>> Important code changes since RC0 (non-docs, non system tests):
>>
>> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
>> * KAFKA-4441; Monitoring incorrect during topic creation and deletion
>> * KAFKA-4734; Trim the time index on old segments
>> * KAFKA-4725; Stop leaking messages in produce request body when requests
>> are delayed
>> * KAFKA-4716: Fix case when controller cannot be reached
>>
>> Release notes for the 0.10.2.0 release:
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/
>>
>> * Javadoc:
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
>>
>> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
>>
>>
>> * Documentation:
>> http://kafka.apache.org/0102/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/0102/protocol.html
>>
>> * Successful Jenkins builds for the 0.10.2 branch:
>> Unit/integration tests: https://builds.apache.org/job/kafka-0.10.2-jdk7/74/
>> System tests: https://jenkins.confluent.io/job/system-test-kafka-0.10.2/25/
>>
>> /**************************************
>>
>> Thanks,
>> Ewen
>
--
By monitor glow
On conference paper due
A grad student naps.
-- PhD Haiku
Re: [VOTE] 0.10.2.0 RC1
Posted by Eno Thereska <en...@gmail.com>.
+1 (non binding)
Checked streams. Verified that stream tests work and examples off confluentinc/examples/kafka-streams work.
Thanks
Eno
> On 10 Feb 2017, at 16:51, Ewen Cheslack-Postava <ew...@confluent.io> wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is RC1 for release of Apache Kafka 0.10.2.0.
>
> This is a minor version release of Apache Kafka. It includes 19 new KIPs.
> See the release notes and release plan (https://cwiki.apache.org/
> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A few
> feature highlights: SASL-SCRAM support, improved client compatibility to
> allow use of clients newer than the broker, session windows and global
> tables in the Kafka Streams API, single message transforms in the Kafka
> Connect framework.
>
> Important note: in addition to the artifacts generated using JDK7 for Scala
> 2.10 and 2.11, this release also includes experimental artifacts built
> using JDK8 for Scala 2.12.
>
> Important code changes since RC0 (non-docs, non system tests):
>
> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
> * KAFKA-4441; Monitoring incorrect during topic creation and deletion
> * KAFKA-4734; Trim the time index on old segments
> * KAFKA-4725; Stop leaking messages in produce request body when requests
> are delayed
> * KAFKA-4716: Fix case when controller cannot be reached
>
> Release notes for the 0.10.2.0 release:
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
>
> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
>
>
> * Documentation:
> http://kafka.apache.org/0102/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0102/protocol.html
>
> * Successful Jenkins builds for the 0.10.2 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-0.10.2-jdk7/74/
> System tests: https://jenkins.confluent.io/job/system-test-kafka-0.10.2/25/
>
> /**************************************
>
> Thanks,
> Ewen
Re: [VOTE] 0.10.2.0 RC1
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Ian,
Thanks for the responses. Another suggestion I had is about your
session.timeout.ms config value (currently set to 10 seconds): since Nina
meaned that a processing of a single record could be as long as minutes,
that session timeout would be insufficient as no heartbeats will be sent by
the streams' consumer and the coordinator may mark it as dead. This could
probably be the reason of unexpected rebalances as well, due to the false
positives of failure detection.
Guozhang
On Mon, Feb 13, 2017 at 4:22 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Ian,
>
> an important hint: it is highly recommended to change "state.dir"
> configuration parameter from "/tmp/kafka-streams" to a different
> directory. It might be, that /tmp gets deleted and thus you loose all
> your cached data.
>
> While this is no an issues with regard to data loss (as all data is
> reliably store in Kafka brokers) it might give you a performance penalty
> as the state need to get re-created.
>
> Furthermore, if the application is running and /tmp gets deleted, your
> app instance might crash, as RocksDB relies on the data it writes to disc.
>
> Could this be related to your problem?
>
>
> -Matthias
>
> On 2/13/17 2:10 PM, Ian Duffy wrote:
> > Hi Guozhang,
> >
> > Thank you for your assistance on this.
> >
> >> About the stack trace pasted before: is it tailing some warning logs
> like
> > "Could not create task ... Will retry"
> >
> > Yes, we see the following:
> >
> > 17/02/13 21:49:55 WARN internals.StreamThread: Could not create task
> 0_93.
> > Will retry.
> > org.apache.kafka.streams.errors.LockException: task [0_93] Failed to
> lock
> > the state directory: /tmp/kafka-streams/text_pipeline_id/0_93
> > #011at
> > org.apache.kafka.streams.processor.internals.
> ProcessorStateManager.<init>(ProcessorStateManager.java:102)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.AbstractTask.<init>(
> AbstractTask.java:73)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.
> StreamTask.<init>(StreamTask.java:108)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.
> StreamThread.createStreamTask(StreamThread.java:834)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.
> createTask(StreamThread.java:1207)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.StreamThread$
> AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.
> StreamThread.addStreamTasks(StreamThread.java:937)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.StreamThread.access$500(
> StreamThread.java:69)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.StreamThread$1.
> onPartitionsAssigned(StreamThread.java:236)
> >
> > #011at
> > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.
> onJoinComplete(ConsumerCoordinator.java:255)
> >
> > #011at
> > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
> joinGroupIfNeeded(AbstractCoordinator.java:339)
> >
> > #011at
> > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
> ensureActiveGroup(AbstractCoordinator.java:303)
> >
> > #011at
> > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(
> ConsumerCoordinator.java:286)
> >
> > #011at
> > org.apache.kafka.clients.consumer.KafkaConsumer.
> pollOnce(KafkaConsumer.java:1030)
> >
> > #011at
> > org.apache.kafka.clients.consumer.KafkaConsumer.poll(
> KafkaConsumer.java:995)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(
> StreamThread.java:582)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.
> StreamThread.run(StreamThread.java:368)
> >
> >
> > It is always partition 93, additionally, we've seen processing of all the
> > other partitions without issue in between the crashes but the offset for
> 93
> > hasn't moved.
> >
> >> it is thrown as a exception all they way to the thread's
> > setUncaughtExceptionHandler, in which case your code should have a
> > `Unexpected Exception caught in thread` error log entry and the thread
> will
> > die.
> >
> > We have seen this too:
> >
> > org.apache.kafka.streams.errors.ProcessorStateException: task directory
> > [/tmp/kafka-streams/dougal_text_pipeline_id/0_94] doesn't exist and
> > couldn't be created
> > #011at
> > org.apache.kafka.streams.processor.internals.StateDirectory.
> directoryForTask(StateDirectory.java:75)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.StateDirectory.lock(
> StateDirectory.java:102)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.StateDirectory.
> cleanRemovedTasks(StateDirectory.java:205)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(
> StreamThread.java:753)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(
> StreamThread.java:664)
> >
> > #011at
> > org.apache.kafka.streams.processor.internals.
> StreamThread.run(StreamThread.java:368)
> >
> > 17/02/13 21:11:01 INFO internals.StreamThread: stream-thread
> > [StreamThread-1] Committing all tasks because the commit interval 30000ms
> > has elapsed
> > 17/02/13 21:11:01 INFO internals.StreamThread: stream-thread
> > [StreamThread-1] Committing task StreamTask 0_70
> > 17/02/13 21:11:01 INFO internals.StateDirectory: Deleting obsolete state
> > directory 0_61 for task 0_61
> >
> > Our initial thoughts with "doesn't exist and couldn't be created" made us
> > think it might be a disk space or permissions issue but we confirmed this
> > not to be the case.
> >
> >> Could you monitor the number of live stream threads during the runtime
> > (originally there should be 8 according to your configs) and see if the
> > count has decreased, meaning some threads have been dead?
> >
> > I'll try get some data on this for you tomorrow.
> >
> >> When you reported that the process "just hang on attempting to rejoin",
> > how long have you observed it hanging before killed the process?
> >
> > With the 10.1.1 client we waited up to an hour, it fails to recover at
> all.
> >
> > With the 10.2 client today it consistently self-recovered but was quickly
> > switching back into a rebalancing state.
> > We've been unable to identify a cause for rebalancing so frequently. We
> > thought it be because processing time can potentially be pretty long
> > (greater than 5min less than10min)
> >
> >> Could you try with fewer number of partitions, from 96 to 8 just for the
> > purpose of trouble shooting to see if this issue still exists?
> >
> > Will try this tomorrow, I would expect we would see a lot less
> rebalancing
> > with fewer partitions.
> >
> > Thanks again for all your help.
> > Ian.
> >
> > On 13 February 2017 at 19:55, Guozhang Wang <wa...@gmail.com> wrote:
> >
> >> @Ian, Nina
> >>
> >> Thanks for the detailed description of your apps. A couple of follow-up
> >> questions I have to help us further investigate your issue:
> >>
> >> 1. About the stack trace pasted before: is it tailing some warning logs
> >> like "Could not create task ... Will retry" (i.e. it is part of that
> >> warning log) or it is thrown as a exception all they way to the
> >> thread's setUncaughtExceptionHandler,
> >> in which case your code should have a `Unexpected Exception caught in
> >> thread` error log entry and the thread will die.
> >>
> >> 2. Could you monitor the number of live stream threads during the
> runtime
> >> (originally there should be 8 according to your configs) and see if the
> >> count has decreased, meaning some threads have been dead?
> >>
> >> 3. When you reported that the process "just hang on attempting to
> rejoin",
> >> how long have you observed it hanging before killed the process?
> >>
> >> 4. Could you try with fewer number of partitions, from 96 to 8 just for
> the
> >> purpose of trouble shooting to see if this issue still exists?
> >>
> >>
> >> Guozhang
> >>
> >>
> >> On Mon, Feb 13, 2017 at 11:38 AM, Guozhang Wang <wa...@gmail.com>
> >> wrote:
> >>
> >>> Thanks for reporting the JIRA Swen.
> >>>
> >>> Jason has a patch ready under KAFKA-4761 and I have reviewed it. You
> >> could
> >>> try it out and see if it has fixed your issue.
> >>>
> >>> After this is merged in, we will need another RC.
> >>>
> >>>
> >>> Guozhang
> >>>
> >>> On Mon, Feb 13, 2017 at 9:52 AM, Moczarski, Swen <smoczarski@ebay-
> >>> kleinanzeigen.de> wrote:
> >>>
> >>>> +0 (non-binding)
> >>>>
> >>>> Thanks for compiling a new release candidate.
> >>>>
> >>>> I get an NullPointerException when setting batch.size=0 on producer
> >>>> config. This worked before with 0.10.1.1.
> >>>> See https://issues.apache.org/jira/browse/KAFKA-4761
> >>>>
> >>>> Regards,
> >>>> Swen
> >>>>
> >>>> Am 2/10/17, 5:51 PM schrieb "Ewen Cheslack-Postava" <
> ewen@confluent.io
> >>> :
> >>>>
> >>>> Hello Kafka users, developers and client-developers,
> >>>>
> >>>> This is RC1 for release of Apache Kafka 0.10.2.0.
> >>>>
> >>>> This is a minor version release of Apache Kafka. It includes 19
> new
> >>>> KIPs.
> >>>> See the release notes and release plan (https://cwiki.apache.org/
> >>>> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more
> details. A
> >>>> few
> >>>> feature highlights: SASL-SCRAM support, improved client
> >> compatibility
> >>>> to
> >>>> allow use of clients newer than the broker, session windows and
> >> global
> >>>> tables in the Kafka Streams API, single message transforms in the
> >>>> Kafka
> >>>> Connect framework.
> >>>>
> >>>> Important note: in addition to the artifacts generated using JDK7
> >> for
> >>>> Scala
> >>>> 2.10 and 2.11, this release also includes experimental artifacts
> >> built
> >>>> using JDK8 for Scala 2.12.
> >>>>
> >>>> Important code changes since RC0 (non-docs, non system tests):
> >>>>
> >>>> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
> >>>> * KAFKA-4441; Monitoring incorrect during topic creation and
> >> deletion
> >>>> * KAFKA-4734; Trim the time index on old segments
> >>>> * KAFKA-4725; Stop leaking messages in produce request body when
> >>>> requests
> >>>> are delayed
> >>>> * KAFKA-4716: Fix case when controller cannot be reached
> >>>>
> >>>> Release notes for the 0.10.2.0 release:
> >>>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
> >> RELEASE_NOTES.html
> >>>>
> >>>> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
> >>>>
> >>>> Kafka's KEYS file containing PGP keys we use to sign the release:
> >>>> http://kafka.apache.org/KEYS
> >>>>
> >>>> * Release artifacts to be voted upon (source and binary):
> >>>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
> >>>>
> >>>> * Maven artifacts to be voted upon:
> >>>> https://repository.apache.org/content/groups/staging/
> >>>>
> >>>> * Javadoc:
> >>>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
> >>>>
> >>>> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
> >>>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> >>>> e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
> >>>>
> >>>>
> >>>> * Documentation:
> >>>> http://kafka.apache.org/0102/documentation.html
> >>>>
> >>>> * Protocol:
> >>>> http://kafka.apache.org/0102/protocol.html
> >>>>
> >>>> * Successful Jenkins builds for the 0.10.2 branch:
> >>>> Unit/integration tests: https://builds.apache.org/job/
> >>>> kafka-0.10.2-jdk7/74/
> >>>> System tests: https://jenkins.confluent.io/j
> >>>> ob/system-test-kafka-0.10.2/25/
> >>>>
> >>>> /**************************************
> >>>>
> >>>> Thanks,
> >>>> Ewen
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>
>
--
-- Guozhang
Re: [VOTE] 0.10.2.0 RC1
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Ian,
an important hint: it is highly recommended to change "state.dir"
configuration parameter from "/tmp/kafka-streams" to a different
directory. It might be, that /tmp gets deleted and thus you loose all
your cached data.
While this is no an issues with regard to data loss (as all data is
reliably store in Kafka brokers) it might give you a performance penalty
as the state need to get re-created.
Furthermore, if the application is running and /tmp gets deleted, your
app instance might crash, as RocksDB relies on the data it writes to disc.
Could this be related to your problem?
-Matthias
On 2/13/17 2:10 PM, Ian Duffy wrote:
> Hi Guozhang,
>
> Thank you for your assistance on this.
>
>> About the stack trace pasted before: is it tailing some warning logs like
> "Could not create task ... Will retry"
>
> Yes, we see the following:
>
> 17/02/13 21:49:55 WARN internals.StreamThread: Could not create task 0_93.
> Will retry.
> org.apache.kafka.streams.errors.LockException: task [0_93] Failed to lock
> the state directory: /tmp/kafka-streams/text_pipeline_id/0_93
> #011at
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.<init>(ProcessorStateManager.java:102)
>
> #011at
> org.apache.kafka.streams.processor.internals.AbstractTask.<init>(AbstractTask.java:73)
>
> #011at
> org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:108)
>
> #011at
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834)
>
> #011at
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207)
>
> #011at
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
>
> #011at
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937)
>
> #011at
> org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
>
> #011at
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236)
>
> #011at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255)
>
> #011at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
>
> #011at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
>
> #011at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
>
> #011at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030)
>
> #011at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
>
> #011at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582)
>
> #011at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
>
>
> It is always partition 93, additionally, we've seen processing of all the
> other partitions without issue in between the crashes but the offset for 93
> hasn't moved.
>
>> it is thrown as a exception all they way to the thread's
> setUncaughtExceptionHandler, in which case your code should have a
> `Unexpected Exception caught in thread` error log entry and the thread will
> die.
>
> We have seen this too:
>
> org.apache.kafka.streams.errors.ProcessorStateException: task directory
> [/tmp/kafka-streams/dougal_text_pipeline_id/0_94] doesn't exist and
> couldn't be created
> #011at
> org.apache.kafka.streams.processor.internals.StateDirectory.directoryForTask(StateDirectory.java:75)
>
> #011at
> org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:102)
>
> #011at
> org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:205)
>
> #011at
> org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:753)
>
> #011at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:664)
>
> #011at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
>
> 17/02/13 21:11:01 INFO internals.StreamThread: stream-thread
> [StreamThread-1] Committing all tasks because the commit interval 30000ms
> has elapsed
> 17/02/13 21:11:01 INFO internals.StreamThread: stream-thread
> [StreamThread-1] Committing task StreamTask 0_70
> 17/02/13 21:11:01 INFO internals.StateDirectory: Deleting obsolete state
> directory 0_61 for task 0_61
>
> Our initial thoughts with "doesn't exist and couldn't be created" made us
> think it might be a disk space or permissions issue but we confirmed this
> not to be the case.
>
>> Could you monitor the number of live stream threads during the runtime
> (originally there should be 8 according to your configs) and see if the
> count has decreased, meaning some threads have been dead?
>
> I'll try get some data on this for you tomorrow.
>
>> When you reported that the process "just hang on attempting to rejoin",
> how long have you observed it hanging before killed the process?
>
> With the 10.1.1 client we waited up to an hour, it fails to recover at all.
>
> With the 10.2 client today it consistently self-recovered but was quickly
> switching back into a rebalancing state.
> We've been unable to identify a cause for rebalancing so frequently. We
> thought it be because processing time can potentially be pretty long
> (greater than 5min less than10min)
>
>> Could you try with fewer number of partitions, from 96 to 8 just for the
> purpose of trouble shooting to see if this issue still exists?
>
> Will try this tomorrow, I would expect we would see a lot less rebalancing
> with fewer partitions.
>
> Thanks again for all your help.
> Ian.
>
> On 13 February 2017 at 19:55, Guozhang Wang <wa...@gmail.com> wrote:
>
>> @Ian, Nina
>>
>> Thanks for the detailed description of your apps. A couple of follow-up
>> questions I have to help us further investigate your issue:
>>
>> 1. About the stack trace pasted before: is it tailing some warning logs
>> like "Could not create task ... Will retry" (i.e. it is part of that
>> warning log) or it is thrown as a exception all they way to the
>> thread's setUncaughtExceptionHandler,
>> in which case your code should have a `Unexpected Exception caught in
>> thread` error log entry and the thread will die.
>>
>> 2. Could you monitor the number of live stream threads during the runtime
>> (originally there should be 8 according to your configs) and see if the
>> count has decreased, meaning some threads have been dead?
>>
>> 3. When you reported that the process "just hang on attempting to rejoin",
>> how long have you observed it hanging before killed the process?
>>
>> 4. Could you try with fewer number of partitions, from 96 to 8 just for the
>> purpose of trouble shooting to see if this issue still exists?
>>
>>
>> Guozhang
>>
>>
>> On Mon, Feb 13, 2017 at 11:38 AM, Guozhang Wang <wa...@gmail.com>
>> wrote:
>>
>>> Thanks for reporting the JIRA Swen.
>>>
>>> Jason has a patch ready under KAFKA-4761 and I have reviewed it. You
>> could
>>> try it out and see if it has fixed your issue.
>>>
>>> After this is merged in, we will need another RC.
>>>
>>>
>>> Guozhang
>>>
>>> On Mon, Feb 13, 2017 at 9:52 AM, Moczarski, Swen <smoczarski@ebay-
>>> kleinanzeigen.de> wrote:
>>>
>>>> +0 (non-binding)
>>>>
>>>> Thanks for compiling a new release candidate.
>>>>
>>>> I get an NullPointerException when setting batch.size=0 on producer
>>>> config. This worked before with 0.10.1.1.
>>>> See https://issues.apache.org/jira/browse/KAFKA-4761
>>>>
>>>> Regards,
>>>> Swen
>>>>
>>>> Am 2/10/17, 5:51 PM schrieb "Ewen Cheslack-Postava" <ewen@confluent.io
>>> :
>>>>
>>>> Hello Kafka users, developers and client-developers,
>>>>
>>>> This is RC1 for release of Apache Kafka 0.10.2.0.
>>>>
>>>> This is a minor version release of Apache Kafka. It includes 19 new
>>>> KIPs.
>>>> See the release notes and release plan (https://cwiki.apache.org/
>>>> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A
>>>> few
>>>> feature highlights: SASL-SCRAM support, improved client
>> compatibility
>>>> to
>>>> allow use of clients newer than the broker, session windows and
>> global
>>>> tables in the Kafka Streams API, single message transforms in the
>>>> Kafka
>>>> Connect framework.
>>>>
>>>> Important note: in addition to the artifacts generated using JDK7
>> for
>>>> Scala
>>>> 2.10 and 2.11, this release also includes experimental artifacts
>> built
>>>> using JDK8 for Scala 2.12.
>>>>
>>>> Important code changes since RC0 (non-docs, non system tests):
>>>>
>>>> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
>>>> * KAFKA-4441; Monitoring incorrect during topic creation and
>> deletion
>>>> * KAFKA-4734; Trim the time index on old segments
>>>> * KAFKA-4725; Stop leaking messages in produce request body when
>>>> requests
>>>> are delayed
>>>> * KAFKA-4716: Fix case when controller cannot be reached
>>>>
>>>> Release notes for the 0.10.2.0 release:
>>>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
>> RELEASE_NOTES.html
>>>>
>>>> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
>>>>
>>>> Kafka's KEYS file containing PGP keys we use to sign the release:
>>>> http://kafka.apache.org/KEYS
>>>>
>>>> * Release artifacts to be voted upon (source and binary):
>>>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
>>>>
>>>> * Maven artifacts to be voted upon:
>>>> https://repository.apache.org/content/groups/staging/
>>>>
>>>> * Javadoc:
>>>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
>>>>
>>>> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
>>>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
>>>> e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
>>>>
>>>>
>>>> * Documentation:
>>>> http://kafka.apache.org/0102/documentation.html
>>>>
>>>> * Protocol:
>>>> http://kafka.apache.org/0102/protocol.html
>>>>
>>>> * Successful Jenkins builds for the 0.10.2 branch:
>>>> Unit/integration tests: https://builds.apache.org/job/
>>>> kafka-0.10.2-jdk7/74/
>>>> System tests: https://jenkins.confluent.io/j
>>>> ob/system-test-kafka-0.10.2/25/
>>>>
>>>> /**************************************
>>>>
>>>> Thanks,
>>>> Ewen
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> -- Guozhang
>>>
>>
>>
>>
>> --
>> -- Guozhang
>>
>
Re: [VOTE] 0.10.2.0 RC1
Posted by Ian Duffy <ia...@ianduffy.ie>.
Hi Guozhang,
Thank you for your assistance on this.
> About the stack trace pasted before: is it tailing some warning logs like
"Could not create task ... Will retry"
Yes, we see the following:
17/02/13 21:49:55 WARN internals.StreamThread: Could not create task 0_93.
Will retry.
org.apache.kafka.streams.errors.LockException: task [0_93] Failed to lock
the state directory: /tmp/kafka-streams/text_pipeline_id/0_93
#011at
org.apache.kafka.streams.processor.internals.ProcessorStateManager.<init>(ProcessorStateManager.java:102)
#011at
org.apache.kafka.streams.processor.internals.AbstractTask.<init>(AbstractTask.java:73)
#011at
org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:108)
#011at
org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834)
#011at
org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207)
#011at
org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
#011at
org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937)
#011at
org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
#011at
org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236)
#011at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255)
#011at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
#011at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
#011at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
#011at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030)
#011at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
#011at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582)
#011at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
It is always partition 93, additionally, we've seen processing of all the
other partitions without issue in between the crashes but the offset for 93
hasn't moved.
> it is thrown as a exception all they way to the thread's
setUncaughtExceptionHandler, in which case your code should have a
`Unexpected Exception caught in thread` error log entry and the thread will
die.
We have seen this too:
org.apache.kafka.streams.errors.ProcessorStateException: task directory
[/tmp/kafka-streams/dougal_text_pipeline_id/0_94] doesn't exist and
couldn't be created
#011at
org.apache.kafka.streams.processor.internals.StateDirectory.directoryForTask(StateDirectory.java:75)
#011at
org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:102)
#011at
org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:205)
#011at
org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:753)
#011at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:664)
#011at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
17/02/13 21:11:01 INFO internals.StreamThread: stream-thread
[StreamThread-1] Committing all tasks because the commit interval 30000ms
has elapsed
17/02/13 21:11:01 INFO internals.StreamThread: stream-thread
[StreamThread-1] Committing task StreamTask 0_70
17/02/13 21:11:01 INFO internals.StateDirectory: Deleting obsolete state
directory 0_61 for task 0_61
Our initial thoughts with "doesn't exist and couldn't be created" made us
think it might be a disk space or permissions issue but we confirmed this
not to be the case.
> Could you monitor the number of live stream threads during the runtime
(originally there should be 8 according to your configs) and see if the
count has decreased, meaning some threads have been dead?
I'll try get some data on this for you tomorrow.
> When you reported that the process "just hang on attempting to rejoin",
how long have you observed it hanging before killed the process?
With the 10.1.1 client we waited up to an hour, it fails to recover at all.
With the 10.2 client today it consistently self-recovered but was quickly
switching back into a rebalancing state.
We've been unable to identify a cause for rebalancing so frequently. We
thought it be because processing time can potentially be pretty long
(greater than 5min less than10min)
> Could you try with fewer number of partitions, from 96 to 8 just for the
purpose of trouble shooting to see if this issue still exists?
Will try this tomorrow, I would expect we would see a lot less rebalancing
with fewer partitions.
Thanks again for all your help.
Ian.
On 13 February 2017 at 19:55, Guozhang Wang <wa...@gmail.com> wrote:
> @Ian, Nina
>
> Thanks for the detailed description of your apps. A couple of follow-up
> questions I have to help us further investigate your issue:
>
> 1. About the stack trace pasted before: is it tailing some warning logs
> like "Could not create task ... Will retry" (i.e. it is part of that
> warning log) or it is thrown as a exception all they way to the
> thread's setUncaughtExceptionHandler,
> in which case your code should have a `Unexpected Exception caught in
> thread` error log entry and the thread will die.
>
> 2. Could you monitor the number of live stream threads during the runtime
> (originally there should be 8 according to your configs) and see if the
> count has decreased, meaning some threads have been dead?
>
> 3. When you reported that the process "just hang on attempting to rejoin",
> how long have you observed it hanging before killed the process?
>
> 4. Could you try with fewer number of partitions, from 96 to 8 just for the
> purpose of trouble shooting to see if this issue still exists?
>
>
> Guozhang
>
>
> On Mon, Feb 13, 2017 at 11:38 AM, Guozhang Wang <wa...@gmail.com>
> wrote:
>
> > Thanks for reporting the JIRA Swen.
> >
> > Jason has a patch ready under KAFKA-4761 and I have reviewed it. You
> could
> > try it out and see if it has fixed your issue.
> >
> > After this is merged in, we will need another RC.
> >
> >
> > Guozhang
> >
> > On Mon, Feb 13, 2017 at 9:52 AM, Moczarski, Swen <smoczarski@ebay-
> > kleinanzeigen.de> wrote:
> >
> >> +0 (non-binding)
> >>
> >> Thanks for compiling a new release candidate.
> >>
> >> I get an NullPointerException when setting batch.size=0 on producer
> >> config. This worked before with 0.10.1.1.
> >> See https://issues.apache.org/jira/browse/KAFKA-4761
> >>
> >> Regards,
> >> Swen
> >>
> >> Am 2/10/17, 5:51 PM schrieb "Ewen Cheslack-Postava" <ewen@confluent.io
> >:
> >>
> >> Hello Kafka users, developers and client-developers,
> >>
> >> This is RC1 for release of Apache Kafka 0.10.2.0.
> >>
> >> This is a minor version release of Apache Kafka. It includes 19 new
> >> KIPs.
> >> See the release notes and release plan (https://cwiki.apache.org/
> >> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A
> >> few
> >> feature highlights: SASL-SCRAM support, improved client
> compatibility
> >> to
> >> allow use of clients newer than the broker, session windows and
> global
> >> tables in the Kafka Streams API, single message transforms in the
> >> Kafka
> >> Connect framework.
> >>
> >> Important note: in addition to the artifacts generated using JDK7
> for
> >> Scala
> >> 2.10 and 2.11, this release also includes experimental artifacts
> built
> >> using JDK8 for Scala 2.12.
> >>
> >> Important code changes since RC0 (non-docs, non system tests):
> >>
> >> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
> >> * KAFKA-4441; Monitoring incorrect during topic creation and
> deletion
> >> * KAFKA-4734; Trim the time index on old segments
> >> * KAFKA-4725; Stop leaking messages in produce request body when
> >> requests
> >> are delayed
> >> * KAFKA-4716: Fix case when controller cannot be reached
> >>
> >> Release notes for the 0.10.2.0 release:
> >> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
> RELEASE_NOTES.html
> >>
> >> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
> >>
> >> Kafka's KEYS file containing PGP keys we use to sign the release:
> >> http://kafka.apache.org/KEYS
> >>
> >> * Release artifacts to be voted upon (source and binary):
> >> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
> >>
> >> * Maven artifacts to be voted upon:
> >> https://repository.apache.org/content/groups/staging/
> >>
> >> * Javadoc:
> >> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
> >>
> >> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
> >> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> >> e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
> >>
> >>
> >> * Documentation:
> >> http://kafka.apache.org/0102/documentation.html
> >>
> >> * Protocol:
> >> http://kafka.apache.org/0102/protocol.html
> >>
> >> * Successful Jenkins builds for the 0.10.2 branch:
> >> Unit/integration tests: https://builds.apache.org/job/
> >> kafka-0.10.2-jdk7/74/
> >> System tests: https://jenkins.confluent.io/j
> >> ob/system-test-kafka-0.10.2/25/
> >>
> >> /**************************************
> >>
> >> Thanks,
> >> Ewen
> >>
> >>
> >>
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> -- Guozhang
>
Re: [VOTE] 0.10.2.0 RC1
Posted by Guozhang Wang <wa...@gmail.com>.
@Ian, Nina
Thanks for the detailed description of your apps. A couple of follow-up
questions I have to help us further investigate your issue:
1. About the stack trace pasted before: is it tailing some warning logs
like "Could not create task ... Will retry" (i.e. it is part of that
warning log) or it is thrown as a exception all they way to the
thread's setUncaughtExceptionHandler,
in which case your code should have a `Unexpected Exception caught in
thread` error log entry and the thread will die.
2. Could you monitor the number of live stream threads during the runtime
(originally there should be 8 according to your configs) and see if the
count has decreased, meaning some threads have been dead?
3. When you reported that the process "just hang on attempting to rejoin",
how long have you observed it hanging before killed the process?
4. Could you try with fewer number of partitions, from 96 to 8 just for the
purpose of trouble shooting to see if this issue still exists?
Guozhang
On Mon, Feb 13, 2017 at 11:38 AM, Guozhang Wang <wa...@gmail.com> wrote:
> Thanks for reporting the JIRA Swen.
>
> Jason has a patch ready under KAFKA-4761 and I have reviewed it. You could
> try it out and see if it has fixed your issue.
>
> After this is merged in, we will need another RC.
>
>
> Guozhang
>
> On Mon, Feb 13, 2017 at 9:52 AM, Moczarski, Swen <smoczarski@ebay-
> kleinanzeigen.de> wrote:
>
>> +0 (non-binding)
>>
>> Thanks for compiling a new release candidate.
>>
>> I get an NullPointerException when setting batch.size=0 on producer
>> config. This worked before with 0.10.1.1.
>> See https://issues.apache.org/jira/browse/KAFKA-4761
>>
>> Regards,
>> Swen
>>
>> Am 2/10/17, 5:51 PM schrieb "Ewen Cheslack-Postava" <ew...@confluent.io>:
>>
>> Hello Kafka users, developers and client-developers,
>>
>> This is RC1 for release of Apache Kafka 0.10.2.0.
>>
>> This is a minor version release of Apache Kafka. It includes 19 new
>> KIPs.
>> See the release notes and release plan (https://cwiki.apache.org/
>> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A
>> few
>> feature highlights: SASL-SCRAM support, improved client compatibility
>> to
>> allow use of clients newer than the broker, session windows and global
>> tables in the Kafka Streams API, single message transforms in the
>> Kafka
>> Connect framework.
>>
>> Important note: in addition to the artifacts generated using JDK7 for
>> Scala
>> 2.10 and 2.11, this release also includes experimental artifacts built
>> using JDK8 for Scala 2.12.
>>
>> Important code changes since RC0 (non-docs, non system tests):
>>
>> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
>> * KAFKA-4441; Monitoring incorrect during topic creation and deletion
>> * KAFKA-4734; Trim the time index on old segments
>> * KAFKA-4725; Stop leaking messages in produce request body when
>> requests
>> are delayed
>> * KAFKA-4716: Fix case when controller cannot be reached
>>
>> Release notes for the 0.10.2.0 release:
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/
>>
>> * Javadoc:
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
>>
>> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
>> e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
>>
>>
>> * Documentation:
>> http://kafka.apache.org/0102/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/0102/protocol.html
>>
>> * Successful Jenkins builds for the 0.10.2 branch:
>> Unit/integration tests: https://builds.apache.org/job/
>> kafka-0.10.2-jdk7/74/
>> System tests: https://jenkins.confluent.io/j
>> ob/system-test-kafka-0.10.2/25/
>>
>> /**************************************
>>
>> Thanks,
>> Ewen
>>
>>
>>
>
>
> --
> -- Guozhang
>
--
-- Guozhang
Re: [VOTE] 0.10.2.0 RC1
Posted by Ian Duffy <ia...@ianduffy.ie>.
Hi Matthias and Guozhang,
+1 (non-binding)
We eventually solved this on our end. It boiled down to dodgy data and
unexpected hidden errors in our processing steps. This eventually resulted
in a session time out which triggered the rebalances.
Thanks again for all your help.
Ian.
On 14 February 2017 at 08:06, Moczarski, Swen <
smoczarski@ebay-kleinanzeigen.de> wrote:
> Thanks a lot for the fast fix!
>
> I tested my code with the fix and it works fine without exception.
>
> Regards,
> Swen
>
> Am 2/13/17, 8:38 PM schrieb "Guozhang Wang" <wa...@gmail.com>:
>
> Thanks for reporting the JIRA Swen.
>
> Jason has a patch ready under KAFKA-4761 and I have reviewed it. You
> could
> try it out and see if it has fixed your issue.
>
> After this is merged in, we will need another RC.
>
>
> Guozhang
>
> On Mon, Feb 13, 2017 at 9:52 AM, Moczarski, Swen <
> smoczarski@ebay-kleinanzeigen.de> wrote:
>
> > +0 (non-binding)
> >
> > Thanks for compiling a new release candidate.
> >
> > I get an NullPointerException when setting batch.size=0 on producer
> > config. This worked before with 0.10.1.1.
> > See https://issues.apache.org/jira/browse/KAFKA-4761
> >
> > Regards,
> > Swen
> >
> > Am 2/10/17, 5:51 PM schrieb "Ewen Cheslack-Postava" <
> ewen@confluent.io>:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is RC1 for release of Apache Kafka 0.10.2.0.
> >
> > This is a minor version release of Apache Kafka. It includes 19
> new
> > KIPs.
> > See the release notes and release plan (
> https://cwiki.apache.org/
> > confluence/display/KAFKA/Release+Plan+0.10.2.0) for more
> details. A
> > few
> > feature highlights: SASL-SCRAM support, improved client
> compatibility
> > to
> > allow use of clients newer than the broker, session windows and
> global
> > tables in the Kafka Streams API, single message transforms in
> the Kafka
> > Connect framework.
> >
> > Important note: in addition to the artifacts generated using
> JDK7 for
> > Scala
> > 2.10 and 2.11, this release also includes experimental artifacts
> built
> > using JDK8 for Scala 2.12.
> >
> > Important code changes since RC0 (non-docs, non system tests):
> >
> > * KAFKA-4728; KafkaConsumer#commitSync should copy its input
> > * KAFKA-4441; Monitoring incorrect during topic creation and
> deletion
> > * KAFKA-4734; Trim the time index on old segments
> > * KAFKA-4725; Stop leaking messages in produce request body when
> > requests
> > are delayed
> > * KAFKA-4716: Fix case when controller cannot be reached
> >
> > Release notes for the 0.10.2.0 release:
> > http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
> RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/
> >
> > * Javadoc:
> > http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> > e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
> >
> >
> > * Documentation:
> > http://kafka.apache.org/0102/documentation.html
> >
> > * Protocol:
> > http://kafka.apache.org/0102/protocol.html
> >
> > * Successful Jenkins builds for the 0.10.2 branch:
> > Unit/integration tests: https://builds.apache.org/job/
> > kafka-0.10.2-jdk7/74/
> > System tests: https://jenkins.confluent.io/
> > job/system-test-kafka-0.10.2/25/
> >
> > /**************************************
> >
> > Thanks,
> > Ewen
> >
> >
> >
>
>
> --
> -- Guozhang
>
>
>
Re: [VOTE] 0.10.2.0 RC1
Posted by "Moczarski, Swen" <sm...@ebay-kleinanzeigen.de>.
Thanks a lot for the fast fix!
I tested my code with the fix and it works fine without exception.
Regards,
Swen
Am 2/13/17, 8:38 PM schrieb "Guozhang Wang" <wa...@gmail.com>:
Thanks for reporting the JIRA Swen.
Jason has a patch ready under KAFKA-4761 and I have reviewed it. You could
try it out and see if it has fixed your issue.
After this is merged in, we will need another RC.
Guozhang
On Mon, Feb 13, 2017 at 9:52 AM, Moczarski, Swen <
smoczarski@ebay-kleinanzeigen.de> wrote:
> +0 (non-binding)
>
> Thanks for compiling a new release candidate.
>
> I get an NullPointerException when setting batch.size=0 on producer
> config. This worked before with 0.10.1.1.
> See https://issues.apache.org/jira/browse/KAFKA-4761
>
> Regards,
> Swen
>
> Am 2/10/17, 5:51 PM schrieb "Ewen Cheslack-Postava" <ew...@confluent.io>:
>
> Hello Kafka users, developers and client-developers,
>
> This is RC1 for release of Apache Kafka 0.10.2.0.
>
> This is a minor version release of Apache Kafka. It includes 19 new
> KIPs.
> See the release notes and release plan (https://cwiki.apache.org/
> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A
> few
> feature highlights: SASL-SCRAM support, improved client compatibility
> to
> allow use of clients newer than the broker, session windows and global
> tables in the Kafka Streams API, single message transforms in the Kafka
> Connect framework.
>
> Important note: in addition to the artifacts generated using JDK7 for
> Scala
> 2.10 and 2.11, this release also includes experimental artifacts built
> using JDK8 for Scala 2.12.
>
> Important code changes since RC0 (non-docs, non system tests):
>
> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
> * KAFKA-4441; Monitoring incorrect during topic creation and deletion
> * KAFKA-4734; Trim the time index on old segments
> * KAFKA-4725; Stop leaking messages in produce request body when
> requests
> are delayed
> * KAFKA-4716: Fix case when controller cannot be reached
>
> Release notes for the 0.10.2.0 release:
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
>
> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
>
>
> * Documentation:
> http://kafka.apache.org/0102/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0102/protocol.html
>
> * Successful Jenkins builds for the 0.10.2 branch:
> Unit/integration tests: https://builds.apache.org/job/
> kafka-0.10.2-jdk7/74/
> System tests: https://jenkins.confluent.io/
> job/system-test-kafka-0.10.2/25/
>
> /**************************************
>
> Thanks,
> Ewen
>
>
>
--
-- Guozhang
Re: [VOTE] 0.10.2.0 RC1
Posted by Guozhang Wang <wa...@gmail.com>.
Thanks for reporting the JIRA Swen.
Jason has a patch ready under KAFKA-4761 and I have reviewed it. You could
try it out and see if it has fixed your issue.
After this is merged in, we will need another RC.
Guozhang
On Mon, Feb 13, 2017 at 9:52 AM, Moczarski, Swen <
smoczarski@ebay-kleinanzeigen.de> wrote:
> +0 (non-binding)
>
> Thanks for compiling a new release candidate.
>
> I get an NullPointerException when setting batch.size=0 on producer
> config. This worked before with 0.10.1.1.
> See https://issues.apache.org/jira/browse/KAFKA-4761
>
> Regards,
> Swen
>
> Am 2/10/17, 5:51 PM schrieb "Ewen Cheslack-Postava" <ew...@confluent.io>:
>
> Hello Kafka users, developers and client-developers,
>
> This is RC1 for release of Apache Kafka 0.10.2.0.
>
> This is a minor version release of Apache Kafka. It includes 19 new
> KIPs.
> See the release notes and release plan (https://cwiki.apache.org/
> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A
> few
> feature highlights: SASL-SCRAM support, improved client compatibility
> to
> allow use of clients newer than the broker, session windows and global
> tables in the Kafka Streams API, single message transforms in the Kafka
> Connect framework.
>
> Important note: in addition to the artifacts generated using JDK7 for
> Scala
> 2.10 and 2.11, this release also includes experimental artifacts built
> using JDK8 for Scala 2.12.
>
> Important code changes since RC0 (non-docs, non system tests):
>
> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
> * KAFKA-4441; Monitoring incorrect during topic creation and deletion
> * KAFKA-4734; Trim the time index on old segments
> * KAFKA-4725; Stop leaking messages in produce request body when
> requests
> are delayed
> * KAFKA-4716: Fix case when controller cannot be reached
>
> Release notes for the 0.10.2.0 release:
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
>
> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
>
>
> * Documentation:
> http://kafka.apache.org/0102/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0102/protocol.html
>
> * Successful Jenkins builds for the 0.10.2 branch:
> Unit/integration tests: https://builds.apache.org/job/
> kafka-0.10.2-jdk7/74/
> System tests: https://jenkins.confluent.io/
> job/system-test-kafka-0.10.2/25/
>
> /**************************************
>
> Thanks,
> Ewen
>
>
>
--
-- Guozhang
Re: [VOTE] 0.10.2.0 RC1
Posted by "Moczarski, Swen" <sm...@ebay-kleinanzeigen.de>.
+0 (non-binding)
Thanks for compiling a new release candidate.
I get an NullPointerException when setting batch.size=0 on producer config. This worked before with 0.10.1.1.
See https://issues.apache.org/jira/browse/KAFKA-4761
Regards,
Swen
Am 2/10/17, 5:51 PM schrieb "Ewen Cheslack-Postava" <ew...@confluent.io>:
Hello Kafka users, developers and client-developers,
This is RC1 for release of Apache Kafka 0.10.2.0.
This is a minor version release of Apache Kafka. It includes 19 new KIPs.
See the release notes and release plan (https://cwiki.apache.org/
confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A few
feature highlights: SASL-SCRAM support, improved client compatibility to
allow use of clients newer than the broker, session windows and global
tables in the Kafka Streams API, single message transforms in the Kafka
Connect framework.
Important note: in addition to the artifacts generated using JDK7 for Scala
2.10 and 2.11, this release also includes experimental artifacts built
using JDK8 for Scala 2.12.
Important code changes since RC0 (non-docs, non system tests):
* KAFKA-4728; KafkaConsumer#commitSync should copy its input
* KAFKA-4441; Monitoring incorrect during topic creation and deletion
* KAFKA-4734; Trim the time index on old segments
* KAFKA-4725; Stop leaking messages in produce request body when requests
are delayed
* KAFKA-4716: Fix case when controller cannot be reached
Release notes for the 0.10.2.0 release:
http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
*** Please download, test and vote by Monday, Feb 13, 5pm PT ***
Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS
* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/
* Javadoc:
http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
* Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
* Documentation:
http://kafka.apache.org/0102/documentation.html
* Protocol:
http://kafka.apache.org/0102/protocol.html
* Successful Jenkins builds for the 0.10.2 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-0.10.2-jdk7/74/
System tests: https://jenkins.confluent.io/job/system-test-kafka-0.10.2/25/
/**************************************
Thanks,
Ewen
Re: [VOTE] 0.10.2.0 RC1
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Ian,
I looked at the releasing artifact and tried running a simple streams app
with multiple threads with the jar but cannot reproduce your issue. If you
are still blocked on this I'm wondering if you could provide some code
sketch without any app business logic for me to try to reproduce it locally?
Guozhang
On Sun, Feb 12, 2017 at 2:54 AM, Ian Duffy <ia...@ianduffy.ie> wrote:
> Hi Matthias,
>
> Thank you for your fast response.
> I am not using any custom class loaders and it is the 10.2 jar that is
> being used.
>
> I'll try clearing out the state on next failure.
>
> The config parameters we are setting are:
>
> consumer:
> heartbeat.interval.ms = "100"
> auto.offset.reset = "earliest"
> group.id = "text-pipeline"
> session.timeout.ms = "10000"
> max.poll.interval.ms = "300000"
> max.poll.records = "500"
>
> stream:
> num.stream.threads = "8"
>
> Within 10.1 we seen this issue:
> https://issues.apache.org/jira/browse/KAFKA-4582 with 10.2 that has
> changed
> to the above exception and hanging on attempting to rejoin the group-id.
>
>
> On 11 February 2017 at 04:23, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > Hi Ian,
> >
> > thanks for reporting this. I had a look at the stack trace and code and
> > the whole situation is quite confusing. The exception itself is expected
> > but we have a try-catch-block that should swallow the exception and it
> > should never bubble up:
> >
> > In
> > AbstractTaskCreator.retryWithBackoff
> >
> > a call to
> > TaskCreator.createTask
> >
> > is done (cf your stack trace). This call is guarded against a
> > LockExcption (cf StreamThread.java code):
> >
> > > try {
> > > createTask(taskId, partitions);
> > > it.remove();
> > > } catch (final LockException e) {
> > > // ignore and retry
> > > log.warn("Could not create task {}. Will retry.", taskId, e);
> > > }
> >
> >
> > Can you verify, that you loaded the correct jar file when running the
> > test? Ie, not caching issue loading old code etc.
> >
> > Another theory is about class loading. Do you use custom class loaders?
> >
> > One more thing you can try out is to delete the local app state
> > directory. This will give you a clean restart -- on the cost of state
> > recreation (for the first start only). Afterward stop and restart you
> > app to see if the issue is resolved.
> >
> >
> > Right now, I cannot reproduce the problem.
> >
> >
> > -Matthias
> >
> >
> >
> > On 2/10/17 9:47 AM, Ian Duffy wrote:
> > > Seeing the following failure when using multi-threaded streams
> > >
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]:
> > > org.apache.kafka.streams.errors.LockException: task [0_21] Failed to
> > lock
> > > the state directory: /tmp/kafka-streams/text_pipeline_id/0_21
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.streams.processor.internals.
> > ProcessorStateManager.<init>(ProcessorStateManager.java:102)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.streams.processor.internals.AbstractTask.<init>(
> > AbstractTask.java:73)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.streams.processor.internals.
> > StreamTask.<init>(StreamTask.java:108)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.streams.processor.internals.
> > StreamThread.createStreamTask(StreamThread.java:834)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.
> > createTask(StreamThread.java:1207)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.streams.processor.internals.StreamThread$
> > AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.streams.processor.internals.
> > StreamThread.addStreamTasks(StreamThread.java:937)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.streams.processor.internals.StreamThread.access$500(
> > StreamThread.java:69)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.streams.processor.internals.StreamThread$1.
> > onPartitionsAssigned(StreamThread.java:236)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.
> > onJoinComplete(ConsumerCoordinator.java:255)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
> > joinGroupIfNeeded(AbstractCoordinator.java:339)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
> > ensureActiveGroup(AbstractCoordinator.java:303)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(
> > ConsumerCoordinator.java:286)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.clients.consumer.KafkaConsumer.
> > pollOnce(KafkaConsumer.java:1030)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.clients.consumer.KafkaConsumer.poll(
> > KafkaConsumer.java:995)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(
> > StreamThread.java:582)
> > > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > > org.apache.kafka.streams.processor.internals.
> > StreamThread.run(StreamThread.java:368)
> > >
> > > On restarting the process it continues to just hang on attempting to
> > rejoin
> > > the group id.
> > > Seen the same issue on 10.1 but thought it was due to be fixed for the
> > 10.2
> > > release.
> > >
> > > Thanks,
> > > Ian.
> > >
> > > On 10 February 2017 at 16:51, Ewen Cheslack-Postava <ewen@confluent.io
> >
> > > wrote:
> > >
> > >> Hello Kafka users, developers and client-developers,
> > >>
> > >> This is RC1 for release of Apache Kafka 0.10.2.0.
> > >>
> > >> This is a minor version release of Apache Kafka. It includes 19 new
> > KIPs.
> > >> See the release notes and release plan (https://cwiki.apache.org/
> > >> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A
> few
> > >> feature highlights: SASL-SCRAM support, improved client compatibility
> to
> > >> allow use of clients newer than the broker, session windows and global
> > >> tables in the Kafka Streams API, single message transforms in the
> Kafka
> > >> Connect framework.
> > >>
> > >> Important note: in addition to the artifacts generated using JDK7 for
> > Scala
> > >> 2.10 and 2.11, this release also includes experimental artifacts built
> > >> using JDK8 for Scala 2.12.
> > >>
> > >> Important code changes since RC0 (non-docs, non system tests):
> > >>
> > >> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
> > >> * KAFKA-4441; Monitoring incorrect during topic creation and deletion
> > >> * KAFKA-4734; Trim the time index on old segments
> > >> * KAFKA-4725; Stop leaking messages in produce request body when
> > requests
> > >> are delayed
> > >> * KAFKA-4716: Fix case when controller cannot be reached
> > >>
> > >> Release notes for the 0.10.2.0 release:
> > >> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
> > >>
> > >> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
> > >>
> > >> Kafka's KEYS file containing PGP keys we use to sign the release:
> > >> http://kafka.apache.org/KEYS
> > >>
> > >> * Release artifacts to be voted upon (source and binary):
> > >> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
> > >>
> > >> * Maven artifacts to be voted upon:
> > >> https://repository.apache.org/content/groups/staging/
> > >>
> > >> * Javadoc:
> > >> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
> > >>
> > >> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
> > >> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> > >> e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
> > >>
> > >>
> > >> * Documentation:
> > >> http://kafka.apache.org/0102/documentation.html
> > >>
> > >> * Protocol:
> > >> http://kafka.apache.org/0102/protocol.html
> > >>
> > >> * Successful Jenkins builds for the 0.10.2 branch:
> > >> Unit/integration tests: https://builds.apache.org/job/
> > >> kafka-0.10.2-jdk7/74/
> > >> System tests: https://jenkins.confluent.io/
> > job/system-test-kafka-0.10.2/
> > >> 25/
> > >>
> > >> /**************************************
> > >>
> > >> Thanks,
> > >> Ewen
> > >>
> > >
> >
> >
>
--
-- Guozhang
Re: [VOTE] 0.10.2.0 RC1
Posted by Ian Duffy <ia...@ianduffy.ie>.
Hi Matthias,
Thank you for your fast response.
I am not using any custom class loaders and it is the 10.2 jar that is
being used.
I'll try clearing out the state on next failure.
The config parameters we are setting are:
consumer:
heartbeat.interval.ms = "100"
auto.offset.reset = "earliest"
group.id = "text-pipeline"
session.timeout.ms = "10000"
max.poll.interval.ms = "300000"
max.poll.records = "500"
stream:
num.stream.threads = "8"
Within 10.1 we seen this issue:
https://issues.apache.org/jira/browse/KAFKA-4582 with 10.2 that has changed
to the above exception and hanging on attempting to rejoin the group-id.
On 11 February 2017 at 04:23, Matthias J. Sax <ma...@confluent.io> wrote:
> Hi Ian,
>
> thanks for reporting this. I had a look at the stack trace and code and
> the whole situation is quite confusing. The exception itself is expected
> but we have a try-catch-block that should swallow the exception and it
> should never bubble up:
>
> In
> AbstractTaskCreator.retryWithBackoff
>
> a call to
> TaskCreator.createTask
>
> is done (cf your stack trace). This call is guarded against a
> LockExcption (cf StreamThread.java code):
>
> > try {
> > createTask(taskId, partitions);
> > it.remove();
> > } catch (final LockException e) {
> > // ignore and retry
> > log.warn("Could not create task {}. Will retry.", taskId, e);
> > }
>
>
> Can you verify, that you loaded the correct jar file when running the
> test? Ie, not caching issue loading old code etc.
>
> Another theory is about class loading. Do you use custom class loaders?
>
> One more thing you can try out is to delete the local app state
> directory. This will give you a clean restart -- on the cost of state
> recreation (for the first start only). Afterward stop and restart you
> app to see if the issue is resolved.
>
>
> Right now, I cannot reproduce the problem.
>
>
> -Matthias
>
>
>
> On 2/10/17 9:47 AM, Ian Duffy wrote:
> > Seeing the following failure when using multi-threaded streams
> >
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]:
> > org.apache.kafka.streams.errors.LockException: task [0_21] Failed to
> lock
> > the state directory: /tmp/kafka-streams/text_pipeline_id/0_21
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.streams.processor.internals.
> ProcessorStateManager.<init>(ProcessorStateManager.java:102)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.streams.processor.internals.AbstractTask.<init>(
> AbstractTask.java:73)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.streams.processor.internals.
> StreamTask.<init>(StreamTask.java:108)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.streams.processor.internals.
> StreamThread.createStreamTask(StreamThread.java:834)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.
> createTask(StreamThread.java:1207)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.streams.processor.internals.StreamThread$
> AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.streams.processor.internals.
> StreamThread.addStreamTasks(StreamThread.java:937)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.streams.processor.internals.StreamThread.access$500(
> StreamThread.java:69)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.streams.processor.internals.StreamThread$1.
> onPartitionsAssigned(StreamThread.java:236)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.
> onJoinComplete(ConsumerCoordinator.java:255)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
> joinGroupIfNeeded(AbstractCoordinator.java:339)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
> ensureActiveGroup(AbstractCoordinator.java:303)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(
> ConsumerCoordinator.java:286)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.clients.consumer.KafkaConsumer.
> pollOnce(KafkaConsumer.java:1030)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.clients.consumer.KafkaConsumer.poll(
> KafkaConsumer.java:995)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(
> StreamThread.java:582)
> > Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> > org.apache.kafka.streams.processor.internals.
> StreamThread.run(StreamThread.java:368)
> >
> > On restarting the process it continues to just hang on attempting to
> rejoin
> > the group id.
> > Seen the same issue on 10.1 but thought it was due to be fixed for the
> 10.2
> > release.
> >
> > Thanks,
> > Ian.
> >
> > On 10 February 2017 at 16:51, Ewen Cheslack-Postava <ew...@confluent.io>
> > wrote:
> >
> >> Hello Kafka users, developers and client-developers,
> >>
> >> This is RC1 for release of Apache Kafka 0.10.2.0.
> >>
> >> This is a minor version release of Apache Kafka. It includes 19 new
> KIPs.
> >> See the release notes and release plan (https://cwiki.apache.org/
> >> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A few
> >> feature highlights: SASL-SCRAM support, improved client compatibility to
> >> allow use of clients newer than the broker, session windows and global
> >> tables in the Kafka Streams API, single message transforms in the Kafka
> >> Connect framework.
> >>
> >> Important note: in addition to the artifacts generated using JDK7 for
> Scala
> >> 2.10 and 2.11, this release also includes experimental artifacts built
> >> using JDK8 for Scala 2.12.
> >>
> >> Important code changes since RC0 (non-docs, non system tests):
> >>
> >> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
> >> * KAFKA-4441; Monitoring incorrect during topic creation and deletion
> >> * KAFKA-4734; Trim the time index on old segments
> >> * KAFKA-4725; Stop leaking messages in produce request body when
> requests
> >> are delayed
> >> * KAFKA-4716: Fix case when controller cannot be reached
> >>
> >> Release notes for the 0.10.2.0 release:
> >> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
> >>
> >> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
> >>
> >> Kafka's KEYS file containing PGP keys we use to sign the release:
> >> http://kafka.apache.org/KEYS
> >>
> >> * Release artifacts to be voted upon (source and binary):
> >> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
> >>
> >> * Maven artifacts to be voted upon:
> >> https://repository.apache.org/content/groups/staging/
> >>
> >> * Javadoc:
> >> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
> >>
> >> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
> >> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> >> e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
> >>
> >>
> >> * Documentation:
> >> http://kafka.apache.org/0102/documentation.html
> >>
> >> * Protocol:
> >> http://kafka.apache.org/0102/protocol.html
> >>
> >> * Successful Jenkins builds for the 0.10.2 branch:
> >> Unit/integration tests: https://builds.apache.org/job/
> >> kafka-0.10.2-jdk7/74/
> >> System tests: https://jenkins.confluent.io/
> job/system-test-kafka-0.10.2/
> >> 25/
> >>
> >> /**************************************
> >>
> >> Thanks,
> >> Ewen
> >>
> >
>
>
Re: [VOTE] 0.10.2.0 RC1
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Hi Ian,
thanks for reporting this. I had a look at the stack trace and code and
the whole situation is quite confusing. The exception itself is expected
but we have a try-catch-block that should swallow the exception and it
should never bubble up:
In
AbstractTaskCreator.retryWithBackoff
a call to
TaskCreator.createTask
is done (cf your stack trace). This call is guarded against a
LockExcption (cf StreamThread.java code):
> try {
> createTask(taskId, partitions);
> it.remove();
> } catch (final LockException e) {
> // ignore and retry
> log.warn("Could not create task {}. Will retry.", taskId, e);
> }
Can you verify, that you loaded the correct jar file when running the
test? Ie, not caching issue loading old code etc.
Another theory is about class loading. Do you use custom class loaders?
One more thing you can try out is to delete the local app state
directory. This will give you a clean restart -- on the cost of state
recreation (for the first start only). Afterward stop and restart you
app to see if the issue is resolved.
Right now, I cannot reproduce the problem.
-Matthias
On 2/10/17 9:47 AM, Ian Duffy wrote:
> Seeing the following failure when using multi-threaded streams
>
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]:
> org.apache.kafka.streams.errors.LockException: task [0_21] Failed to lock
> the state directory: /tmp/kafka-streams/text_pipeline_id/0_21
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.<init>(ProcessorStateManager.java:102)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.AbstractTask.<init>(AbstractTask.java:73)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:108)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
>
> On restarting the process it continues to just hang on attempting to rejoin
> the group id.
> Seen the same issue on 10.1 but thought it was due to be fixed for the 10.2
> release.
>
> Thanks,
> Ian.
>
> On 10 February 2017 at 16:51, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is RC1 for release of Apache Kafka 0.10.2.0.
>>
>> This is a minor version release of Apache Kafka. It includes 19 new KIPs.
>> See the release notes and release plan (https://cwiki.apache.org/
>> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A few
>> feature highlights: SASL-SCRAM support, improved client compatibility to
>> allow use of clients newer than the broker, session windows and global
>> tables in the Kafka Streams API, single message transforms in the Kafka
>> Connect framework.
>>
>> Important note: in addition to the artifacts generated using JDK7 for Scala
>> 2.10 and 2.11, this release also includes experimental artifacts built
>> using JDK8 for Scala 2.12.
>>
>> Important code changes since RC0 (non-docs, non system tests):
>>
>> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
>> * KAFKA-4441; Monitoring incorrect during topic creation and deletion
>> * KAFKA-4734; Trim the time index on old segments
>> * KAFKA-4725; Stop leaking messages in produce request body when requests
>> are delayed
>> * KAFKA-4716: Fix case when controller cannot be reached
>>
>> Release notes for the 0.10.2.0 release:
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/
>>
>> * Javadoc:
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
>>
>> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
>> e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
>>
>>
>> * Documentation:
>> http://kafka.apache.org/0102/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/0102/protocol.html
>>
>> * Successful Jenkins builds for the 0.10.2 branch:
>> Unit/integration tests: https://builds.apache.org/job/
>> kafka-0.10.2-jdk7/74/
>> System tests: https://jenkins.confluent.io/job/system-test-kafka-0.10.2/
>> 25/
>>
>> /**************************************
>>
>> Thanks,
>> Ewen
>>
>
Re: [VOTE] 0.10.2.0 RC1
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Hi Ian,
thanks for reporting this. I had a look at the stack trace and code and
the whole situation is quite confusing. The exception itself is expected
but we have a try-catch-block that should swallow the exception and it
should never bubble up:
In
AbstractTaskCreator.retryWithBackoff
a call to
TaskCreator.createTask
is done (cf your stack trace). This call is guarded against a
LockExcption (cf StreamThread.java code):
> try {
> createTask(taskId, partitions);
> it.remove();
> } catch (final LockException e) {
> // ignore and retry
> log.warn("Could not create task {}. Will retry.", taskId, e);
> }
Can you verify, that you loaded the correct jar file when running the
test? Ie, not caching issue loading old code etc.
Another theory is about class loading. Do you use custom class loaders?
One more thing you can try out is to delete the local app state
directory. This will give you a clean restart -- on the cost of state
recreation (for the first start only). Afterward stop and restart you
app to see if the issue is resolved.
Right now, I cannot reproduce the problem.
-Matthias
On 2/10/17 9:47 AM, Ian Duffy wrote:
> Seeing the following failure when using multi-threaded streams
>
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]:
> org.apache.kafka.streams.errors.LockException: task [0_21] Failed to lock
> the state directory: /tmp/kafka-streams/text_pipeline_id/0_21
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.<init>(ProcessorStateManager.java:102)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.AbstractTask.<init>(AbstractTask.java:73)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:108)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582)
> Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
>
> On restarting the process it continues to just hang on attempting to rejoin
> the group id.
> Seen the same issue on 10.1 but thought it was due to be fixed for the 10.2
> release.
>
> Thanks,
> Ian.
>
> On 10 February 2017 at 16:51, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is RC1 for release of Apache Kafka 0.10.2.0.
>>
>> This is a minor version release of Apache Kafka. It includes 19 new KIPs.
>> See the release notes and release plan (https://cwiki.apache.org/
>> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A few
>> feature highlights: SASL-SCRAM support, improved client compatibility to
>> allow use of clients newer than the broker, session windows and global
>> tables in the Kafka Streams API, single message transforms in the Kafka
>> Connect framework.
>>
>> Important note: in addition to the artifacts generated using JDK7 for Scala
>> 2.10 and 2.11, this release also includes experimental artifacts built
>> using JDK8 for Scala 2.12.
>>
>> Important code changes since RC0 (non-docs, non system tests):
>>
>> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
>> * KAFKA-4441; Monitoring incorrect during topic creation and deletion
>> * KAFKA-4734; Trim the time index on old segments
>> * KAFKA-4725; Stop leaking messages in produce request body when requests
>> are delayed
>> * KAFKA-4716: Fix case when controller cannot be reached
>>
>> Release notes for the 0.10.2.0 release:
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/
>>
>> * Javadoc:
>> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
>>
>> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
>> e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
>>
>>
>> * Documentation:
>> http://kafka.apache.org/0102/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/0102/protocol.html
>>
>> * Successful Jenkins builds for the 0.10.2 branch:
>> Unit/integration tests: https://builds.apache.org/job/
>> kafka-0.10.2-jdk7/74/
>> System tests: https://jenkins.confluent.io/job/system-test-kafka-0.10.2/
>> 25/
>>
>> /**************************************
>>
>> Thanks,
>> Ewen
>>
>
Re: [VOTE] 0.10.2.0 RC1
Posted by Ian Duffy <ia...@ianduffy.ie>.
Seeing the following failure when using multi-threaded streams
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]:
org.apache.kafka.streams.errors.LockException: task [0_21] Failed to lock
the state directory: /tmp/kafka-streams/text_pipeline_id/0_21
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.ProcessorStateManager.<init>(ProcessorStateManager.java:102)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.AbstractTask.<init>(AbstractTask.java:73)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:108)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
On restarting the process it continues to just hang on attempting to rejoin
the group id.
Seen the same issue on 10.1 but thought it was due to be fixed for the 10.2
release.
Thanks,
Ian.
On 10 February 2017 at 16:51, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Hello Kafka users, developers and client-developers,
>
> This is RC1 for release of Apache Kafka 0.10.2.0.
>
> This is a minor version release of Apache Kafka. It includes 19 new KIPs.
> See the release notes and release plan (https://cwiki.apache.org/
> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A few
> feature highlights: SASL-SCRAM support, improved client compatibility to
> allow use of clients newer than the broker, session windows and global
> tables in the Kafka Streams API, single message transforms in the Kafka
> Connect framework.
>
> Important note: in addition to the artifacts generated using JDK7 for Scala
> 2.10 and 2.11, this release also includes experimental artifacts built
> using JDK8 for Scala 2.12.
>
> Important code changes since RC0 (non-docs, non system tests):
>
> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
> * KAFKA-4441; Monitoring incorrect during topic creation and deletion
> * KAFKA-4734; Trim the time index on old segments
> * KAFKA-4725; Stop leaking messages in produce request body when requests
> are delayed
> * KAFKA-4716: Fix case when controller cannot be reached
>
> Release notes for the 0.10.2.0 release:
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
>
> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
>
>
> * Documentation:
> http://kafka.apache.org/0102/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0102/protocol.html
>
> * Successful Jenkins builds for the 0.10.2 branch:
> Unit/integration tests: https://builds.apache.org/job/
> kafka-0.10.2-jdk7/74/
> System tests: https://jenkins.confluent.io/job/system-test-kafka-0.10.2/
> 25/
>
> /**************************************
>
> Thanks,
> Ewen
>
Re: [VOTE] 0.10.2.0 RC1
Posted by Eno Thereska <en...@gmail.com>.
+1 (non binding)
Checked streams. Verified that stream tests work and examples off confluentinc/examples/kafka-streams work.
Thanks
Eno
> On 10 Feb 2017, at 16:51, Ewen Cheslack-Postava <ew...@confluent.io> wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is RC1 for release of Apache Kafka 0.10.2.0.
>
> This is a minor version release of Apache Kafka. It includes 19 new KIPs.
> See the release notes and release plan (https://cwiki.apache.org/
> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A few
> feature highlights: SASL-SCRAM support, improved client compatibility to
> allow use of clients newer than the broker, session windows and global
> tables in the Kafka Streams API, single message transforms in the Kafka
> Connect framework.
>
> Important note: in addition to the artifacts generated using JDK7 for Scala
> 2.10 and 2.11, this release also includes experimental artifacts built
> using JDK8 for Scala 2.12.
>
> Important code changes since RC0 (non-docs, non system tests):
>
> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
> * KAFKA-4441; Monitoring incorrect during topic creation and deletion
> * KAFKA-4734; Trim the time index on old segments
> * KAFKA-4725; Stop leaking messages in produce request body when requests
> are delayed
> * KAFKA-4716: Fix case when controller cannot be reached
>
> Release notes for the 0.10.2.0 release:
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
>
> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
>
>
> * Documentation:
> http://kafka.apache.org/0102/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0102/protocol.html
>
> * Successful Jenkins builds for the 0.10.2 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-0.10.2-jdk7/74/
> System tests: https://jenkins.confluent.io/job/system-test-kafka-0.10.2/25/
>
> /**************************************
>
> Thanks,
> Ewen
Re: [VOTE] 0.10.2.0 RC1
Posted by "Moczarski, Swen" <sm...@ebay-kleinanzeigen.de>.
+0 (non-binding)
Thanks for compiling a new release candidate.
I get an NullPointerException when setting batch.size=0 on producer config. This worked before with 0.10.1.1.
See https://issues.apache.org/jira/browse/KAFKA-4761
Regards,
Swen
Am 2/10/17, 5:51 PM schrieb "Ewen Cheslack-Postava" <ew...@confluent.io>:
Hello Kafka users, developers and client-developers,
This is RC1 for release of Apache Kafka 0.10.2.0.
This is a minor version release of Apache Kafka. It includes 19 new KIPs.
See the release notes and release plan (https://cwiki.apache.org/
confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A few
feature highlights: SASL-SCRAM support, improved client compatibility to
allow use of clients newer than the broker, session windows and global
tables in the Kafka Streams API, single message transforms in the Kafka
Connect framework.
Important note: in addition to the artifacts generated using JDK7 for Scala
2.10 and 2.11, this release also includes experimental artifacts built
using JDK8 for Scala 2.12.
Important code changes since RC0 (non-docs, non system tests):
* KAFKA-4728; KafkaConsumer#commitSync should copy its input
* KAFKA-4441; Monitoring incorrect during topic creation and deletion
* KAFKA-4734; Trim the time index on old segments
* KAFKA-4725; Stop leaking messages in produce request body when requests
are delayed
* KAFKA-4716: Fix case when controller cannot be reached
Release notes for the 0.10.2.0 release:
http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
*** Please download, test and vote by Monday, Feb 13, 5pm PT ***
Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS
* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/
* Javadoc:
http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
* Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
* Documentation:
http://kafka.apache.org/0102/documentation.html
* Protocol:
http://kafka.apache.org/0102/protocol.html
* Successful Jenkins builds for the 0.10.2 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-0.10.2-jdk7/74/
System tests: https://jenkins.confluent.io/job/system-test-kafka-0.10.2/25/
/**************************************
Thanks,
Ewen
Re: [VOTE] 0.10.2.0 RC1
Posted by Ian Duffy <ia...@ianduffy.ie>.
Seeing the following failure when using multi-threaded streams
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]:
org.apache.kafka.streams.errors.LockException: task [0_21] Failed to lock
the state directory: /tmp/kafka-streams/text_pipeline_id/0_21
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.ProcessorStateManager.<init>(ProcessorStateManager.java:102)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.AbstractTask.<init>(AbstractTask.java:73)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:108)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582)
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: #011at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
On restarting the process it continues to just hang on attempting to rejoin
the group id.
Seen the same issue on 10.1 but thought it was due to be fixed for the 10.2
release.
Thanks,
Ian.
On 10 February 2017 at 16:51, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Hello Kafka users, developers and client-developers,
>
> This is RC1 for release of Apache Kafka 0.10.2.0.
>
> This is a minor version release of Apache Kafka. It includes 19 new KIPs.
> See the release notes and release plan (https://cwiki.apache.org/
> confluence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A few
> feature highlights: SASL-SCRAM support, improved client compatibility to
> allow use of clients newer than the broker, session windows and global
> tables in the Kafka Streams API, single message transforms in the Kafka
> Connect framework.
>
> Important note: in addition to the artifacts generated using JDK7 for Scala
> 2.10 and 2.11, this release also includes experimental artifacts built
> using JDK8 for Scala 2.12.
>
> Important code changes since RC0 (non-docs, non system tests):
>
> * KAFKA-4728; KafkaConsumer#commitSync should copy its input
> * KAFKA-4441; Monitoring incorrect during topic creation and deletion
> * KAFKA-4734; Trim the time index on old segments
> * KAFKA-4725; Stop leaking messages in produce request body when requests
> are delayed
> * KAFKA-4716: Fix case when controller cannot be reached
>
> Release notes for the 0.10.2.0 release:
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Feb 13, 5pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~ewencp/kafka-0.10.2.0-rc1/javadoc/
>
> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> e825b7994bf8c8c4871d1e0973e287e6d31c7ae4
>
>
> * Documentation:
> http://kafka.apache.org/0102/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0102/protocol.html
>
> * Successful Jenkins builds for the 0.10.2 branch:
> Unit/integration tests: https://builds.apache.org/job/
> kafka-0.10.2-jdk7/74/
> System tests: https://jenkins.confluent.io/job/system-test-kafka-0.10.2/
> 25/
>
> /**************************************
>
> Thanks,
> Ewen
>