You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Fredrik Emilsson <Fr...@mblox.com> on 2012/08/09 13:55:29 UTC
Partition problems
Hello,
I'm trying to send messages to a topic with many partitions but only
partition 0 is created and contains a kafka-file. There is no error when
adding the messages. I'm using kafka 0.7.0 and running a default Kafka
server.
Anyone that have any idea what's the problem? I includes my simple
message generator and the configuration.
Regards,
Fredrik
NOTICE - This message and any attached files may contain information that is confidential and/or subject of legal privilege intended only for use by the intended recipient. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, be advised that you have received this message in error and that any dissemination, copying or use of this message or attachment is strictly forbidden, as is the disclosure of the information therein. If you have received this message in error please notify the sender immediately and delete the message.
RE: Partition problems
Posted by Fredrik Emilsson <Fr...@mblox.com>.
Thanks. I used wrong properties-file where num.partitions was 1. Now it
works well!
Regards,
Fredrik
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: 09 August 2012 17:06
To: kafka-users@incubator.apache.org
Subject: Re: Partition problems
Did you configure num.partitions on the broker to be larger than 1?
Thanks,
Jun
On Thu, Aug 9, 2012 at 5:14 AM, Fredrik Emilsson
<Fredrik.Emilsson@mblox.com
> wrote:
> I also found some relevant error in the kafka server log:
>
>
>
> [2012-08-09 14:11:54,007] WARN Wrong partition 1 valid partitions
> (0,0)
> (kafka.log.LogManager)
>
> [2012-08-09 14:11:54,007] ERROR Error processing MultiProducerRequest
> on
> STest0:1 (kafka.server.KafkaRequestHandlers)
>
> kafka.common.InvalidPartitionException: wrong partition 1
>
> at kafka.log.LogManager.getOrCreateLog(LogManager.scala:169)
>
> at
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$h
> an
> dleProducerRequest(KafkaRequestHandlers.scala:75)
>
> at
>
kafka.server.KafkaRequestHandlers$$anonfun$handleMultiProducerRequest$1.
> apply(KafkaRequestHandlers.scala:68)
>
> at
>
kafka.server.KafkaRequestHandlers$$anonfun$handleMultiProducerRequest$1.
> apply(KafkaRequestHandlers.scala:68)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.
> sc
> ala:206)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.
> sc
> ala:206)
>
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized
> .s
> cala:34)
>
> at
> scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
>
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
>
> at scala.collection.mutable.ArrayOps.map(ArrayOps.scala:34)
>
> at
> kafka.server.KafkaRequestHandlers.handleMultiProducerRequest(KafkaRequ
> es
> tHandlers.scala:68)
>
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaReq
> ue
> stHandlers.scala:46)
>
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaReq
> ue
> stHandlers.scala:46)
>
> at kafka.network.Processor.handle(SocketServer.scala:289)
>
> at kafka.network.Processor.read(SocketServer.scala:312)
>
> at kafka.network.Processor.run(SocketServer.scala:207)
>
> at java.lang.Thread.run(Thread.java:722)
>
> [2012-08-09 14:11:54,009] ERROR Closing socket for /192.168.0.54
> because of error (kafka.network.Processor)
>
> kafka.common.InvalidPartitionException: wrong partition 1
>
> at kafka.log.LogManager.getOrCreateLog(LogManager.scala:169)
>
> at
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$h
> an
> dleProducerRequest(KafkaRequestHandlers.scala:75)
>
> at
>
kafka.server.KafkaRequestHandlers$$anonfun$handleMultiProducerRequest$1.
> apply(KafkaRequestHandlers.scala:68)
>
> at
>
kafka.server.KafkaRequestHandlers$$anonfun$handleMultiProducerRequest$1.
> apply(KafkaRequestHandlers.scala:68)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.
> sc
> ala:206)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.
> sc
> ala:206)
>
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized
> .s
> cala:34)
>
> at
> scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
>
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
>
> at scala.collection.mutable.ArrayOps.map(ArrayOps.scala:34)
>
> at
> kafka.server.KafkaRequestHandlers.handleMultiProducerRequest(KafkaRequ
> es
> tHandlers.scala:68)
>
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaReq
> ue
> stHandlers.scala:46)
>
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaReq
> ue
> stHandlers.scala:46)
>
> at kafka.network.Processor.handle(SocketServer.scala:289)
>
> at kafka.network.Processor.read(SocketServer.scala:312)
>
> at kafka.network.Processor.run(SocketServer.scala:207)
>
> at java.lang.Thread.run(Thread.java:722)
>
> [2012-08-09 14:11:54,044] INFO Closing socket connection to
> /192.168.0.54. (kafka.network.Processor)
>
>
>
> From: Fredrik Emilsson [mailto:Fredrik.Emilsson@mblox.com]
> Sent: 09 August 2012 13:55
> To: kafka-users@incubator.apache.org
> Subject: Partition problems
>
>
>
> Hello,
>
>
>
> I'm trying to send messages to a topic with many partitions but only
> partition 0 is created and contains a kafka-file. There is no error
> when adding the messages. I'm using kafka 0.7.0 and running a default
> Kafka server.
>
>
>
> Anyone that have any idea what's the problem? I includes my simple
> message generator and the configuration.
>
>
>
> Regards,
>
> Fredrik
>
>
>
>
>
>
>
>
>
>
> NOTICE - This message and any attached files may contain information
> that is confidential and/or subject of legal privilege intended only
> for use by the intended recipient. If you are not the intended
> recipient or the person responsible for delivering the message to the
> intended recipient, be advised that you have received this message in
> error and that any dissemination, copying or use of this message or
> attachment is strictly forbidden, as is the disclosure of the
> information therein. If you have received this message in error please
notify the sender
> immediately and delete the message. --
>
>
> NOTICE - This message and any attached files may contain information
> that is confidential and/or subject of legal privilege intended only
> for use by the intended recipient. If you are not the intended
> recipient or the person responsible for delivering the message to the
> intended recipient, be advised that you have received this message in
> error and that any dissemination, copying or use of this message or
> attachment is strictly forbidden, as is the disclosure of the
> information therein. If you have received this message in error
> please notify the sender immediately and delete the message.
>
NOTICE - This message and any attached files may contain information that is confidential and/or subject of legal privilege intended only for use by the intended recipient. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, be advised that you have received this message in error and that any dissemination, copying or use of this message or attachment is strictly forbidden, as is the disclosure of the information therein. If you have received this message in error please notify the sender immediately and delete the message.
Re: Partition problems
Posted by Jun Rao <ju...@gmail.com>.
Did you configure num.partitions on the broker to be larger than 1?
Thanks,
Jun
On Thu, Aug 9, 2012 at 5:14 AM, Fredrik Emilsson <Fredrik.Emilsson@mblox.com
> wrote:
> I also found some relevant error in the kafka server log:
>
>
>
> [2012-08-09 14:11:54,007] WARN Wrong partition 1 valid partitions (0,0)
> (kafka.log.LogManager)
>
> [2012-08-09 14:11:54,007] ERROR Error processing MultiProducerRequest on
> STest0:1 (kafka.server.KafkaRequestHandlers)
>
> kafka.common.InvalidPartitionException: wrong partition 1
>
> at kafka.log.LogManager.getOrCreateLog(LogManager.scala:169)
>
> at
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$han
> dleProducerRequest(KafkaRequestHandlers.scala:75)
>
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handleMultiProducerRequest$1.
> apply(KafkaRequestHandlers.scala:68)
>
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handleMultiProducerRequest$1.
> apply(KafkaRequestHandlers.scala:68)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sc
> ala:206)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sc
> ala:206)
>
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.s
> cala:34)
>
> at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
>
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
>
> at scala.collection.mutable.ArrayOps.map(ArrayOps.scala:34)
>
> at
> kafka.server.KafkaRequestHandlers.handleMultiProducerRequest(KafkaReques
> tHandlers.scala:68)
>
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaReque
> stHandlers.scala:46)
>
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaReque
> stHandlers.scala:46)
>
> at kafka.network.Processor.handle(SocketServer.scala:289)
>
> at kafka.network.Processor.read(SocketServer.scala:312)
>
> at kafka.network.Processor.run(SocketServer.scala:207)
>
> at java.lang.Thread.run(Thread.java:722)
>
> [2012-08-09 14:11:54,009] ERROR Closing socket for /192.168.0.54 because
> of error (kafka.network.Processor)
>
> kafka.common.InvalidPartitionException: wrong partition 1
>
> at kafka.log.LogManager.getOrCreateLog(LogManager.scala:169)
>
> at
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$han
> dleProducerRequest(KafkaRequestHandlers.scala:75)
>
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handleMultiProducerRequest$1.
> apply(KafkaRequestHandlers.scala:68)
>
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handleMultiProducerRequest$1.
> apply(KafkaRequestHandlers.scala:68)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sc
> ala:206)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sc
> ala:206)
>
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.s
> cala:34)
>
> at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
>
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
>
> at scala.collection.mutable.ArrayOps.map(ArrayOps.scala:34)
>
> at
> kafka.server.KafkaRequestHandlers.handleMultiProducerRequest(KafkaReques
> tHandlers.scala:68)
>
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaReque
> stHandlers.scala:46)
>
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaReque
> stHandlers.scala:46)
>
> at kafka.network.Processor.handle(SocketServer.scala:289)
>
> at kafka.network.Processor.read(SocketServer.scala:312)
>
> at kafka.network.Processor.run(SocketServer.scala:207)
>
> at java.lang.Thread.run(Thread.java:722)
>
> [2012-08-09 14:11:54,044] INFO Closing socket connection to
> /192.168.0.54. (kafka.network.Processor)
>
>
>
> From: Fredrik Emilsson [mailto:Fredrik.Emilsson@mblox.com]
> Sent: 09 August 2012 13:55
> To: kafka-users@incubator.apache.org
> Subject: Partition problems
>
>
>
> Hello,
>
>
>
> I'm trying to send messages to a topic with many partitions but only
> partition 0 is created and contains a kafka-file. There is no error when
> adding the messages. I'm using kafka 0.7.0 and running a default Kafka
> server.
>
>
>
> Anyone that have any idea what's the problem? I includes my simple
> message generator and the configuration.
>
>
>
> Regards,
>
> Fredrik
>
>
>
>
>
>
>
>
>
>
> NOTICE - This message and any attached files may contain information
> that is confidential and/or subject of legal privilege intended only for
> use by the intended recipient. If you are not the intended recipient or
> the person responsible for delivering the message to the intended
> recipient, be advised that you have received this message in error and
> that any dissemination, copying or use of this message or attachment is
> strictly forbidden, as is the disclosure of the information therein. If
> you have received this message in error please notify the sender
> immediately and delete the message. --
>
>
> NOTICE - This message and any attached files may contain information that
> is confidential and/or subject of legal privilege intended only for use by
> the intended recipient. If you are not the intended recipient or the person
> responsible for delivering the message to the intended recipient, be
> advised that you have received this message in error and that any
> dissemination, copying or use of this message or attachment is strictly
> forbidden, as is the disclosure of the information therein. If you have
> received this message in error please notify the sender immediately and
> delete the message.
>
RE: Partition problems
Posted by Fredrik Emilsson <Fr...@mblox.com>.
I also found some relevant error in the kafka server log:
[2012-08-09 14:11:54,007] WARN Wrong partition 1 valid partitions (0,0)
(kafka.log.LogManager)
[2012-08-09 14:11:54,007] ERROR Error processing MultiProducerRequest on
STest0:1 (kafka.server.KafkaRequestHandlers)
kafka.common.InvalidPartitionException: wrong partition 1
at kafka.log.LogManager.getOrCreateLog(LogManager.scala:169)
at
kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$han
dleProducerRequest(KafkaRequestHandlers.scala:75)
at
kafka.server.KafkaRequestHandlers$$anonfun$handleMultiProducerRequest$1.
apply(KafkaRequestHandlers.scala:68)
at
kafka.server.KafkaRequestHandlers$$anonfun$handleMultiProducerRequest$1.
apply(KafkaRequestHandlers.scala:68)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sc
ala:206)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sc
ala:206)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.s
cala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.mutable.ArrayOps.map(ArrayOps.scala:34)
at
kafka.server.KafkaRequestHandlers.handleMultiProducerRequest(KafkaReques
tHandlers.scala:68)
at
kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaReque
stHandlers.scala:46)
at
kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaReque
stHandlers.scala:46)
at kafka.network.Processor.handle(SocketServer.scala:289)
at kafka.network.Processor.read(SocketServer.scala:312)
at kafka.network.Processor.run(SocketServer.scala:207)
at java.lang.Thread.run(Thread.java:722)
[2012-08-09 14:11:54,009] ERROR Closing socket for /192.168.0.54 because
of error (kafka.network.Processor)
kafka.common.InvalidPartitionException: wrong partition 1
at kafka.log.LogManager.getOrCreateLog(LogManager.scala:169)
at
kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$han
dleProducerRequest(KafkaRequestHandlers.scala:75)
at
kafka.server.KafkaRequestHandlers$$anonfun$handleMultiProducerRequest$1.
apply(KafkaRequestHandlers.scala:68)
at
kafka.server.KafkaRequestHandlers$$anonfun$handleMultiProducerRequest$1.
apply(KafkaRequestHandlers.scala:68)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sc
ala:206)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sc
ala:206)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.s
cala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.mutable.ArrayOps.map(ArrayOps.scala:34)
at
kafka.server.KafkaRequestHandlers.handleMultiProducerRequest(KafkaReques
tHandlers.scala:68)
at
kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaReque
stHandlers.scala:46)
at
kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaReque
stHandlers.scala:46)
at kafka.network.Processor.handle(SocketServer.scala:289)
at kafka.network.Processor.read(SocketServer.scala:312)
at kafka.network.Processor.run(SocketServer.scala:207)
at java.lang.Thread.run(Thread.java:722)
[2012-08-09 14:11:54,044] INFO Closing socket connection to
/192.168.0.54. (kafka.network.Processor)
From: Fredrik Emilsson [mailto:Fredrik.Emilsson@mblox.com]
Sent: 09 August 2012 13:55
To: kafka-users@incubator.apache.org
Subject: Partition problems
Hello,
I'm trying to send messages to a topic with many partitions but only
partition 0 is created and contains a kafka-file. There is no error when
adding the messages. I'm using kafka 0.7.0 and running a default Kafka
server.
Anyone that have any idea what's the problem? I includes my simple
message generator and the configuration.
Regards,
Fredrik
NOTICE - This message and any attached files may contain information
that is confidential and/or subject of legal privilege intended only for
use by the intended recipient. If you are not the intended recipient or
the person responsible for delivering the message to the intended
recipient, be advised that you have received this message in error and
that any dissemination, copying or use of this message or attachment is
strictly forbidden, as is the disclosure of the information therein. If
you have received this message in error please notify the sender
immediately and delete the message. --
NOTICE - This message and any attached files may contain information that is confidential and/or subject of legal privilege intended only for use by the intended recipient. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, be advised that you have received this message in error and that any dissemination, copying or use of this message or attachment is strictly forbidden, as is the disclosure of the information therein. If you have received this message in error please notify the sender immediately and delete the message.