You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by David Montgomery <da...@gmail.com> on 2015/07/12 06:08:11 UTC
stunning error - Request of length 1550939497 is not valid, it is
larger than the maximum size of 104857600 bytes
I cant send this soooo simple payload using python.
topic: topic-test-development
payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
No handlers could be found for logger "kafka.conn"
Traceback (most recent call last):
File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line 81,
in <module>
test_send_data_to_realtimenode()
File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line 38,
in test_send_data_to_realtimenode
response = producer.send_messages(test_topic,test_payload)
File "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
line 54, in send_messages
topic, partition, *msg
File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
line 349, in send_messages
return self._send_messages(topic, partition, *msg)
File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
line 390, in _send_messages
fail_on_error=self.sync_fail_on_error
File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 480,
in send_produce_request
(not fail_on_error or not self._raise_on_response_error(resp))]
File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 247,
in _raise_on_response_error
raise resp
kafka.common.FailedPayloadsError
Here is what is in my logs
[2015-07-12 03:29:58,103] INFO Closing socket connection to
/xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497 is
not valid, it is larger than the maximum size of 104857600 bytes.
(kafka.network.Processor)
Server is 4 gigs of ram.
I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in kafka-server-start.sh
So.....why?
Re: stunning error - Request of length 1550939497 is not valid, it is
larger than the maximum size of 104857600 bytes
Posted by Todd Palino <tp...@gmail.com>.
It could be a client error, but we're seeing it show up in Mirror Maker.
-Todd
On Tue, Jul 14, 2015 at 1:27 PM, JIEFU GONG <jg...@berkeley.edu> wrote:
> Got it, looks like I didn't understand the request process and am failing
> to use AB properly. Thanks for the help everyone! I suspect you might be
> running into a similar error, David.
>
> On Tue, Jul 14, 2015 at 11:56 AM, Jay Kreps <ja...@confluent.io> wrote:
>
> > This is almost certainly a client bug. Kafka's request format is size
> > delimited messages in the form
> > <4 byte size N><N byte payload>
> > If the client sends a request with an invalid size or sends a partial
> > request the server will see effectively random bytes from the next
> request
> > as the size of the next message and generally reject the request (or fail
> > to parse it).
> >
> > -Jay
> >
> > On Sat, Jul 11, 2015 at 9:08 PM, David Montgomery <
> > davidmontgomery@gmail.com
> > > wrote:
> >
> > > I cant send this soooo simple payload using python.
> > >
> > > topic: topic-test-development
> > > payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
> > >
> > >
> > > No handlers could be found for logger "kafka.conn"
> > > Traceback (most recent call last):
> > > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
> > 81,
> > > in <module>
> > > test_send_data_to_realtimenode()
> > > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
> > 38,
> > > in test_send_data_to_realtimenode
> > > response = producer.send_messages(test_topic,test_payload)
> > > File
> "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
> > > line 54, in send_messages
> > > topic, partition, *msg
> > > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > > line 349, in send_messages
> > > return self._send_messages(topic, partition, *msg)
> > > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > > line 390, in _send_messages
> > > fail_on_error=self.sync_fail_on_error
> > > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> > 480,
> > > in send_produce_request
> > > (not fail_on_error or not self._raise_on_response_error(resp))]
> > > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> > 247,
> > > in _raise_on_response_error
> > > raise resp
> > > kafka.common.FailedPayloadsError
> > >
> > > Here is what is in my logs
> > > [2015-07-12 03:29:58,103] INFO Closing socket connection to
> > > /xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497
> is
> > > not valid, it is larger than the maximum size of 104857600 bytes.
> > > (kafka.network.Processor)
> > >
> > >
> > >
> > > Server is 4 gigs of ram.
> > >
> > > I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in
> kafka-server-start.sh
> > >
> > > So.....why?
> > >
> >
>
>
>
> --
>
> Jiefu Gong
> University of California, Berkeley | Class of 2017
> B.A Computer Science | College of Letters and Sciences
>
> jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
>
Re: stunning error - Request of length 1550939497 is not valid, it is
larger than the maximum size of 104857600 bytes
Posted by JIEFU GONG <jg...@berkeley.edu>.
Got it, looks like I didn't understand the request process and am failing
to use AB properly. Thanks for the help everyone! I suspect you might be
running into a similar error, David.
On Tue, Jul 14, 2015 at 11:56 AM, Jay Kreps <ja...@confluent.io> wrote:
> This is almost certainly a client bug. Kafka's request format is size
> delimited messages in the form
> <4 byte size N><N byte payload>
> If the client sends a request with an invalid size or sends a partial
> request the server will see effectively random bytes from the next request
> as the size of the next message and generally reject the request (or fail
> to parse it).
>
> -Jay
>
> On Sat, Jul 11, 2015 at 9:08 PM, David Montgomery <
> davidmontgomery@gmail.com
> > wrote:
>
> > I cant send this soooo simple payload using python.
> >
> > topic: topic-test-development
> > payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
> >
> >
> > No handlers could be found for logger "kafka.conn"
> > Traceback (most recent call last):
> > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
> 81,
> > in <module>
> > test_send_data_to_realtimenode()
> > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
> 38,
> > in test_send_data_to_realtimenode
> > response = producer.send_messages(test_topic,test_payload)
> > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
> > line 54, in send_messages
> > topic, partition, *msg
> > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > line 349, in send_messages
> > return self._send_messages(topic, partition, *msg)
> > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > line 390, in _send_messages
> > fail_on_error=self.sync_fail_on_error
> > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> 480,
> > in send_produce_request
> > (not fail_on_error or not self._raise_on_response_error(resp))]
> > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> 247,
> > in _raise_on_response_error
> > raise resp
> > kafka.common.FailedPayloadsError
> >
> > Here is what is in my logs
> > [2015-07-12 03:29:58,103] INFO Closing socket connection to
> > /xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497 is
> > not valid, it is larger than the maximum size of 104857600 bytes.
> > (kafka.network.Processor)
> >
> >
> >
> > Server is 4 gigs of ram.
> >
> > I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in kafka-server-start.sh
> >
> > So.....why?
> >
>
--
Jiefu Gong
University of California, Berkeley | Class of 2017
B.A Computer Science | College of Letters and Sciences
jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
Re: stunning error - Request of length 1550939497 is not valid, it is
larger than the maximum size of 104857600 bytes
Posted by Jay Kreps <ja...@confluent.io>.
This is almost certainly a client bug. Kafka's request format is size
delimited messages in the form
<4 byte size N><N byte payload>
If the client sends a request with an invalid size or sends a partial
request the server will see effectively random bytes from the next request
as the size of the next message and generally reject the request (or fail
to parse it).
-Jay
On Sat, Jul 11, 2015 at 9:08 PM, David Montgomery <davidmontgomery@gmail.com
> wrote:
> I cant send this soooo simple payload using python.
>
> topic: topic-test-development
> payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
>
>
> No handlers could be found for logger "kafka.conn"
> Traceback (most recent call last):
> File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line 81,
> in <module>
> test_send_data_to_realtimenode()
> File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line 38,
> in test_send_data_to_realtimenode
> response = producer.send_messages(test_topic,test_payload)
> File "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
> line 54, in send_messages
> topic, partition, *msg
> File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> line 349, in send_messages
> return self._send_messages(topic, partition, *msg)
> File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> line 390, in _send_messages
> fail_on_error=self.sync_fail_on_error
> File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 480,
> in send_produce_request
> (not fail_on_error or not self._raise_on_response_error(resp))]
> File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 247,
> in _raise_on_response_error
> raise resp
> kafka.common.FailedPayloadsError
>
> Here is what is in my logs
> [2015-07-12 03:29:58,103] INFO Closing socket connection to
> /xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497 is
> not valid, it is larger than the maximum size of 104857600 bytes.
> (kafka.network.Processor)
>
>
>
> Server is 4 gigs of ram.
>
> I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in kafka-server-start.sh
>
> So.....why?
>
Re: stunning error - Request of length 1550939497 is not valid, it is
larger than the maximum size of 104857600 bytes
Posted by Gwen Shapira <gs...@cloudera.com>.
I am not familiar with Apache Bench. Can you share more details on
what you are doing?
On Tue, Jul 14, 2015 at 11:45 AM, JIEFU GONG <jg...@berkeley.edu> wrote:
> So I'm trying to make a request with a simple ASCII text file, but what's
> strange is even if I change files to send or the contents of the file I get
> the same error message, even specifically the number of bytes of the
> message which seems weird if I'm changing the content? Should I be using
> Avro with my file before I try to send a request?
>
> On Tue, Jul 14, 2015 at 11:29 AM, Todd Palino <tp...@gmail.com> wrote:
>
>> This is interesting. We have seen something similar internally at LinkedIn
>> with one particular topic (and Avro schema), and only once in a while.
>> We've seen it happen 2 or 3 times so far. We had chalked it up to bad
>> content in the message, figuring that the sender was doing something like
>> sending a long stream of a single character, in error, which was creating a
>> highly compressible message. Given these cases, I'm no longer certain
>> that's the case.
>>
>> Becket, you had been taking a look at this internally. Do you have any
>> thoughts on this?
>>
>> -Todd
>>
>>
>> On Tue, Jul 14, 2015 at 11:18 AM, JIEFU GONG <jg...@berkeley.edu> wrote:
>>
>> > @Gwen
>> > I am having a very very similar issue where I am attempting to send a
>> > rather small message and it's blowing up on me (my specific error is:
>> > Invalid receive (size = 1347375956 larger than 104857600)). I tried to
>> > change the relevant settings but it seems that this particular request is
>> > of 1340 mbs (and davids will be 1500 mb) and attempting to change the
>> > setting will give you another error saying there is not enough memory in
>> > the java heap. Any insight here?
>> >
>> > Specifically I am speculating the issue is indeed what Shayne has said
>> > about encoding: I am trying to use apachebench to send a post request to
>> a
>> > kafka server but it is returning the above error -- do I have to format
>> the
>> > data in any way as this might be the reason why I'm experience this
>> issue.
>> >
>> >
>> > On Sun, Jul 12, 2015 at 6:35 AM, Shayne S <sh...@gmail.com> wrote:
>> >
>> > > Your payload is so small that I suspect it's an encoding issue. Is your
>> > > producer set to expect a byte array and you're passing a string? Or
>> vice
>> > > versa?
>> > >
>> > > On Sat, Jul 11, 2015 at 11:08 PM, David Montgomery <
>> > > davidmontgomery@gmail.com> wrote:
>> > >
>> > > > I cant send this soooo simple payload using python.
>> > > >
>> > > > topic: topic-test-development
>> > > > payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
>> > > >
>> > > >
>> > > > No handlers could be found for logger "kafka.conn"
>> > > > Traceback (most recent call last):
>> > > > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py",
>> line
>> > > 81,
>> > > > in <module>
>> > > > test_send_data_to_realtimenode()
>> > > > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py",
>> line
>> > > 38,
>> > > > in test_send_data_to_realtimenode
>> > > > response = producer.send_messages(test_topic,test_payload)
>> > > > File
>> > "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
>> > > > line 54, in send_messages
>> > > > topic, partition, *msg
>> > > > File
>> "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
>> > > > line 349, in send_messages
>> > > > return self._send_messages(topic, partition, *msg)
>> > > > File
>> "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
>> > > > line 390, in _send_messages
>> > > > fail_on_error=self.sync_fail_on_error
>> > > > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
>> > > 480,
>> > > > in send_produce_request
>> > > > (not fail_on_error or not self._raise_on_response_error(resp))]
>> > > > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
>> > > 247,
>> > > > in _raise_on_response_error
>> > > > raise resp
>> > > > kafka.common.FailedPayloadsError
>> > > >
>> > > > Here is what is in my logs
>> > > > [2015-07-12 03:29:58,103] INFO Closing socket connection to
>> > > > /xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497
>> > is
>> > > > not valid, it is larger than the maximum size of 104857600 bytes.
>> > > > (kafka.network.Processor)
>> > > >
>> > > >
>> > > >
>> > > > Server is 4 gigs of ram.
>> > > >
>> > > > I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in
>> > kafka-server-start.sh
>> > > >
>> > > > So.....why?
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > Jiefu Gong
>> > University of California, Berkeley | Class of 2017
>> > B.A Computer Science | College of Letters and Sciences
>> >
>> > jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
>> >
>>
>
>
>
> --
>
> Jiefu Gong
> University of California, Berkeley | Class of 2017
> B.A Computer Science | College of Letters and Sciences
>
> jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
Re: stunning error - Request of length 1550939497 is not valid, it is
larger than the maximum size of 104857600 bytes
Posted by JIEFU GONG <jg...@berkeley.edu>.
So I'm trying to make a request with a simple ASCII text file, but what's
strange is even if I change files to send or the contents of the file I get
the same error message, even specifically the number of bytes of the
message which seems weird if I'm changing the content? Should I be using
Avro with my file before I try to send a request?
On Tue, Jul 14, 2015 at 11:29 AM, Todd Palino <tp...@gmail.com> wrote:
> This is interesting. We have seen something similar internally at LinkedIn
> with one particular topic (and Avro schema), and only once in a while.
> We've seen it happen 2 or 3 times so far. We had chalked it up to bad
> content in the message, figuring that the sender was doing something like
> sending a long stream of a single character, in error, which was creating a
> highly compressible message. Given these cases, I'm no longer certain
> that's the case.
>
> Becket, you had been taking a look at this internally. Do you have any
> thoughts on this?
>
> -Todd
>
>
> On Tue, Jul 14, 2015 at 11:18 AM, JIEFU GONG <jg...@berkeley.edu> wrote:
>
> > @Gwen
> > I am having a very very similar issue where I am attempting to send a
> > rather small message and it's blowing up on me (my specific error is:
> > Invalid receive (size = 1347375956 larger than 104857600)). I tried to
> > change the relevant settings but it seems that this particular request is
> > of 1340 mbs (and davids will be 1500 mb) and attempting to change the
> > setting will give you another error saying there is not enough memory in
> > the java heap. Any insight here?
> >
> > Specifically I am speculating the issue is indeed what Shayne has said
> > about encoding: I am trying to use apachebench to send a post request to
> a
> > kafka server but it is returning the above error -- do I have to format
> the
> > data in any way as this might be the reason why I'm experience this
> issue.
> >
> >
> > On Sun, Jul 12, 2015 at 6:35 AM, Shayne S <sh...@gmail.com> wrote:
> >
> > > Your payload is so small that I suspect it's an encoding issue. Is your
> > > producer set to expect a byte array and you're passing a string? Or
> vice
> > > versa?
> > >
> > > On Sat, Jul 11, 2015 at 11:08 PM, David Montgomery <
> > > davidmontgomery@gmail.com> wrote:
> > >
> > > > I cant send this soooo simple payload using python.
> > > >
> > > > topic: topic-test-development
> > > > payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
> > > >
> > > >
> > > > No handlers could be found for logger "kafka.conn"
> > > > Traceback (most recent call last):
> > > > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py",
> line
> > > 81,
> > > > in <module>
> > > > test_send_data_to_realtimenode()
> > > > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py",
> line
> > > 38,
> > > > in test_send_data_to_realtimenode
> > > > response = producer.send_messages(test_topic,test_payload)
> > > > File
> > "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
> > > > line 54, in send_messages
> > > > topic, partition, *msg
> > > > File
> "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > > > line 349, in send_messages
> > > > return self._send_messages(topic, partition, *msg)
> > > > File
> "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > > > line 390, in _send_messages
> > > > fail_on_error=self.sync_fail_on_error
> > > > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> > > 480,
> > > > in send_produce_request
> > > > (not fail_on_error or not self._raise_on_response_error(resp))]
> > > > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> > > 247,
> > > > in _raise_on_response_error
> > > > raise resp
> > > > kafka.common.FailedPayloadsError
> > > >
> > > > Here is what is in my logs
> > > > [2015-07-12 03:29:58,103] INFO Closing socket connection to
> > > > /xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497
> > is
> > > > not valid, it is larger than the maximum size of 104857600 bytes.
> > > > (kafka.network.Processor)
> > > >
> > > >
> > > >
> > > > Server is 4 gigs of ram.
> > > >
> > > > I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in
> > kafka-server-start.sh
> > > >
> > > > So.....why?
> > > >
> > >
> >
> >
> >
> > --
> >
> > Jiefu Gong
> > University of California, Berkeley | Class of 2017
> > B.A Computer Science | College of Letters and Sciences
> >
> > jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
> >
>
--
Jiefu Gong
University of California, Berkeley | Class of 2017
B.A Computer Science | College of Letters and Sciences
jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
Re: stunning error - Request of length 1550939497 is not valid, it is
larger than the maximum size of 104857600 bytes
Posted by Todd Palino <tp...@gmail.com>.
This is interesting. We have seen something similar internally at LinkedIn
with one particular topic (and Avro schema), and only once in a while.
We've seen it happen 2 or 3 times so far. We had chalked it up to bad
content in the message, figuring that the sender was doing something like
sending a long stream of a single character, in error, which was creating a
highly compressible message. Given these cases, I'm no longer certain
that's the case.
Becket, you had been taking a look at this internally. Do you have any
thoughts on this?
-Todd
On Tue, Jul 14, 2015 at 11:18 AM, JIEFU GONG <jg...@berkeley.edu> wrote:
> @Gwen
> I am having a very very similar issue where I am attempting to send a
> rather small message and it's blowing up on me (my specific error is:
> Invalid receive (size = 1347375956 larger than 104857600)). I tried to
> change the relevant settings but it seems that this particular request is
> of 1340 mbs (and davids will be 1500 mb) and attempting to change the
> setting will give you another error saying there is not enough memory in
> the java heap. Any insight here?
>
> Specifically I am speculating the issue is indeed what Shayne has said
> about encoding: I am trying to use apachebench to send a post request to a
> kafka server but it is returning the above error -- do I have to format the
> data in any way as this might be the reason why I'm experience this issue.
>
>
> On Sun, Jul 12, 2015 at 6:35 AM, Shayne S <sh...@gmail.com> wrote:
>
> > Your payload is so small that I suspect it's an encoding issue. Is your
> > producer set to expect a byte array and you're passing a string? Or vice
> > versa?
> >
> > On Sat, Jul 11, 2015 at 11:08 PM, David Montgomery <
> > davidmontgomery@gmail.com> wrote:
> >
> > > I cant send this soooo simple payload using python.
> > >
> > > topic: topic-test-development
> > > payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
> > >
> > >
> > > No handlers could be found for logger "kafka.conn"
> > > Traceback (most recent call last):
> > > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
> > 81,
> > > in <module>
> > > test_send_data_to_realtimenode()
> > > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
> > 38,
> > > in test_send_data_to_realtimenode
> > > response = producer.send_messages(test_topic,test_payload)
> > > File
> "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
> > > line 54, in send_messages
> > > topic, partition, *msg
> > > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > > line 349, in send_messages
> > > return self._send_messages(topic, partition, *msg)
> > > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > > line 390, in _send_messages
> > > fail_on_error=self.sync_fail_on_error
> > > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> > 480,
> > > in send_produce_request
> > > (not fail_on_error or not self._raise_on_response_error(resp))]
> > > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> > 247,
> > > in _raise_on_response_error
> > > raise resp
> > > kafka.common.FailedPayloadsError
> > >
> > > Here is what is in my logs
> > > [2015-07-12 03:29:58,103] INFO Closing socket connection to
> > > /xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497
> is
> > > not valid, it is larger than the maximum size of 104857600 bytes.
> > > (kafka.network.Processor)
> > >
> > >
> > >
> > > Server is 4 gigs of ram.
> > >
> > > I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in
> kafka-server-start.sh
> > >
> > > So.....why?
> > >
> >
>
>
>
> --
>
> Jiefu Gong
> University of California, Berkeley | Class of 2017
> B.A Computer Science | College of Letters and Sciences
>
> jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
>
Re: stunning error - Request of length 1550939497 is not valid, it is
larger than the maximum size of 104857600 bytes
Posted by JIEFU GONG <jg...@berkeley.edu>.
@Gwen
I am having a very very similar issue where I am attempting to send a
rather small message and it's blowing up on me (my specific error is:
Invalid receive (size = 1347375956 larger than 104857600)). I tried to
change the relevant settings but it seems that this particular request is
of 1340 mbs (and davids will be 1500 mb) and attempting to change the
setting will give you another error saying there is not enough memory in
the java heap. Any insight here?
Specifically I am speculating the issue is indeed what Shayne has said
about encoding: I am trying to use apachebench to send a post request to a
kafka server but it is returning the above error -- do I have to format the
data in any way as this might be the reason why I'm experience this issue.
On Sun, Jul 12, 2015 at 6:35 AM, Shayne S <sh...@gmail.com> wrote:
> Your payload is so small that I suspect it's an encoding issue. Is your
> producer set to expect a byte array and you're passing a string? Or vice
> versa?
>
> On Sat, Jul 11, 2015 at 11:08 PM, David Montgomery <
> davidmontgomery@gmail.com> wrote:
>
> > I cant send this soooo simple payload using python.
> >
> > topic: topic-test-development
> > payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
> >
> >
> > No handlers could be found for logger "kafka.conn"
> > Traceback (most recent call last):
> > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
> 81,
> > in <module>
> > test_send_data_to_realtimenode()
> > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
> 38,
> > in test_send_data_to_realtimenode
> > response = producer.send_messages(test_topic,test_payload)
> > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
> > line 54, in send_messages
> > topic, partition, *msg
> > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > line 349, in send_messages
> > return self._send_messages(topic, partition, *msg)
> > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > line 390, in _send_messages
> > fail_on_error=self.sync_fail_on_error
> > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> 480,
> > in send_produce_request
> > (not fail_on_error or not self._raise_on_response_error(resp))]
> > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> 247,
> > in _raise_on_response_error
> > raise resp
> > kafka.common.FailedPayloadsError
> >
> > Here is what is in my logs
> > [2015-07-12 03:29:58,103] INFO Closing socket connection to
> > /xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497 is
> > not valid, it is larger than the maximum size of 104857600 bytes.
> > (kafka.network.Processor)
> >
> >
> >
> > Server is 4 gigs of ram.
> >
> > I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in kafka-server-start.sh
> >
> > So.....why?
> >
>
--
Jiefu Gong
University of California, Berkeley | Class of 2017
B.A Computer Science | College of Letters and Sciences
jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
Re: stunning error - Request of length 1550939497 is not valid, it is
larger than the maximum size of 104857600 bytes
Posted by Shayne S <sh...@gmail.com>.
Your payload is so small that I suspect it's an encoding issue. Is your
producer set to expect a byte array and you're passing a string? Or vice
versa?
On Sat, Jul 11, 2015 at 11:08 PM, David Montgomery <
davidmontgomery@gmail.com> wrote:
> I cant send this soooo simple payload using python.
>
> topic: topic-test-development
> payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
>
>
> No handlers could be found for logger "kafka.conn"
> Traceback (most recent call last):
> File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line 81,
> in <module>
> test_send_data_to_realtimenode()
> File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line 38,
> in test_send_data_to_realtimenode
> response = producer.send_messages(test_topic,test_payload)
> File "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
> line 54, in send_messages
> topic, partition, *msg
> File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> line 349, in send_messages
> return self._send_messages(topic, partition, *msg)
> File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> line 390, in _send_messages
> fail_on_error=self.sync_fail_on_error
> File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 480,
> in send_produce_request
> (not fail_on_error or not self._raise_on_response_error(resp))]
> File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 247,
> in _raise_on_response_error
> raise resp
> kafka.common.FailedPayloadsError
>
> Here is what is in my logs
> [2015-07-12 03:29:58,103] INFO Closing socket connection to
> /xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497 is
> not valid, it is larger than the maximum size of 104857600 bytes.
> (kafka.network.Processor)
>
>
>
> Server is 4 gigs of ram.
>
> I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in kafka-server-start.sh
>
> So.....why?
>
Re: stunning error - Request of length 1550939497 is not valid, it is
larger than the maximum size of 104857600 bytes
Posted by Gwen Shapira <gs...@cloudera.com>.
Did you try setting message.max.bytes and replica.fetch.max.bytes to
values larger than the message you are trying to send?
>From the error message, they should be at least 1550939497.
On Sat, Jul 11, 2015 at 10:14 PM, David Montgomery
<da...@gmail.com> wrote:
> Hi
>
>
> Below is my server.properties
>
> I am not having an issue with consuming from my kafka broker. I have
> having an issue writing to my broker. One send bombs.
>
>
> # limitations under the License.
> # see kafka.server.KafkaConfig for additional details and defaults
>
> ############################# Server Basics #############################
>
> # The id of the broker. This must be set to a unique integer for each
> broker.
> broker.id=<%=@broker_id%>
>
> ############################# Socket Server Settings
> #############################
>
> # The port the socket server listens on
> port=9092
>
> # Hostname the broker will bind to and advertise to producers and consumers.
> # If not set, the server will bind to all interfaces and advertise the
> value returned from
> # from java.net.InetAddress.getCanonicalHostName().
> host.name=<%=@ipaddress%>
>
> # The number of threads handling network requests
> num.network.threads=2
>
> # The number of threads doing disk I/O
> num.io.threads=2
>
> # The send buffer (SO_SNDBUF) used by the socket server
> socket.send.buffer.bytes=1048576
>
> # The receive buffer (SO_RCVBUF) used by the socket server
> socket.receive.buffer.bytes=1048576
>
> # The maximum size of a request that the socket server will accept
> (protection against OOM)
> socket.request.max.bytes=104857600
>
>
> ############################# Log Basics #############################
>
> # A comma seperated list of directories under which to store log files
> log.dirs=/tmp/kafka-logs
>
>
>
> ############################# Log Flush Policy #############################
>
> # The following configurations control the flush of data to disk. This is
> among the most
> # important performance knob in kafka.
> # There are a few important trade-offs here:
> # 1. Durability: Unflushed data may be lost if you are not using
> replication.
> # 2. Latency: Very large flush intervals may lead to latency spikes when
> the flush does occur as there will be a lot of data to flush.
> # 3. Throughput: The flush is generally the most expensive operation,
> and a small flush interval may lead to exceessive seeks.
> # The settings below allow one to configure the flush policy to flush data
> after a period of time or
> # every N messages (or both). This can be done globally and overridden on a
> per-topic basis.
>
> # The number of messages to accept before forcing a flush of data to disk
> log.flush.interval.messages=10000
>
> # The maximum amount of time a message can sit in a log before we force a
> flush
> log.flush.interval.ms=1000
>
> # Per-topic overrides for log.flush.interval.ms
> #log.flush.intervals.ms.per.topic=topic1:1000, topic2:3000
>
> ############################# Log Retention Policy
> #############################
>
> # The following configurations control the disposal of log segments. The
> policy can
> # be set to delete segments after a period of time, or after a given size
> has accumulated.
> # A segment will be deleted whenever *either* of these criteria are met.
> Deletion always happens
> # from the end of the log.
>
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=168
>
> # A size-based retention policy for logs. Segments are pruned from the log
> as long as the remaining
> # segments don't drop below log.retention.bytes.
> #log.retention.bytes=1073741824
>
> # The maximum size of a log segment file. When this size is reached a new
> log segment will be created.
> log.segment.bytes=536870912
>
> # The interval at which log segments are checked to see if they can be
> deleted according
> # to the retention policies
> log.cleanup.interval.mins=1
>
> ############################# Zookeeper #############################
>
> # Zookeeper connection string (see zookeeper docs for details).
> # This is a comma separated host:port pairs, each corresponding to a zk
> # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
> # You can also append an optional chroot string to the urls to specify the
> # root directory for all kafka znodes.
> #zookeeper.connect=localhost:2181
> zookeeper.connect=<%=@zookeeper%>
>
>
> # Timeout in ms for connecting to zookeeper
> zookeeper.connection.timeout.ms=1000000
>
>
> # The number of logical partitions per topic per server. More partitions
> allow greater parallelism
> # for consumption, but also mean more files.
> num.partitions=<%=@paritions%>
>
>
>
>
>
>
>
>
>
> On Sun, Jul 12, 2015 at 12:21 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
>> You need to configure the Kafka broker to allow you to send larger
>> messages.
>> The relevant parameters are:
>>
>> message.max.bytes (default:1000000) – Maximum size of a message the
>> broker will accept. This has to be smaller than the consumer
>> fetch.message.max.bytes, or the broker will have messages that can’t
>> be consumed, causing consumers to hang.
>> replica.fetch.max.bytes (default: 1MB) – Maximum size of data that a
>> broker can replicate. This has to be larger than message.max.bytes, or
>> a broker will accept messages and fail to replicate them. Leading to
>> potential data loss.
>>
>> Gwen
>>
>> On Sat, Jul 11, 2015 at 9:08 PM, David Montgomery
>> <da...@gmail.com> wrote:
>> > I cant send this soooo simple payload using python.
>> >
>> > topic: topic-test-development
>> > payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
>> >
>> >
>> > No handlers could be found for logger "kafka.conn"
>> > Traceback (most recent call last):
>> > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
>> 81,
>> > in <module>
>> > test_send_data_to_realtimenode()
>> > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
>> 38,
>> > in test_send_data_to_realtimenode
>> > response = producer.send_messages(test_topic,test_payload)
>> > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
>> > line 54, in send_messages
>> > topic, partition, *msg
>> > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
>> > line 349, in send_messages
>> > return self._send_messages(topic, partition, *msg)
>> > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
>> > line 390, in _send_messages
>> > fail_on_error=self.sync_fail_on_error
>> > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
>> 480,
>> > in send_produce_request
>> > (not fail_on_error or not self._raise_on_response_error(resp))]
>> > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
>> 247,
>> > in _raise_on_response_error
>> > raise resp
>> > kafka.common.FailedPayloadsError
>> >
>> > Here is what is in my logs
>> > [2015-07-12 03:29:58,103] INFO Closing socket connection to
>> > /xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497 is
>> > not valid, it is larger than the maximum size of 104857600 bytes.
>> > (kafka.network.Processor)
>> >
>> >
>> >
>> > Server is 4 gigs of ram.
>> >
>> > I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in kafka-server-start.sh
>> >
>> > So.....why?
>>
Re: stunning error - Request of length 1550939497 is not valid, it is
larger than the maximum size of 104857600 bytes
Posted by David Montgomery <da...@gmail.com>.
Hi
Below is my server.properties
I am not having an issue with consuming from my kafka broker. I have
having an issue writing to my broker. One send bombs.
# limitations under the License.
# see kafka.server.KafkaConfig for additional details and defaults
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each
broker.
broker.id=<%=@broker_id%>
############################# Socket Server Settings
#############################
# The port the socket server listens on
port=9092
# Hostname the broker will bind to and advertise to producers and consumers.
# If not set, the server will bind to all interfaces and advertise the
value returned from
# from java.net.InetAddress.getCanonicalHostName().
host.name=<%=@ipaddress%>
# The number of threads handling network requests
num.network.threads=2
# The number of threads doing disk I/O
num.io.threads=2
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=1048576
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=1048576
# The maximum size of a request that the socket server will accept
(protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs
############################# Log Flush Policy #############################
# The following configurations control the flush of data to disk. This is
among the most
# important performance knob in kafka.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data may be lost if you are not using
replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when
the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation,
and a small flush interval may lead to exceessive seeks.
# The settings below allow one to configure the flush policy to flush data
after a period of time or
# every N messages (or both). This can be done globally and overridden on a
per-topic basis.
# The number of messages to accept before forcing a flush of data to disk
log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a
flush
log.flush.interval.ms=1000
# Per-topic overrides for log.flush.interval.ms
#log.flush.intervals.ms.per.topic=topic1:1000, topic2:3000
############################# Log Retention Policy
#############################
# The following configurations control the disposal of log segments. The
policy can
# be set to delete segments after a period of time, or after a given size
has accumulated.
# A segment will be deleted whenever *either* of these criteria are met.
Deletion always happens
# from the end of the log.
# The minimum age of a log file to be eligible for deletion
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log
as long as the remaining
# segments don't drop below log.retention.bytes.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new
log segment will be created.
log.segment.bytes=536870912
# The interval at which log segments are checked to see if they can be
deleted according
# to the retention policies
log.cleanup.interval.mins=1
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
#zookeeper.connect=localhost:2181
zookeeper.connect=<%=@zookeeper%>
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=1000000
# The number of logical partitions per topic per server. More partitions
allow greater parallelism
# for consumption, but also mean more files.
num.partitions=<%=@paritions%>
On Sun, Jul 12, 2015 at 12:21 PM, Gwen Shapira <gs...@cloudera.com>
wrote:
> You need to configure the Kafka broker to allow you to send larger
> messages.
> The relevant parameters are:
>
> message.max.bytes (default:1000000) – Maximum size of a message the
> broker will accept. This has to be smaller than the consumer
> fetch.message.max.bytes, or the broker will have messages that can’t
> be consumed, causing consumers to hang.
> replica.fetch.max.bytes (default: 1MB) – Maximum size of data that a
> broker can replicate. This has to be larger than message.max.bytes, or
> a broker will accept messages and fail to replicate them. Leading to
> potential data loss.
>
> Gwen
>
> On Sat, Jul 11, 2015 at 9:08 PM, David Montgomery
> <da...@gmail.com> wrote:
> > I cant send this soooo simple payload using python.
> >
> > topic: topic-test-development
> > payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
> >
> >
> > No handlers could be found for logger "kafka.conn"
> > Traceback (most recent call last):
> > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
> 81,
> > in <module>
> > test_send_data_to_realtimenode()
> > File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
> 38,
> > in test_send_data_to_realtimenode
> > response = producer.send_messages(test_topic,test_payload)
> > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
> > line 54, in send_messages
> > topic, partition, *msg
> > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > line 349, in send_messages
> > return self._send_messages(topic, partition, *msg)
> > File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > line 390, in _send_messages
> > fail_on_error=self.sync_fail_on_error
> > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> 480,
> > in send_produce_request
> > (not fail_on_error or not self._raise_on_response_error(resp))]
> > File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> 247,
> > in _raise_on_response_error
> > raise resp
> > kafka.common.FailedPayloadsError
> >
> > Here is what is in my logs
> > [2015-07-12 03:29:58,103] INFO Closing socket connection to
> > /xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497 is
> > not valid, it is larger than the maximum size of 104857600 bytes.
> > (kafka.network.Processor)
> >
> >
> >
> > Server is 4 gigs of ram.
> >
> > I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in kafka-server-start.sh
> >
> > So.....why?
>
Re: stunning error - Request of length 1550939497 is not valid, it is
larger than the maximum size of 104857600 bytes
Posted by Gwen Shapira <gs...@cloudera.com>.
You need to configure the Kafka broker to allow you to send larger messages.
The relevant parameters are:
message.max.bytes (default:1000000) – Maximum size of a message the
broker will accept. This has to be smaller than the consumer
fetch.message.max.bytes, or the broker will have messages that can’t
be consumed, causing consumers to hang.
replica.fetch.max.bytes (default: 1MB) – Maximum size of data that a
broker can replicate. This has to be larger than message.max.bytes, or
a broker will accept messages and fail to replicate them. Leading to
potential data loss.
Gwen
On Sat, Jul 11, 2015 at 9:08 PM, David Montgomery
<da...@gmail.com> wrote:
> I cant send this soooo simple payload using python.
>
> topic: topic-test-development
> payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
>
>
> No handlers could be found for logger "kafka.conn"
> Traceback (most recent call last):
> File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line 81,
> in <module>
> test_send_data_to_realtimenode()
> File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line 38,
> in test_send_data_to_realtimenode
> response = producer.send_messages(test_topic,test_payload)
> File "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
> line 54, in send_messages
> topic, partition, *msg
> File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> line 349, in send_messages
> return self._send_messages(topic, partition, *msg)
> File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> line 390, in _send_messages
> fail_on_error=self.sync_fail_on_error
> File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 480,
> in send_produce_request
> (not fail_on_error or not self._raise_on_response_error(resp))]
> File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 247,
> in _raise_on_response_error
> raise resp
> kafka.common.FailedPayloadsError
>
> Here is what is in my logs
> [2015-07-12 03:29:58,103] INFO Closing socket connection to
> /xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497 is
> not valid, it is larger than the maximum size of 104857600 bytes.
> (kafka.network.Processor)
>
>
>
> Server is 4 gigs of ram.
>
> I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in kafka-server-start.sh
>
> So.....why?