You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jay Kreps (Created) (JIRA)" <ji...@apache.org> on 2011/10/26 19:53:32 UTC
[jira] [Created] (KAFKA-171) Kafka producer should do a single
write to send message sets
Kafka producer should do a single write to send message sets
------------------------------------------------------------
Key: KAFKA-171
URL: https://issues.apache.org/jira/browse/KAFKA-171
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.7, 0.8
Reporter: Jay Kreps
Assignee: Jay Kreps
Fix For: 0.8
>From email thread:
http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
I think this is because in BoundedByteBufferSend.scala we do essentially
channel.write(sizeBuffer)
channel.write(dataBuffer)
The correct solution is to use vector I/O and instead do
channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141272#comment-13141272 ]
Jun Rao commented on KAFKA-171:
-------------------------------
MessageSet has a couple of unused imports. Other than that, the patch looks good.
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch, KAFKA-171-v2.patch, KAFKA-171.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Jay Kreps (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Kreps updated KAFKA-171:
----------------------------
Attachment: KAFKA-171.patch
Okay this patch completes the conversion to GatheringByteChannel. I recommend we take this even though there doesn't seem to be real perf difference just because the network profile is better.
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch, KAFKA-171.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Jay Kreps (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136223#comment-13136223 ]
Jay Kreps commented on KAFKA-171:
---------------------------------
Moving off localhost between my mac laptop and dev workstation (linux) I see similar results:
TRUNK:
jkreps-mn:kafka-git jkreps$ bin/kafka-producer-perf-test.sh --topic test --brokerinfo zk.connect=jkreps-ld:2181 --messages 500000 --message-size 10 --batch-size 1 --threads 1
[2011-10-26 11:59:51,795] INFO Total Num Messages: 500000 bytes: 5000000 in 13.046 secs (kafka.tools.ProducerPerformance$)
[2011-10-26 11:59:51,795] INFO Messages/sec: 38325.9237 (kafka.tools.ProducerPerformance$)
[2011-10-26 11:59:51,795] INFO MB/sec: 0.3655 (kafka.tools.ProducerPerformance$)
PATCHED:
jkreps-mn:kafka-git jkreps$ bin/kafka-producer-perf-test.sh --topic test --brokerinfo zk.connect=jkreps-ld:2181 --messages 500000 --message-size 10 --batch-size 1 --threads 1
[2011-10-26 11:58:42,335] INFO Total Num Messages: 500000 bytes: 5000000 in 13.125 secs (kafka.tools.ProducerPerformance$)
[2011-10-26 11:58:42,335] INFO Messages/sec: 38095.2381 (kafka.tools.ProducerPerformance$)
[2011-10-26 11:58:42,335] INFO MB/sec: 0.3633 (kafka.tools.ProducerPerformance$)
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Jay Kreps (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Kreps resolved KAFKA-171.
-----------------------------
Resolution: Fixed
Committed.
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch, KAFKA-171-v2.patch, KAFKA-171.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Jay Kreps (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136193#comment-13136193 ]
Jay Kreps commented on KAFKA-171:
---------------------------------
Attached is a draft patch which turns the request into a single write. This is just a draft if this actually improves performance we should change Receive to use ScatteringByteChannel for consistency and also clean up a few more files with the same trick.
On my mac laptop I do see a change in tcpdump which seems to eliminate the 4 byte send. However I don't see any positive result in performance for synchronous single-threaded sends of 10 byte messages (which should be the worst case for this). I think this may just be because I am testing over localhost.
Here are the details on the results I have:
TRUNK:
jkreps-mn:kafka-git jkreps$ sudo tcpdump -i lo0 port 9093
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo0, link-type NULL (BSD loopback), capture size 96 bytes
10:32:30.128938 IP jkreps-mn.linkedin.biz.56953 > jkreps-mn.linkedin.biz.9093: S 323648854:323648854(0) win 65535 <mss 16344,nop,wscale 3,nop,nop,timestamp 377871870 0,sackOK,eol>
10:32:30.129004 IP jkreps-mn.linkedin.biz.9093 > jkreps-mn.linkedin.biz.56953: S 526915069:526915069(0) ack 323648855 win 65535 <mss 16344,nop,wscale 3,nop,nop,timestamp 377871870 377871870,sackOK,eol>
10:32:30.129013 IP jkreps-mn.linkedin.biz.56953 > jkreps-mn.linkedin.biz.9093: . ack 1 win 65535 <nop,nop,timestamp 377871870 377871870>
10:32:30.129022 IP jkreps-mn.linkedin.biz.9093 > jkreps-mn.linkedin.biz.56953: . ack 1 win 65535 <nop,nop,timestamp 377871870 377871870>
10:32:30.129306 IP jkreps-mn.linkedin.biz.56953 > jkreps-mn.linkedin.biz.9093: P 1:5(4) ack 1 win 65535 <nop,nop,timestamp 377871870 377871870>
10:32:30.129319 IP jkreps-mn.linkedin.biz.9093 > jkreps-mn.linkedin.biz.56953: . ack 5 win 65535 <nop,nop,timestamp 377871870 377871870>
10:32:30.129339 IP jkreps-mn.linkedin.biz.56953 > jkreps-mn.linkedin.biz.9093: P 5:41(36) ack 1 win 65535 <nop,nop,timestamp 377871870 377871870>
10:32:30.129350 IP jkreps-mn.linkedin.biz.9093 > jkreps-mn.linkedin.biz.56953: . ack 41 win 65535 <nop,nop,timestamp 377871870 377871870>
10:32:30.151892 IP jkreps-mn.linkedin.biz.56953 > jkreps-mn.linkedin.biz.9093: F 41:41(0) ack 1 win 65535 <nop,nop,timestamp 377871870 377871870>
10:32:30.151938 IP jkreps-mn.linkedin.biz.9093 > jkreps-mn.linkedin.biz.56953: . ack 42 win 65535 <nop,nop,timestamp 377871870 377871870>
10:32:30.151946 IP jkreps-mn.linkedin.biz.56953 > jkreps-mn.linkedin.biz.9093: . ack 1 win 65535 <nop,nop,timestamp 377871870 377871870>
10:32:30.152554 IP jkreps-mn.linkedin.biz.9093 > jkreps-mn.linkedin.biz.56953: F 1:1(0) ack 42 win 65535 <nop,nop,timestamp 377871870 377871870>
10:32:30.152571 IP jkreps-mn.linkedin.biz.56953 > jkreps-mn.linkedin.biz.9093: . ack 2 win 65535 <nop,nop,timestamp 377871870 377871870>
PATCHED:
jkreps-mn:kafka-git jkreps$ sudo tcpdump -i lo0 port 9093
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo0, link-type NULL (BSD loopback), capture size 96 bytes
10:35:40.637220 IP jkreps-mn.linkedin.biz.56993 > jkreps-mn.linkedin.biz.9093: S 1456363353:1456363353(0) win 65535 <mss 16344,nop,wscale 3,nop,nop,timestamp 377873772 0,sackOK,eol>
10:35:40.637287 IP jkreps-mn.linkedin.biz.9093 > jkreps-mn.linkedin.biz.56993: S 1260172914:1260172914(0) ack 1456363354 win 65535 <mss 16344,nop,wscale 3,nop,nop,timestamp 377873772 377873772,sackOK,eol>
10:35:40.637296 IP jkreps-mn.linkedin.biz.56993 > jkreps-mn.linkedin.biz.9093: . ack 1 win 65535 <nop,nop,timestamp 377873772 377873772>
10:35:40.637306 IP jkreps-mn.linkedin.biz.9093 > jkreps-mn.linkedin.biz.56993: . ack 1 win 65535 <nop,nop,timestamp 377873772 377873772>
10:35:40.657848 IP jkreps-mn.linkedin.biz.56993 > jkreps-mn.linkedin.biz.9093: P 1:41(40) ack 1 win 65535 <nop,nop,timestamp 377873773 377873772>
10:35:40.657886 IP jkreps-mn.linkedin.biz.9093 > jkreps-mn.linkedin.biz.56993: . ack 41 win 65535 <nop,nop,timestamp 377873773 377873773>
10:35:40.711399 IP jkreps-mn.linkedin.biz.56993 > jkreps-mn.linkedin.biz.9093: F 41:41(0) ack 1 win 65535 <nop,nop,timestamp 377873773 377873773>
10:35:40.711430 IP jkreps-mn.linkedin.biz.9093 > jkreps-mn.linkedin.biz.56993: . ack 42 win 65535 <nop,nop,timestamp 377873773 377873773>
10:35:40.711437 IP jkreps-mn.linkedin.biz.56993 > jkreps-mn.linkedin.biz.9093: . ack 1 win 65535 <nop,nop,timestamp 377873773 377873773>
10:35:40.762640 IP jkreps-mn.linkedin.biz.9093 > jkreps-mn.linkedin.biz.56993: F 1:1(0) ack 42 win 65535 <nop,nop,timestamp 377873774 377873773>
10:35:40.762678 IP jkreps-mn.linkedin.biz.56993 > jkreps-mn.linkedin.biz.9093: . ack 2 win 65535 <nop,nop,timestamp 377873774 377873774>
TRUNK:
bin/kafka-producer-perf-test.sh --topic test --brokerinfo zk.connect=localhost:2181 --messages 300000 --message-size 10 --batch-size 1 --threads 1
...
[2011-10-26 10:33:58,458] INFO Total Num Messages: 300000 bytes: 3000000 in 13.636 secs (kafka.tools.ProducerPerformance$)
[2011-10-26 10:33:58,459] INFO Messages/sec: 22000.5867 (kafka.tools.ProducerPerformance$)
[2011-10-26 10:33:58,459] INFO MB/sec: 0.2098 (kafka.tools.ProducerPerformance$)
PATCHED:
jkreps-mn:kafka-git jkreps$ bin/kafka-producer-perf-test.sh --topic test --brokerinfo zk.connect=localhost:2181 --messages 300000 --message-size 10 --batch-size 1 --threads 1
...
[2011-10-26 10:38:03,965] INFO Total Num Messages: 300000 bytes: 3000000 in 13.254 secs (kafka.tools.ProducerPerformance$)
[2011-10-26 10:38:03,965] INFO Messages/sec: 22634.6763 (kafka.tools.ProducerPerformance$)
[2011-10-26 10:38:03,966] INFO MB/sec: 0.2159 (kafka.tools.ProducerPerformance$)
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Neha Narkhede (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141374#comment-13141374 ]
Neha Narkhede commented on KAFKA-171:
-------------------------------------
You can check it into trunk. 0.7 is going off its own branch
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch, KAFKA-171-v2.patch, KAFKA-171.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Jay Kreps (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141295#comment-13141295 ]
Jay Kreps commented on KAFKA-171:
---------------------------------
Cool, will clean up imports before checking in. I am going to hold off on this until after 0.7 goes out.
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch, KAFKA-171-v2.patch, KAFKA-171.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Jay Kreps (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136662#comment-13136662 ]
Jay Kreps commented on KAFKA-171:
---------------------------------
Yes, I think we should do it. My concern was just that I might be misunderstanding tcpdump or something since I find this a little counter-intuitive..
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Jay Kreps (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Kreps updated KAFKA-171:
----------------------------
Attachment: KAFKA-171-draft.patch
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Neha Narkhede (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140857#comment-13140857 ]
Neha Narkhede commented on KAFKA-171:
-------------------------------------
Since we are changing WritableByteChannel to GatheringByteChannel, would it be better to change the return type of writeTo and writeCompletely to return long, instead of int. This will avoid the coercion to Int in BoundedByteBufferSend.scala.
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch, KAFKA-171.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Neha Narkhede (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136201#comment-13136201 ]
Neha Narkhede commented on KAFKA-171:
-------------------------------------
This is a good change to make. A couple of comments -
1. Since we are changing WritableByteChannel to GatheringByteChannel, it is better to change the return type of writeTo and writeCompletely to return long, instead of int. This will avoid the coercion to Int in BoundedByteBufferSend.scala.
2. There are a couple of other places, where we do these double writes, e.g. OffsetArraySend, MessageSetSend etc. We might as well fix those ?
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Chris Burroughs (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136622#comment-13136622 ]
Chris Burroughs commented on KAFKA-171:
---------------------------------------
Even if this doesn't measurably improve node to node performance (and I'm not sure we should expect it to since we don't have to wait for an ACK to send the next packet), isn't it definitely making life better for network engineer?
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Jay Kreps (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Kreps updated KAFKA-171:
----------------------------
Attachment: KAFKA-171-v2.patch
I actually don't think we should make the writeTo method return a long since we use 4 byte ints in the protocol to delimit size of request so we really can't take a buffer > Int.MaxValue. I added a check on this to avoid an overflow if the size is > Int.MaxValue - 4, which is unlikely but possible.
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch, KAFKA-171-v2.patch, KAFKA-171.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-171) Kafka producer should do a single
write to send message sets
Posted by "Neha Narkhede (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140935#comment-13140935 ]
Neha Narkhede commented on KAFKA-171:
-------------------------------------
nnarkhed-mn:kafka-171 nnarkhed$ find . -name "*scala" -exec grep -Hi "asInstanceOf\[Int\]" {} \;
./core/src/main/scala/kafka/api/OffsetRequest.scala: header.putInt(size.asInstanceOf[Int] + 2)
./core/src/main/scala/kafka/api/ProducerRequest.scala: def sizeInBytes(): Int = 2 + topic.length + 4 + 4 + messages.sizeInBytes.asInstanceOf[Int]
./core/src/main/scala/kafka/network/BoundedByteBufferSend.scala: written.asInstanceOf[Int]
./core/src/main/scala/kafka/producer/SyncProducer.scala: val setSize = messages.sizeInBytes.asInstanceOf[Int]
./core/src/main/scala/kafka/server/MessageSetSend.scala: header.putInt(size.asInstanceOf[Int] + 2)
./core/src/main/scala/kafka/server/MessageSetSend.scala: written += fileBytesSent.asInstanceOf[Int]
./core/src/main/scala/kafka/server/MessageSetSend.scala: def sendSize: Int = size.asInstanceOf[Int] + header.capacity
./core/src/main/scala/kafka/utils/Utils.scala: buffer.putInt((value & 0xffffffffL).asInstanceOf[Int])
./core/src/main/scala/kafka/utils/Utils.scala: buffer.putInt(index, (value & 0xffffffffL).asInstanceOf[Int])
Its not great that we have so many places where we need to worry about coercion, but we can clean this up the next time we change the on wire protocol.
+1 on the latest patch
> Kafka producer should do a single write to send message sets
> ------------------------------------------------------------
>
> Key: KAFKA-171
> URL: https://issues.apache.org/jira/browse/KAFKA-171
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Fix For: 0.8
>
> Attachments: KAFKA-171-draft.patch, KAFKA-171-v2.patch, KAFKA-171.patch
>
>
> From email thread:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201110.mbox/%3cCAFbh0Q1PYUj32thBaYQ29E6J4wT_mrG5SuUsfdeGWj6rmEx9Gw@mail.gmail.com%3e
> > Before sending an actual message, kafka producer do send a (control) message of 4 bytes to the server. Kafka producer always does this action before send some message to the server.
> I think this is because in BoundedByteBufferSend.scala we do essentially
> channel.write(sizeBuffer)
> channel.write(dataBuffer)
> The correct solution is to use vector I/O and instead do
> channel.write(Array(sizeBuffer, dataBuffer))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira