You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2007/08/28 01:33:30 UTC
[jira] Created: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Increase the buffer size of pipes from 1k to 128k
-------------------------------------------------
Key: HADOOP-1788
URL: https://issues.apache.org/jira/browse/HADOOP-1788
Project: Hadoop
Issue Type: Bug
Components: pipes
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-1788:
---------------------------------------------
Status: Patch Available (was: Open)
added patch to change the socket buffer size.
And verified the changed values.
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1788:
----------------------------------
Status: Open (was: Patch Available)
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788-1.txt, patch-1788-2.txt, patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534732 ]
Christian Kunz commented on HADOOP-1788:
----------------------------------------
By using setvbuf on the c++ side and BufferedOutputStream / BufferedInputStream on the java side, I was able to change the data transfer to larger chunks, especially important when submitting many key-value pairs with small values.
strace shows now larger reads and writes.
I attach patch-1788-1.txt
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Kunz updated HADOOP-1788:
-----------------------------------
Attachment: patch-1788-2.txt
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788-1.txt, patch-1788-2.txt, patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535523 ]
Hudson commented on HADOOP-1788:
--------------------------------
Integrated in Hadoop-Nightly #275 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/275/])
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Affects Versions: 0.14.0
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: patch-1788-1.txt, patch-1788-2.txt, patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-1788) Increase the buffer size
of pipes from 1k to 128k
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534843 ]
amareshwari edited comment on HADOOP-1788 at 10/15/07 5:48 AM:
----------------------------------------------------------------------------
Thanks Christian.
I ran my 500KB write example with this patch and now, there is 128KB write followed by the rest.
mmap2(NULL, 507904, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7ec4000
write(3, "oooooooooooooooooooooooooooooooo"..., 131072) = 131072
write(3, "oooooooooooooooooooooooooooooooo"..., 393216) = 393216
munmap(0xb7ec4000, 507904) = 0
Patch is fine, except some clarifications
1. 128*1028 has to be replaced by 128*1024.
2. do we need the change in procedure progress() ?
3. do we require setsockopt change now?
4. should we do setvbuf for the case of 'hadoop.pipes.command.file' environment also?
was (Author: amareshwari):
Thanks Christian.
I ran my 500KB write example with this patch and now, there is 128KB write followed by the rest.
mmap2(NULL, 507904, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7ec4000
write(3, "oooooooooooooooooooooooooooooooo"..., 131072) = 131072
write(3, "oooooooooooooooooooooooooooooooo"..., 393216) = 393216
munmap(0xb7ec4000, 507904) = 0
Patch is fine, except some clarifications
1. 128*1028 has to be replaced by 128*1024.
2. do we need the change in procedure progress() ?
4. do we require setsockopt change now?
3. should we do setvbuf for the case of 'hadoop.pipes.command.file' environment also?
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788-1.txt, patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Kunz updated HADOOP-1788:
-----------------------------------
Attachment: patch-1788-1.txt
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788-1.txt, patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535070 ]
Amareshwari Sri Ramadasu commented on HADOOP-1788:
--------------------------------------------------
+1
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788-1.txt, patch-1788-2.txt, patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1788:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Amareshwari and Christian!
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Affects Versions: 0.14.0
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: patch-1788-1.txt, patch-1788-2.txt, patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534843 ]
Amareshwari Sri Ramadasu commented on HADOOP-1788:
--------------------------------------------------
Thanks Christian.
I ran my 500KB write example with this patch and now, there is 128KB write followed by the rest.
mmap2(NULL, 507904, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7ec4000
write(3, "oooooooooooooooooooooooooooooooo"..., 131072) = 131072
write(3, "oooooooooooooooooooooooooooooooo"..., 393216) = 393216
munmap(0xb7ec4000, 507904) = 0
Patch is fine, except some clarifications
1. 128*1028 has to be replaced by 128*1024.
2. do we need the change in procedure progress() ?
4. do we require setsockopt change now?
3. should we do setvbuf for the case of 'hadoop.pipes.command.file' environment also?
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788-1.txt, patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532646 ]
Hadoop QA commented on HADOOP-1788:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12367138/patch-1788.txt
against trunk revision r582165.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/892/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/892/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/892/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/892/console
This message is automatically generated.
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534878 ]
Christian Kunz commented on HADOOP-1788:
----------------------------------------
to 1: silly mistake: thank you for pointing it out
to 2: first it seemed to me an overkill to send progress upstream every second (especially because it is flushed) but unless it is coordinated with the configurable 'mapred.task.timeout' it might be not enough to send it just once per minute. Also it hardly makes a big difference in performance. For now I will take it out.
to 3: I did not test it without the setsockopt, but I doubt it makes a difference -- I will take it out.
to 4: as far as I know, 'hadoop.pipes.command.file' is only for debugging, so I did not bother to do anything about it.
I upload a patch reflecting the changes due to 1-3.
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788-1.txt, patch-1788-2.txt, patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das reassigned HADOOP-1788:
-----------------------------------
Assignee: Amareshwari Sri Ramadasu (was: Owen O'Malley)
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1788:
----------------------------------
Status: Patch Available (was: Open)
submit new version of patch to qa review
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788-1.txt, patch-1788-2.txt, patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1788:
----------------------------------
Fix Version/s: 0.15.0
Priority: Blocker (was: Major)
Affects Version/s: 0.14.0
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Affects Versions: 0.14.0
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: patch-1788-1.txt, patch-1788-2.txt, patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534880 ]
Christian Kunz commented on HADOOP-1788:
----------------------------------------
Forgot to mention the name of the patch: patch-1788-2.txt
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788-1.txt, patch-1788-2.txt, patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535005 ]
Hadoop QA commented on HADOOP-1788:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12367757/patch-1788-2.txt
against trunk revision r584919.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/948/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/948/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/948/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/948/console
This message is automatically generated.
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788-1.txt, patch-1788-2.txt, patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534012 ]
Amareshwari Sri Ramadasu commented on HADOOP-1788:
--------------------------------------------------
I also observed that there is no change in writes and reads with the patch.
It seems like socket buffer size tuning is done by operating system automatically, atleast on linux.
/proc/sys/net/core/rmem_default, /proc/sys/net/core/wmem_default, /proc/sys/net/core/rmem_max and /proc/sys/net/core/wmem_max values influence the buffer size.
With machine /proc/sys/net/core/rmem_max and /proc/sys/net/core/rmem_max values 128KB, I did writes of 100KB keys. The behavior is same with or without patch. It does a 1024byte write followed by 100K write.
write(3, "oooooooooooooooooooooooooooooooo"..., 1024) = 1024
write(3, "oooooooooooooooooooooooooooooooo"..., 99328) = 99328
When I did 500KB key writes without the patch, the kernel did memory mapped io.
mmap2(NULL, 507904, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f61000
write(3, "oooooooooooooooooooooooooooooooo"..., 1024) = 1024
write(3, "oooooooooooooooooooooooooooooooo"..., 498688) = 498688
munmap(0xb7f61000, 507904)
The effect of setting buffer size might differ from one OS to the other. So at this point I am not too sure whether we should control this from our end especially because the processes are in the same machine and the kernel does optmizations like using the mem mapping.
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-1788:
---------------------------------------------
Attachment: patch-1788.txt
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533890 ]
Christian Kunz commented on HADOOP-1788:
----------------------------------------
I applied the patch, but do not see any difference when running strace against the C++ application or the java side:
c++:
..
read(29, "\7\0\6\34\30\0\0\0\201R\214\307y\3\363~\364\344\321\277"..., 1024) = 1024
..
gettimeofday({1192048782, 69558}, {0, 0}) = 0
..
write(29, "%\316\tU\351r2V\302\37\0\0\0\0\0002\30\24\0\0\0\364\343"..., 1024) = 1024
..
java:
...
recv(234, "2", 1, 0) = 1
recv(234, "\30", 1, 0) = 1
recv(234, "\24\0\0\0\253\335\263\314\217|5\236\226\tC\34v\1~%\0\0"..., 24, 0) = 24
recv(234, "\0", 1, 0) = 1
recv(234, "2", 1, 0) = 1
recv(234, "\30", 1, 0) = 1
recv(234, "\24\0\0\0\253\335\263\322\'\30ivA\213\26\242K5m\312\0\0"..., 24, 0) = 24
recv(234, "\0", 1, 0) = 1
...
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1788) Increase the buffer size of pipes
from 1k to 128k
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532768 ]
Owen O'Malley commented on HADOOP-1788:
---------------------------------------
Did you verify that the buffer increased using strace?
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.