You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2009/10/13 03:23:31 UTC
[jira] Created: (HADOOP-6311) Add support for unix domain sockets
to JNI libs
Add support for unix domain sockets to JNI libs
-----------------------------------------------
Key: HADOOP-6311
URL: https://issues.apache.org/jira/browse/HADOOP-6311
Project: Hadoop Common
Issue Type: New Feature
Components: native
Reporter: Todd Lipcon
Assignee: Todd Lipcon
For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets
to JNI libs
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765154#action_12765154 ]
Doug Cutting commented on HADOOP-6311:
--------------------------------------
Forking is certainly often not an optimal path. That said, forking is quite different than using a snapshot build. In that case we committed the jar. Here, Todd's proposing we commit the source, properly forking. We'd be taking on the maintenance of this code, simply using the Android code as a starting point. That might be warranted if Android is not a project that's in the business of producing independent utility jar file releases, which it does not appear to be.
If you agree with the above analysis, then the question has little to do with the source of the code but rather with whether Hadoop's codebase should include unix domain socket utilities. My general preference is to refactor later rather than earlier: until we have another use of unix domain sockets within Hadoop, or at least have another expected use, keep the code with its single user in HDFS. Do you have another expected use?
> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets
to JNI libs
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765171#action_12765171 ]
Todd Lipcon commented on HADOOP-6311:
-------------------------------------
bq. Okay, but I'd then vote that we not commit this until either there's another dependency on it that's committed or we're about to commit HDFS-347.
+1 on not committing until we're ready to do the whole thing.
However, JIRA won't let me turn it into a subtask of HDFS-347: "Only non-sub-task issues from the same project (HADOOP) can be selected."
> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6311) Add support for unix domain sockets
to JNI libs
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-6311:
--------------------------------
Attachment: hadoop-6311.txt
Attaching preliminary patch. The patch is large due to a lot of autotools-generated files.
Most of this code comes from Android, with some modifications (added timeouts, got rid of android-specific stuff, cleaned up APIs a little)
> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets
to JNI libs
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765199#action_12765199 ]
Todd Lipcon commented on HADOOP-6311:
-------------------------------------
bq. You propose to commit MBs of new code, which we don't know nothing about and take the burden of supporting it, if I understood Doug correctly.
As I said above, the patch is huge due to having to rerun autoconf/aclocal/etc. The actual amount of new code in the patch:
{noformat}
build.xml | 10 +
.../org/apache/hadoop/net/unix/Credentials.java | 52 +
.../apache/hadoop/net/unix/LocalServerSocket.java | 122 +
.../org/apache/hadoop/net/unix/LocalSocket.java | 288 +
.../apache/hadoop/net/unix/LocalSocketAddress.java | 95 +
.../apache/hadoop/net/unix/LocalSocketImpl.java | 499 +
.../src/org/apache/hadoop/net/unix/JNIHelp.cpp | 85 +
.../src/org/apache/hadoop/net/unix/JNIHelp.h | 76 +
.../src/org/apache/hadoop/net/unix/Makefile.am | 51 +
.../hadoop/net/unix/java_io_FileDescriptor.cpp | 201 +
.../org_apache_hadoop_net_unix_LocalSocketImpl.cpp | 1012 +
.../apache/hadoop/net/unix/TestLocalSockets.java | 208 +
{noformat}
So, this isn't a tiny patch, but it's certainly not huge either. If you discount comments and whitespace, it's 867 lines of cpp, 628 lines of java (120 are test).
If you prefer, I can rip out all of the code but the very minimal required, but I don't consider this a particularly large patch.
As for alternatives, I think many of them were thrown out due to security concerns in HDFS-347. I will certainly look into Raghu's proposed improvement to FSInputChecker, and if that gets the same gains as this, I'm happy to throw out all of the localsockets complication.
> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets
to JNI libs
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765669#action_12765669 ]
Sanjay Radia commented on HADOOP-6311:
--------------------------------------
Todd, have you considered simply providing a jar and c-library from( or is derived from) the the Android code and making it available for hadoop? This way one can track the bug fixes and improvements in original Andriod.
> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets
to JNI libs
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765168#action_12765168 ]
Doug Cutting commented on HADOOP-6311:
--------------------------------------
> I put this code in common since that's the easiest integration point for additional native code.
Okay, but I'd then vote that we not commit this until either there's another dependency on it that's committed or we're about to commit HDFS-347.
Longer term, perhaps we'll add native code to the HDFS project and HDFS-347 could become a self-contained HDFS-only patch.
> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets
to JNI libs
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765261#action_12765261 ]
Konstantin Shvachko commented on HADOOP-6311:
---------------------------------------------
> I will certainly look into Raghu's proposed improvement ...
Thanks, this would be really good. Let's try what we can with internal solutions then turn to external ones. Is Android the best one? There should be alternatives there as well. Btw., does this library work on Windows, Solaris, etc.?
I think Dhruba is making a good point in HDFS-347 that until we really understand what causes the slowdown we cannot efficiently fight it.
On a side note, this project becomes fairly complicated, should we consider branching it?
> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets
to JNI libs
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765230#action_12765230 ]
Owen O'Malley commented on HADOOP-6311:
---------------------------------------
I'll remove my -1, but stick by a -0. I think this is a very bad idea. Forking code out of existing projects is a very dangerous practice and this is a huge chunk of code to be incorporating into Hadoop. This will become critical code in the heart of HDFS that none of us developed. That is troubling.
I agree that we should not commit this until HDFS-347 is ready to commit.
I suspect that putting it into HDFS is harder than it is worth, since HDFS doesn't have a native component already.
> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets
to JNI libs
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765169#action_12765169 ]
Doug Cutting commented on HADOOP-6311:
--------------------------------------
In other words, I view this as a sub-task of HDFS-347. If you agree, perhaps you could convert it?
> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets
to JNI libs
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765128#action_12765128 ]
Owen O'Malley commented on HADOOP-6311:
---------------------------------------
-1
Forking part of Android into Hadoop, while legal, is a really bad idea. *We* will become responsible for maintaining the code and will need to track all of the bug fixes that go into it. This is a huge maintenance burden.
This might be acceptable if you can get the Android project to make releases that we can use. (Or if they split it out into a stand alone project that they also use...)
For some related history, look at the pain that happened when we used a snapshot build from Apache Commons CLI. We eventually had to roll back to the released version including downgrading all of our usage of the new features.
> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets
to JNI libs
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765195#action_12765195 ]
Konstantin Shvachko commented on HADOOP-6311:
---------------------------------------------
I am really hesitant we should commit to Hadoop sources from another project. As Owen mentioned the main concern is the support of the code. You propose to commit MBs of new code, which we don't know nothing about and take the burden of supporting it, if I understood Doug correctly.
- Ok, so does the purpose justifies the sacrifice?
Not sure that even 30% improvement in read performance justifies this. Looking at your benchmarks I think the number will go down to 10-20% for MapReduce in general because of data-node overhead, and since not all task reads are local.
- Are there alternatives?
You probably need at least to consider alternatives, and explain why they don't work. Seems like Dhruba and Raghu made attempts to turn this to solving problems inside data-node code, which has good chances imo to reach the same results without massive infusion of alien sources.
> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets
to JNI libs
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765161#action_12765161 ]
Todd Lipcon commented on HADOOP-6311:
-------------------------------------
bq. This might be acceptable if you can get the Android project to make releases that we can use. (Or if they split it out into a stand alone project that they also use...)
I think there's little chance of this. The original code has (a) some android specific things like a special android socket namespace, (b) reliance on their custom build system, and (c) reliance on a bunch of other utility code elsewhere in Android. This is also such a small part of Android that I am confident they'd have no interest in splitting it out to a separate library - it would just be a waste of time for them.
Regarding maintenance and bugs, I think if you look at the code you'll see that it's not super complicated; obviously there's always a chance for bugs, especially on different platforms, but I don't see this code evolving much from where it is today (in Android it has had only one edit in the year since it was contributed). Also, since Android targets a very specific platform (and specific nonstandard JVM, even) I don't think we'd see the same bugs they would in practice.
bq. until we have another use of unix domain sockets within Hadoop, or at least have another expected use, keep the code with its single user in HDFS.
I put this code in common since that's the easiest integration point for additional native code. If I were to put it in an entirely separate JNI library, we'd have to make changes to (or duplicate) NativeCodeLoader, ask users to install multiple native libraries, add appropriate build infrastructure to the hdfs build, etc. I think it makes most sense to have a single "native-code based optimizations" library that contains these optional extensions in common even if there is currently only a single user.
> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.