You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Allen Wittenauer (JIRA)" <ji...@apache.org> on 2011/06/20 06:20:47 UTC
[jira] [Created] (HADOOP-7405) libhadoop is all or nothing
libhadoop is all or nothing
---------------------------
Key: HADOOP-7405
URL: https://issues.apache.org/jira/browse/HADOOP-7405
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 0.20.203.0, 0.23.0
Environment: Everything not Linux
Reporter: Allen Wittenauer
Priority: Blocker
As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably. The most common problem is native compression, but other issues such as Mac OS X's group support broke as well. The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported. This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051800#comment-13051800 ]
Allen Wittenauer commented on HADOOP-7405:
------------------------------------------
That's an interesting idea. How would this work? Are you thinking that during the native load, we detect what soversion got loaded and then base available functionality on that?
I was thinking (far) down the road that we could basically dlopen() modules from within libhadoop.so. Then as functionality got ported, one would just drop in the appropriate shared lib. This also makes codec support very interesting: instead of having different bootstrap code for every codec, one could generalize it more than it is today.
> libhadoop is all or nothing
> ---------------------------
>
> Key: HADOOP-7405
> URL: https://issues.apache.org/jira/browse/HADOOP-7405
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.20.203.0, 0.23.0
> Environment: Everything not Linux
> Reporter: Allen Wittenauer
> Priority: Blocker
> Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably. The most common problem is native compression, but other issues such as Mac OS X's group support broke as well. The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported. This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051808#comment-13051808 ]
Todd Lipcon commented on HADOOP-7405:
-------------------------------------
I'm thinking it would just expect a particular version, and if you have an old version on your system, it should refuse to load with an error logged.
Supporting past versions of the .so on new versions of Hadoop is a large matrix to test -- we already do a bad job of testing native code.
If there are regressions with regard to OS support, we should catch those by having people with strange OSes testing trunk and voting on releases. Building a complicated framework into the code doesn't seem like the easiest solution to me.
> libhadoop is all or nothing
> ---------------------------
>
> Key: HADOOP-7405
> URL: https://issues.apache.org/jira/browse/HADOOP-7405
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.20.203.0, 0.23.0
> Environment: Everything not Linux
> Reporter: Allen Wittenauer
> Priority: Blocker
> Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably. The most common problem is native compression, but other issues such as Mac OS X's group support broke as well. The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported. This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051813#comment-13051813 ]
Allen Wittenauer commented on HADOOP-7405:
------------------------------------------
Ah, ok. The soversion essentially moves lockstep with a given release. If we were to go to that method, we'd need to split libhadoop in half to fix the regression problem.
Granted, it is simple, but it doesn't necessarily prevent the problem that we've found ourselves. Unfortunately, I do not believe anyone on the PMC attempts to run non-Linux on a standard basis. Using release votes to prevent these situations from happening is pretty much a non-starter.
> libhadoop is all or nothing
> ---------------------------
>
> Key: HADOOP-7405
> URL: https://issues.apache.org/jira/browse/HADOOP-7405
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.20.203.0, 0.23.0
> Environment: Everything not Linux
> Reporter: Allen Wittenauer
> Priority: Blocker
> Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably. The most common problem is native compression, but other issues such as Mac OS X's group support broke as well. The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported. This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051796#comment-13051796 ]
Todd Lipcon commented on HADOOP-7405:
-------------------------------------
It seems the correct answer here is to actually use soversions properly, no?
> libhadoop is all or nothing
> ---------------------------
>
> Key: HADOOP-7405
> URL: https://issues.apache.org/jira/browse/HADOOP-7405
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.20.203.0, 0.23.0
> Environment: Everything not Linux
> Reporter: Allen Wittenauer
> Priority: Blocker
> Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably. The most common problem is native compression, but other issues such as Mac OS X's group support broke as well. The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported. This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052122#comment-13052122 ]
Todd Lipcon commented on HADOOP-7405:
-------------------------------------
Allen, if you want to write a patch that adds a call like:
boolean isNativeFeatureSupported(long featureCode);
...with a set of enums for various features, I would happily review it.
> libhadoop is all or nothing
> ---------------------------
>
> Key: HADOOP-7405
> URL: https://issues.apache.org/jira/browse/HADOOP-7405
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.20.203.0, 0.23.0
> Environment: Everything not Linux
> Reporter: Allen Wittenauer
> Priority: Blocker
> Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably. The most common problem is native compression, but other issues such as Mac OS X's group support broke as well. The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported. This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-7405) libhadoop is all or nothing
Posted by "Allen Wittenauer (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer resolved HADOOP-7405.
--------------------------------------
Resolution: Won't Fix
Hadoop is not focused on portability.
> libhadoop is all or nothing
> ---------------------------
>
> Key: HADOOP-7405
> URL: https://issues.apache.org/jira/browse/HADOOP-7405
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.20.203.0, 0.23.0
> Environment: Everything not Linux
> Reporter: Allen Wittenauer
> Priority: Blocker
> Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably. The most common problem is native compression, but other issues such as Mac OS X's group support broke as well. The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported. This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052118#comment-13052118 ]
Allen Wittenauer commented on HADOOP-7405:
------------------------------------------
bq. Since Hadoop Kerberos Mac OS X support was never fully there, it is not possible to compile libhadoop due to some compiler errors.
The compiler errors are fairly simple to fix on Darwin. I don't know why, but it seems like 9 times out of 10, we favor BSD functionality when we go with something non-portable.
bg. Because of this my take is that if we require native code to run Hadoop, we should provide the full set of native code for each platform we are building for.
Regardless of what happens in this jira, we need a testsuite for the C code anyway. OS X actually proves out that even if the code compiles, it doesn't necessarily mean it works properly. (See HADOOP-7367).
bg. A while ago I've opened a HADOOP-7083 to enable running Hadoop with Kerberos ON without relying on some libhadoop functionality and the argument there was that doing that was a security risk.
Right. It wanted to create a third security mode where some stuff worked and some stuff didn't. That's not quite what I'm asking for here and it wouldn't actually fix the problem we're hitting anyway. The security functionality is orthogonal to the compression functionality. That's the base, surface issue. Since it is in one big chunk, we broke *both*.
(While I guess it wasn't obvious, I should probably state that I'm not looking for a "partially working" security mode. The scope of what constitutes a working unit would still need to be defined. It is more than reasonable to say that all of the functions that are directly security related would need to be ported and treated like one block. Asking libhadoop.so if it "supports security" seems like a reasonable thing to ask it.)
The problem that we've got is that we have a lot of unrelated code sitting in libhadoop.so. Every time we add something we run the risk of regressing features out of platforms other than Linux since those other platforms are an afterthought. HADOOP-7206 may actually be a great example of this: if we go with a pure native implementation, we won't be able to support Snappy on anything but Linux with the current state of things. Lack of compression support has a *direct* impact on the client. I'd be surprised if the majority of shops are only using Linux clients.
Wouldn't it be great to be able to ask the lib "do you support gzip, do you support snappy, do you support lzo, do you support security, ..."? Then we could add code as needed, do ports as needed, etc. An alternative would be that we start breaking libhadoop up into at least related functionality.
I suppose the other outcome might be that we as a community just admit that we don't support Hadoop on anything but Linux and give up on any semblance of portability. More and more code is being added or rewritten in C. I would be surprised if this trend changes.
> libhadoop is all or nothing
> ---------------------------
>
> Key: HADOOP-7405
> URL: https://issues.apache.org/jira/browse/HADOOP-7405
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.20.203.0, 0.23.0
> Environment: Everything not Linux
> Reporter: Allen Wittenauer
> Priority: Blocker
> Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably. The most common problem is native compression, but other issues such as Mac OS X's group support broke as well. The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported. This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052033#comment-13052033 ]
Alejandro Abdelnur commented on HADOOP-7405:
--------------------------------------------
Since Hadoop Kerberos Mac OS X support was never fully there, it is not possible to compile libhadoop due to some compiler errors.
Having libhadoop working partially on different platforms does not seem a good idea as it will mislead users to believe that a particular platform is supported when is 'partially' supported. I'd rather go for the principle of least surprise here (all or nothing), this will make Hadoop easier to user.
A while ago I've opened a HADOOP-7083 to enable running Hadoop with Kerberos ON without relying on some libhadoop functionality and the argument there was that doing that was a security risk.
I'm not arguing for HADOOP-7083 to be considered again, what I'm trying to state is that if we go the partial libhadoop functionality we'll have to come up with a checklist that will have to identify if certain functionality provided by native libs is optional, can hadoop work without it, is a security risk, etc. This will complicate the not only the lives of developers but of users.
Because of this my take is that if we require native code to run Hadoop, we should provide the full set of native code for each platform we are building for. Only for functionality that there is Java fallback (at a cost of performance) native-code should be optional.
> libhadoop is all or nothing
> ---------------------------
>
> Key: HADOOP-7405
> URL: https://issues.apache.org/jira/browse/HADOOP-7405
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.20.203.0, 0.23.0
> Environment: Everything not Linux
> Reporter: Allen Wittenauer
> Priority: Blocker
> Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably. The most common problem is native compression, but other issues such as Mac OS X's group support broke as well. The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported. This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira