You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Allen Wittenauer (JIRA)" <ji...@apache.org> on 2011/06/08 19:19:58 UTC
[jira] [Created] (HADOOP-7367) getgrouplist() in getGroup.c is not
portable
getgrouplist() in getGroup.c is not portable
--------------------------------------------
Key: HADOOP-7367
URL: https://issues.apache.org/jira/browse/HADOOP-7367
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 0.22.0, 0.23.0
Environment: System V
Reporter: Allen Wittenauer
getGroupIDList uses getgrouplist() to fetch the groups for a user. Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7367) getgrouplist() in getGroup.c is
not portable
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046090#comment-13046090 ]
Allen Wittenauer commented on HADOOP-7367:
------------------------------------------
A few points:
* The inclusion of this routine is preventing the native code from being compiled on platforms where it previously worked.
* We have multiple ways to try to solve this:
a) Rewrite the entire routine to remove the dependency on getgrouplist()
b) Do auto-detection to use the built-in getgrouplist, otherwise use one we supply
c) Use the current version when getgrouplist is present, otherwise use a completely different code stream that uses getgrent() etc.
> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
> Key: HADOOP-7367
> URL: https://issues.apache.org/jira/browse/HADOOP-7367
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.22.0, 0.23.0
> Environment: System V
> Reporter: Allen Wittenauer
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user. Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7367) getgrouplist() in getGroup.c is not
portable
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated HADOOP-7367:
-------------------------------------
Status: Patch Available (was: Open)
Several things about this patch:
* A check has been added to configure to look for a getgrouplist routine
* If it is found, we use a getgrouplist routine that tries to do it all in one pass, based upon allocating POSIX_NGROUPS_MAX groups (up to 32) for the initial buffer. If the user has more than 32, we depend upon getgrouplist properly returning the number of groups for the user. (So this will only match the first 32 on Darwin)
* If getgrouplist isn't there, we run through the entire groups db, looking for entries.
Please note that at least Solaris requires _POSIX_PTHREAD_SEMANTICS defined for the 5 param version of get*_r. I'll cover that in a separate patch.
> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
> Key: HADOOP-7367
> URL: https://issues.apache.org/jira/browse/HADOOP-7367
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.22.0, 0.23.0
> Environment: System V (Solaris, HP-UX, AIX?)
> Mac OS X
> Reporter: Allen Wittenauer
> Attachments: hadoop-7367.patch
>
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user. Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7367) getgrouplist() in getGroup.c is not
portable
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated HADOOP-7367:
-------------------------------------
Attachment: hadoop-7367.patch
> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
> Key: HADOOP-7367
> URL: https://issues.apache.org/jira/browse/HADOOP-7367
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.22.0, 0.23.0
> Environment: System V (Solaris, HP-UX, AIX?)
> Mac OS X
> Reporter: Allen Wittenauer
> Attachments: hadoop-7367.patch
>
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user. Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7367) getgrouplist() in getGroup.c is not
portable
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated HADOOP-7367:
-------------------------------------
Environment:
System V (Solaris, HP-UX, AIX?)
Mac OS X
was:System V
> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
> Key: HADOOP-7367
> URL: https://issues.apache.org/jira/browse/HADOOP-7367
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.22.0, 0.23.0
> Environment: System V (Solaris, HP-UX, AIX?)
> Mac OS X
> Reporter: Allen Wittenauer
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user. Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7367) getgrouplist() in getGroup.c is
not portable
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046096#comment-13046096 ]
Allen Wittenauer commented on HADOOP-7367:
------------------------------------------
It also appears this code doesn't work on OS X... which is... interesting.
> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
> Key: HADOOP-7367
> URL: https://issues.apache.org/jira/browse/HADOOP-7367
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.22.0, 0.23.0
> Environment: System V
> Reporter: Allen Wittenauer
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user. Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7367) getgrouplist() in getGroup.c is not
portable
Posted by "Allen Wittenauer (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated HADOOP-7367:
-------------------------------------
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
Hadoop is not focused on portability. Patch is here for those that need it though.
> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
> Key: HADOOP-7367
> URL: https://issues.apache.org/jira/browse/HADOOP-7367
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.20.203.0, 0.22.0, 0.23.0
> Environment: System V (Solaris, HP-UX, AIX?)
> Mac OS X
> Reporter: Allen Wittenauer
> Assignee: Allen Wittenauer
> Labels: regression
> Attachments: hadoop-7367.patch
>
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user. Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7367) getgrouplist() in getGroup.c is
not portable
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046209#comment-13046209 ]
Allen Wittenauer commented on HADOOP-7367:
------------------------------------------
>From the bit of playing I've done, getgrouplist on OS X doesn't update ngroups to have the total number of groups the user is in. Since the first call uses 0, it returns 0, which then dumps out of the rest of the loop.
So a fix here needs to take that into consideration. (It may be worthwhile to just use sysconf(_POSIX_NGROUPS_MAX) for the first call so that getgrouplist mostly works on OS X and others with different semantics than the Linux version).
> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
> Key: HADOOP-7367
> URL: https://issues.apache.org/jira/browse/HADOOP-7367
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.22.0, 0.23.0
> Environment: System V
> Reporter: Allen Wittenauer
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user. Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7367) getgrouplist() in getGroup.c is not
portable
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated HADOOP-7367:
-------------------------------------
Affects Version/s: 0.20.203.0
Assignee: Allen Wittenauer
Labels: regression (was: )
Adding the regression and 0.20.203 info, since this broke functionality that previously worked due to the compression codecs being in the same library. (Maybe libhadoop.so needs to get broken up?)
> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
> Key: HADOOP-7367
> URL: https://issues.apache.org/jira/browse/HADOOP-7367
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.20.203.0, 0.22.0, 0.23.0
> Environment: System V (Solaris, HP-UX, AIX?)
> Mac OS X
> Reporter: Allen Wittenauer
> Assignee: Allen Wittenauer
> Labels: regression
> Attachments: hadoop-7367.patch
>
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user. Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7367) getgrouplist() in getGroup.c is
not portable
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046303#comment-13046303 ]
Hadoop QA commented on HADOOP-7367:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12481884/hadoop-7367.patch
against trunk revision 1133125.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/596//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/596//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/596//console
This message is automatically generated.
> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
> Key: HADOOP-7367
> URL: https://issues.apache.org/jira/browse/HADOOP-7367
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.22.0, 0.23.0
> Environment: System V (Solaris, HP-UX, AIX?)
> Mac OS X
> Reporter: Allen Wittenauer
> Attachments: hadoop-7367.patch
>
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user. Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira