You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amar Kamat (JIRA)" <ji...@apache.org> on 2009/04/24 09:22:30 UTC
[jira] Created: (HADOOP-5737) UGI checks in testcases is broken
UGI checks in testcases is broken
---------------------------------
Key: HADOOP-5737
URL: https://issues.apache.org/jira/browse/HADOOP-5737
Project: Hadoop Core
Issue Type: Bug
Components: test
Reporter: Amar Kamat
While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
{code}
Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
===================================================================
--- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
+++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
@@ -40,6 +40,7 @@
if (LOG.isDebugEnabled()) {
LOG.debug("ugi=" + ugi);
}
+ LOG.info("ugi=" + ugi);
if (ugi != null) {
user = ugi.getUserName();
{code}
While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706424#action_12706424 ]
Amar Kamat commented on HADOOP-5737:
------------------------------------
Ant tests passed on my box.
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Amar Kamat
> Attachments: HADOOP-5737-v1.3.patch
>
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5737:
-------------------------------
Summary: UGI checks in testcases are broken (was: UGI checks in testcases is broken)
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Amar Kamat
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702306#action_12702306 ]
Arun C Murthy commented on HADOOP-5737:
---------------------------------------
Does this anomaly happen in all test-cases? Or only in TestMiniMRWithDFSWithDistinctUsers?
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Amar Kamat
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708336#action_12708336 ]
Arun C Murthy commented on HADOOP-5737:
---------------------------------------
+1
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-5737-v1.3.patch, HADOOP-5737-v1.5.patch, HADOOP-5737-v1.7.patch
>
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708637#action_12708637 ]
Hudson commented on HADOOP-5737:
--------------------------------
Integrated in Hadoop-trunk #834 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/834/])
. Fixes a problem in the way the JobTracker used to talk to other daemons like the NameNode to get the job's files. Also adds APIs in the JobTracker to get the FileSystem objects as per the JobTracker's configuration. Contributed by Amar Kamat.
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-5737-v1.3.patch, HADOOP-5737-v1.5.patch, HADOOP-5737-v1.7.patch
>
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707232#action_12707232 ]
Devaraj Das commented on HADOOP-5737:
-------------------------------------
I just looked at the patch and it seems okay to me. Passing the JobTracker object to JobHistory seems less preferable when things can work without that change.
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-5737-v1.3.patch, HADOOP-5737-v1.5.patch
>
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702309#action_12702309 ]
Arun C Murthy commented on HADOOP-5737:
---------------------------------------
Also, another thing to check is whether it happens only in when using the Mini* clusters, and not in single-node cluster etc.
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Amar Kamat
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5737:
-------------------------------
Attachment: HADOOP-5737-v1.7.patch
Attaching a patch incorporating Arun's comment. Result of test-patch
{code}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] -1 release audit. The applied patch generated 484 release audit warnings (more than the trunk's current 483 warnings).
{code}
No extra files are added. Release audit warning is because of change to a public api (JobHistory.init()).
Running ant test now.
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-5737-v1.3.patch, HADOOP-5737-v1.5.patch, HADOOP-5737-v1.7.patch
>
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702320#action_12702320 ]
Amar Kamat commented on HADOOP-5737:
------------------------------------
bq. Does this anomaly happen in all test-cases? Or only in TestMiniMRWithDFSWithDistinctUsers?
This happens whenever the users for jobtracker and job are different. For example, the jobtracker runs as _foo_ and a user _bar_ submits a job. So what happens in these testcase is that
# user bar submits a job
# jobtracker (running as foo) tries to do a job.init()
# fs does read-permission check and uses {{ugi = UserGroupInformation.getCurrentUGI()}} which points to user bar.
Ideally the ugi should point to jobtracker's user.
bq. Also, another thing to check is whether it happens only in when using the Mini* clusters, and not in single-node cluster etc.
Arun this is a bug with the testcase only. I will still try it out on a single node cluster.
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Amar Kamat
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5737:
-------------------------------
Attachment: HADOOP-5737-v1.5.patch
Attaching a patch the reduces code duplication. I think its ok to pass open up jobtracker conf as only jip requires it now. JobHistory already has it. To pass the jobtracker's fs to JobHistory, we need to pass the jobtracker handle to JobHistory. Result of test-patch
{code}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
Ant tests passed on my box.
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-5737-v1.3.patch, HADOOP-5737-v1.5.patch
>
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das resolved HADOOP-5737.
---------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
I just committed this. Thanks, Amar!
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-5737-v1.3.patch, HADOOP-5737-v1.5.patch, HADOOP-5737-v1.7.patch
>
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706516#action_12706516 ]
Arun C Murthy commented on HADOOP-5737:
---------------------------------------
The patch looks fine. I'd rather have a JobTracker.getFileSystem() and JobTracker.getFileSystem(Path) than spread the code all over... please document the rationale for these apis too for posterity.
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Amar Kamat
> Attachments: HADOOP-5737-v1.3.patch
>
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5737:
-------------------------------
Attachment: HADOOP-5737-v1.3.patch
Attaching a patch that uses jobtracker's credentials for contacting the namenode instead of using job's credentials. This patch solves the issue. Result of test-patch
{code}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
Running ant test now.
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Amar Kamat
> Attachments: HADOOP-5737-v1.3.patch
>
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-5737:
-------------------------------------------
Component/s: (was: test)
mapred
Assignee: Amar Kamat
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks ...
You may set the LOG to debug level instead of changing the code.
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-5737-v1.3.patch
>
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706718#action_12706718 ]
Amar Kamat commented on HADOOP-5737:
------------------------------------
Hey Arun thanks for looking. JobInProgress requires access to JobTracker's conf for
# Making connections to namenode
# Passing it to other classes like CleanupQueue etc.
For case #1, JobTracker.getFileSystem() makes sense but for case #2, its better to pass the conf. Hence the {{getConf()}} api.
bq. please document the rationale for these apis too for posterity.
If we keep the {{getConf()}} api, do we need any documentation for this?
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Amar Kamat
> Attachments: HADOOP-5737-v1.3.patch
>
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707126#action_12707126 ]
Arun C Murthy commented on HADOOP-5737:
---------------------------------------
bq. For case #1, JobTracker.getFileSystem() makes sense but for case #2, its better to pass the conf.
How is it better to pass the conf? JobTracker.getFileSystem(path) would take the path, resolve it against it's own conf and return the FileSystem - that way it's more explicit that you need the JobTracker's view of the filesytem. getConf() is too generic and it's too easy to forget to call getConf() in the future etc.
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-5737-v1.3.patch
>
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5737) UGI checks in testcases are broken
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707671#action_12707671 ]
Amar Kamat commented on HADOOP-5737:
------------------------------------
Ant tests passed on my box.
> UGI checks in testcases are broken
> ----------------------------------
>
> Key: HADOOP-5737
> URL: https://issues.apache.org/jira/browse/HADOOP-5737
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-5737-v1.3.patch, HADOOP-5737-v1.5.patch, HADOOP-5737-v1.7.patch
>
>
> While running {{TestMiniMRWithDFSWithDistinctUsers}}, I used this patch to test the ugi checks
> {code}
> Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java
> ===================================================================
> --- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (revision 768189)
> +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java (working copy)
> @@ -40,6 +40,7 @@
> if (LOG.isDebugEnabled()) {
> LOG.debug("ugi=" + ugi);
> }
> + LOG.info("ugi=" + ugi);
>
> if (ugi != null) {
> user = ugi.getUserName();
> {code}
> While initializing a job, the ugi information should point to jobtracker as jobtracker does a dfs read. But today we will see that the log shows _pi_ as the caller instead of the jobtracker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.