You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Alejandro Abdelnur (Created) (JIRA)" <ji...@apache.org> on 2012/01/25 20:33:41 UTC
[jira] [Created] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
jobtoken location property in jobconf refers to wrong jobtoken file
-------------------------------------------------------------------
Key: MAPREDUCE-3727
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: security
Affects Versions: 1.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated MAPREDUCE-3727:
------------------------------------------
Fix Version/s: 1.1.0
still investigating if this affects trunk or not, code change significantly.
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 1.1.0
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated MAPREDUCE-3727:
------------------------------------------
Attachment: MAPREDUCE-3727-branch1.patch
patch for branch-1
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727-branch1.patch, MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200208#comment-13200208 ]
Hudson commented on MAPREDUCE-3727:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1669 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1669/])
MAPREDUCE-3727. jobtoken location property in jobconf refers to wrong jobtoken file (tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240410
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727-branch1.patch, MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated MAPREDUCE-3727:
------------------------------------------
Attachment: MAPREDUCE-3727.patch
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3727:
-------------------------------------
Fix Version/s: (was: 0.23.1)
0.24.0
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.24.0, 1.1.0
>
> Attachments: MAPREDUCE-3727-branch1.patch, MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199249#comment-13199249 ]
Hadoop QA commented on MAPREDUCE-3727:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513027/MAPREDUCE-3727.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1752//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1752//console
This message is automatically generated.
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193240#comment-13193240 ]
Alejandro Abdelnur commented on MAPREDUCE-3727:
-----------------------------------------------
One possible solution would be the JobClient to remove this property before submitting the job.
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 1.1.0
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated MAPREDUCE-3727:
------------------------------------------
Resolution: Fixed
Fix Version/s: 1.2.0
Target Version/s: (was: 1.1.1)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
committed to branch-1
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 1.2.0, 0.23.1
>
> Attachments: MAPREDUCE-3727-branch1.patch, MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3727:
-------------------------------
Fix Version/s: (was: 0.23.0)
0.23.1
(Sorry, bad selection)
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3727-branch1.patch, MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200183#comment-13200183 ]
Hudson commented on MAPREDUCE-3727:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #494 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/494/])
Merge -r 1240409:1240410 from trunk to branch. FIXES: MAPREDUCE-3727
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240411
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200179#comment-13200179 ]
Hudson commented on MAPREDUCE-3727:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #481 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/481/])
Merge -r 1240409:1240410 from trunk to branch. FIXES: MAPREDUCE-3727
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240411
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193382#comment-13193382 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3727:
----------------------------------------------------
This happens on trunk also. Clearly there is a TODO in the code which says the current place is inappropriate for reading the tokens file.
If this code gets triggered, it is not just the reading of token file that will fail. Whenever this happens (don't know by what trigger), the client will try to obtain delegation tokens from the NameNode which will also fail? Have any clue why that isn't happening?
bq. One possible solution would be the JobClient to remove this property before submitting the job.
I guess you mean as part of the individual Oozie actions. Right?
The alternative could be to not fail with exception when it cannot read the file, but instead just log an error and continue. Not sure whether that's a good fix though.
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 1.1.0
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200173#comment-13200173 ]
Hudson commented on MAPREDUCE-3727:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1725 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1725/])
MAPREDUCE-3727. jobtoken location property in jobconf refers to wrong jobtoken file (tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240410
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200397#comment-13200397 ]
Hudson commented on MAPREDUCE-3727:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #946 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/946/])
MAPREDUCE-3727. jobtoken location property in jobconf refers to wrong jobtoken file (tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240410
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727-branch1.patch, MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193428#comment-13193428 ]
Alejandro Abdelnur commented on MAPREDUCE-3727:
-----------------------------------------------
Vinod, thanks for confirming this is an issue in trunk as well.
It is not possible to do it at Oozie level, the property is used within the scope of the hadoop submission code (by TokenCache, ie, computing splits), hadoop submission code should remove this property from the jobconf before writing the jobconf to hdfs.
I guess a warn and continue would work too. but it would be quite confusing to get a warning for a file from some other jobtoken file.
I think we should remove on submission (as I state before) as that property (current value) is meaningful only in the context the the current submission. If somebody is doing a submission from the submitted job (like Oozie), then they should start the whole cycle again (getting the ENV var and setting the property again in the context of the submitted job).
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 1.1.0
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3727:
-------------------------------
Target Version/s: 1.1.1
Fix Version/s: (was: 1.1.0)
(was: 0.24.0)
0.23.0
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3727-branch1.patch, MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated MAPREDUCE-3727:
------------------------------------------
Affects Version/s: 0.23.1
Fix Version/s: 0.23.1
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200451#comment-13200451 ]
Hudson commented on MAPREDUCE-3727:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #979 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/979/])
MAPREDUCE-3727. jobtoken location property in jobconf refers to wrong jobtoken file (tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240410
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727-branch1.patch, MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200178#comment-13200178 ]
Hudson commented on MAPREDUCE-3727:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #471 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/471/])
Merge -r 1240409:1240410 from trunk to branch. FIXES: MAPREDUCE-3727
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240411
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated MAPREDUCE-3727:
------------------------------------------
Status: Patch Available (was: Open)
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 1.0.0, 0.23.1
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3727:
-------------------------------
Affects Version/s: (was: 0.23.1)
0.23.0
Affected was > Fixed.
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3727-branch1.patch, MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199215#comment-13199215 ]
Tom White commented on MAPREDUCE-3727:
--------------------------------------
+1 looks good to me.
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200217#comment-13200217 ]
Hadoop QA commented on MAPREDUCE-3727:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513213/MAPREDUCE-3727-branch1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1773//console
This message is automatically generated.
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727-branch1.patch, MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200428#comment-13200428 ]
Hudson commented on MAPREDUCE-3727:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #181 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/181/])
Merge -r 1240409:1240410 from trunk to branch. FIXES: MAPREDUCE-3727
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240411
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727-branch1.patch, MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200176#comment-13200176 ]
Hudson commented on MAPREDUCE-3727:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1654 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1654/])
MAPREDUCE-3727. jobtoken location property in jobconf refers to wrong jobtoken file (tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240410
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3727) jobtoken location property in
jobconf refers to wrong jobtoken file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200408#comment-13200408 ]
Hudson commented on MAPREDUCE-3727:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #159 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/159/])
Merge -r 1240409:1240410 from trunk to branch. FIXES: MAPREDUCE-3727
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240411
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> jobtoken location property in jobconf refers to wrong jobtoken file
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-3727
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3727
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.1, 1.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 1.1.0
>
> Attachments: MAPREDUCE-3727-branch1.patch, MAPREDUCE-3727.patch
>
>
> Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
> The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
> The problem is that the *mapreduce.job.credentials.binary* property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.
> If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.
> More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira