You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Olga Natkovich (JIRA)" <ji...@apache.org> on 2010/02/02 19:16:19 UTC
[jira] Created: (PIG-1215) Make Hadoop jobId more prominent in the
client log
Make Hadoop jobId more prominent in the client log
--------------------------------------------------
Key: PIG-1215
URL: https://issues.apache.org/jira/browse/PIG-1215
Project: Pig
Issue Type: Improvement
Reporter: Olga Natkovich
Assignee: Ashutosh Chauhan
Fix For: 0.7.0
This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
The woould like to see each job id on a separate line in the following format:
hadoopJobId: job_123456789
They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1215) Make Hadoop jobId more prominent in the
client log
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1215:
----------------------------------
Status: Patch Available (was: Open)
Running through hudson to catch if there are failures. Not ready for commit yet.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1215) Make Hadoop jobId more prominent in the
client log
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1215:
----------------------------------
Status: Open (was: Patch Available)
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1215) Make Hadoop jobId more prominent in
the client log
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831817#action_12831817 ]
Olga Natkovich commented on PIG-1215:
-------------------------------------
can we also make the value NOT_AVAILABLE rather than NOT AVAILABLE to make it easier for tools to parse
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1215) Make Hadoop jobId more prominent in
the client log
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833335#action_12833335 ]
Hadoop QA commented on PIG-1215:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12435733/pig-1215_1.patch
against trunk revision 909584.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 213 javac compiler warnings (more than the trunk's current 212 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/204/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/204/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/204/console
This message is automatically generated.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch, pig-1215_1.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1215) Make Hadoop jobId more prominent in the
client log
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1215:
----------------------------------
Status: Patch Available (was: Open)
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch, pig-1215_1.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1215) Make Hadoop jobId more prominent in the
client log
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1215:
----------------------------------
Attachment: pig-1215_3.patch
This patch adds a new property called pig.logfile.level which if provided, sets the log level to the value of the property and then all the log messages above that log level goes to the log file. The name of log file could be provided with -l option of Pig. If -l option is not provided log file name would be randomly generated by Pig (as was the case earlier).
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch, pig-1215_1.patch, pig-1215_3.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1215) Make Hadoop jobId more prominent in
the client log
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832775#action_12832775 ]
Ashutosh Chauhan commented on PIG-1215:
---------------------------------------
Pig already has extensive support for logging via log4j. Applications wanting to mine pig logs can configure the pig logging using -4 option of Pig. for example if you provide "pig -4 logger.conf myscript.pig" log4j properties specified in file logger.conf will be picked up. So, application can configure log4j as they want. One such config which I tried is following:
{noformat}
$ cat logger.conf
log4j.rootLogger=INFO, R
log4j.logger.org.apache.pig=INFO
log4j.appender.R=org.apache.log4j.RollingFileAppender
# file name in which log messages will be printed.
log4j.appender.R.File=piglogs.log
log4j.appender.R.MaxFileSize=100KB
# Keep one backup file
log4j.appender.R.MaxBackupIndex=1
log4j.appender.R.layout=org.apache.log4j.PatternLayout
# Pattern to output the caller's file name and line number.
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
{noformat}
Above log4j configuration prints all the info level messages in piglogs.log file. All the other properties of log4j (like log rotation, format of log messages etc) is configurable through this file. More info at: http://logging.apache.org/log4j/1.2/manual.html
Thus no new property or any other change in Pig is required to redirect log messages to file.
Unit test failure is one of those temporary "port out of range" failures unrelated to patch. Javac warning is deprecation warning due to use of a deprecated Hadoop api. These deprecation warnings will be addressed when we move to new Hadoop api.
This patch is ready for review.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1215) Make Hadoop jobId more prominent in
the client log
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832743#action_12832743 ]
Hadoop QA commented on PIG-1215:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12435554/pig-1215.patch
against trunk revision 908324.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 213 javac compiler warnings (more than the trunk's current 212 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/201/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/201/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/201/console
This message is automatically generated.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1215) Make Hadoop jobId more prominent in
the client log
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831812#action_12831812 ]
Olga Natkovich commented on PIG-1215:
-------------------------------------
I would like to request an additional change to make sure that we can write HadoopId information to the client side log file not just stdout. This would happen only if special property is used.
So the additional ask is to implement handling of this new property and when it is present to make sure that all messages at the level of INFO are written to the log file. This can be accomplished by changing the log listener for the log file so it picks up INFO level log events.
We don't want to do this by default because it would drastically increase the number of log files created by Pig since now we only create the file when there is a real problem executing it.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1215) Make Hadoop jobId more prominent in
the client log
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831316#action_12831316 ]
Hadoop QA commented on PIG-1215:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12435211/pig-1215.patch
against trunk revision 907760.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/194/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/194/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/194/console
This message is automatically generated.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1215) Make Hadoop jobId more prominent in the
client log
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1215:
----------------------------------
Status: Open (was: Patch Available)
First part of the patch requires no new test cases since it doesnt add or delete any existing functionality, but rather modifies the existing one. That is already covered by existing test cases. Second part of patch related to -D switch was manually tested.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch, pig-1215_1.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1215) Make Hadoop jobId more prominent in
the client log
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834458#action_12834458 ]
Olga Natkovich commented on PIG-1215:
-------------------------------------
The log part of the change is not quite right. I think we need to have a property that controls the level of output to the log file. We don't want new file - we want to re-use the existing log file and just put more infomration there.
I am still looking at the second half of the patch.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch, pig-1215_1.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1215) Make Hadoop jobId more prominent in the
client log
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1215:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch checked-in.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch, pig-1215_1.patch, pig-1215_3.patch, pig-1215_4.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1215) Make Hadoop jobId more prominent in
the client log
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831309#action_12831309 ]
Hadoop QA commented on PIG-1215:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12435211/pig-1215.patch
against trunk revision 907760.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/205/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/205/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/205/console
This message is automatically generated.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1215) Make Hadoop jobId more prominent in the
client log
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1215:
----------------------------------
Attachment: pig-1215_1.patch
Added a -D switch "-Dpig.logging.PutAllMessagesInFile=myfile.log" If this switch is specified all the log messages will go to file specified in value of switch (myfile.log in the given example). Jobids will be printed separately on each line as:
{noformat}
HadoopJobId: job_200912011426_19926
{noformat}
Other changes as in previous patch.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch, pig-1215_1.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1215) Make Hadoop jobId more prominent in
the client log
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835558#action_12835558 ]
Olga Natkovich commented on PIG-1215:
-------------------------------------
Changes look good and I also ran a few tests.
The only thing that needs to be changes is that when loglevel is specified we still want to print to screen in addition to printing to the file.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch, pig-1215_1.patch, pig-1215_3.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1215) Make Hadoop jobId more prominent in the
client log
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1215:
----------------------------------
Attachment: pig-1215.patch
Okay, so I think with this patch Pig will reliably print the job-ids of the jobs it is submitting without risking an infinite loop. Got rid of time-outs as well as explicitly checking job's status. Still, need to add the property which will toggle the info level messages to file instead of std-out. Putting up the patch, so that other developers can review the logic.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (PIG-1215) Make Hadoop jobId more prominent in the
client log
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai closed PIG-1215.
---------------------------
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch, pig-1215_1.patch, pig-1215_3.patch, pig-1215_4.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1215) Make Hadoop jobId more prominent in the
client log
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1215:
----------------------------------
Status: Patch Available (was: Open)
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1215) Make Hadoop jobId more prominent in the
client log
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1215:
----------------------------------
Attachment: pig-1215_4.patch
Change as suggested by Olga. Other parts of patch are as before.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch, pig-1215.patch, pig-1215_1.patch, pig-1215_3.patch, pig-1215_4.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1215) Make Hadoop jobId more prominent in the
client log
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1215:
----------------------------------
Attachment: pig-1215.patch
With this patch, Job ids will now be printed as:
2010-02-08 13:54:26,441 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201001051416_2282
We currently wait for ~10 seconds for job tracker to assign the job-ids. Its possible that a busy job-tracker may not assign the job-id within that time. In such cases, Pig will log :
2010-02-08 13:54:26,441 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: NOT AVAILABLE
Note that job may still complete successfully in such cases.
Also, these job ids will be printed in order they are submitted to job-tracker.
> Make Hadoop jobId more prominent in the client log
> --------------------------------------------------
>
> Key: PIG-1215
> URL: https://issues.apache.org/jira/browse/PIG-1215
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-1215.patch
>
>
> This is a request from applications that want to be able to programmatically parse client logs to find hadoop Ids.
> The woould like to see each job id on a separate line in the following format:
> hadoopJobId: job_123456789
> They would also like to see the jobs in the order they are executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.