You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Senthil Subramanian (JIRA)" <ji...@apache.org> on 2007/06/26 23:18:26 UTC
[jira] Created: (HADOOP-1533) Distcp should log to specified
location
Distcp should log to specified location
---------------------------------------
Key: HADOOP-1533
URL: https://issues.apache.org/jira/browse/HADOOP-1533
Project: Hadoop
Issue Type: Improvement
Components: util
Reporter: Senthil Subramanian
Fix For: 0.14.0
Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1533) Distcp should log to specified
location
Posted by "Senthil Subramanian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Senthil Subramanian updated HADOOP-1533:
----------------------------------------
Attachment: (was: HADOOP-1533_1.patch)
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_1.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1533) Distcp should log to specified
location
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1533:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
This has been committed. Thanks, Senthil.
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Assignee: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_2.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1533) Distcp should log to specified
location
Posted by "Senthil Subramanian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Senthil Subramanian updated HADOOP-1533:
----------------------------------------
Attachment: HADOOP-1533_1.patch
Fixed findbug warning
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_1.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1533) Distcp should log to specified
location
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12508699 ]
Hadoop QA commented on HADOOP-1533:
-----------------------------------
+1
http://issues.apache.org/jira/secure/attachment/12360699/HADOOP-1533_1.patch applied and successfully tested against trunk revision r551361.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/342/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/342/console
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_1.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1533) Distcp should log to specified
location
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12508688 ]
Hadoop QA commented on HADOOP-1533:
-----------------------------------
+0, new Findbugs warnings
http://issues.apache.org/jira/secure/attachment/12360696/HADOOP-1533_1.patch
applied and successfully tested against trunk revision r551361,
but there appear to be new Findbugs warnings introduced by this patch.
New Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/341/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/341/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/341/console
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_1.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1533) Distcp should log to specified
location
Posted by "Senthil Subramanian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Senthil Subramanian updated HADOOP-1533:
----------------------------------------
Attachment: HADOOP-1533_2.patch
Based on Owen's comments, I have made the following changes:
- logPath is now clearly a Path
- logPath is now optional and can be specified with a -log switch. Default is "$output_dir/_distcp_logs
_System.currentTimeMillis()"
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_2.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1533) Distcp should log to specified
location
Posted by "Senthil Subramanian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Senthil Subramanian updated HADOOP-1533:
----------------------------------------
Attachment: (was: HADOOP-1533_1.patch)
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Fix For: 0.14.0
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1533) Distcp should log to specified
location
Posted by "Senthil Subramanian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Senthil Subramanian updated HADOOP-1533:
----------------------------------------
Status: Patch Available (was: Open)
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_1.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1533) Distcp should log to specified
location
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1533:
----------------------------------
Status: Open (was: Patch Available)
This patch seems confused whether the "log path" is a Path, URI, or a String. Since you are implementing it as the output of the Map (with reduces = 0), it pretty clearly must be a Path. It should be coded as such.
Instead of requiring a parameter for the log path, I think it should be an option like:
... - log hdfs://server:1234/my/log
and I'd propose that the default be: "$output_dir/_distcp_logs"
That will work well in most cases and keep the output/logs together.
The chunk of code that looks like:
+ String logURL = "hdfs://" + fsConfig.get("fs.default.name", "local") +
+ archiveDirectory + "distcpLogDir." + System.currentTimeMillis();
is problematic, since it won't construct useful urls in a wide variety of values for fs.default.name.
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_1.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-1533) Distcp should log to specified
location
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko reopened HADOOP-1533:
-----------------------------------------
This breaks Windows build. checkPath() fails.
Copy failed: java.lang.IllegalArgumentException: Wrong FS: file://C:/Work/nutch-0.7.1/Hadoop/build/test/data/srcdat, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:204)
at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:50)
at org.apache.hadoop.fs.RawLocalFileSystem.exists(RawLocalFileSystem.java:219)
at org.apache.hadoop.fs.FilterFileSystem.exists(FilterFileSystem.java:156)
at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:832)
at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
at org.apache.hadoop.fs.TestCopyFiles.testCopyFromLocalToLocal(TestCopyFiles.java:166)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at junit.framework.TestCase.runTest(TestCase.java:154)
at junit.framework.TestCase.runBare(TestCase.java:127)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:297)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:672)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:567)
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Assignee: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_2.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1533) Distcp should log to specified
location
Posted by "Senthil Subramanian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Senthil Subramanian updated HADOOP-1533:
----------------------------------------
Status: Patch Available (was: Open)
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_2.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1533) Distcp should log to specified
location
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting reassigned HADOOP-1533:
------------------------------------
Assignee: Senthil Subramanian
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Assignee: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_2.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1533) Distcp should log to specified
location
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12512051 ]
Hudson commented on HADOOP-1533:
--------------------------------
Integrated in Hadoop-Nightly #152 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/152/])
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Assignee: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_2.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1533) Distcp should log to specified
location
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12509739 ]
Hadoop QA commented on HADOOP-1533:
-----------------------------------
+1
http://issues.apache.org/jira/secure/attachment/12360981/HADOOP-1533_2.patch applied and successfully tested against trunk revision r552548.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/356/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/356/console
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_2.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1533) Distcp should log to specified
location
Posted by "Senthil Subramanian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Senthil Subramanian updated HADOOP-1533:
----------------------------------------
Status: Open (was: Patch Available)
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_1.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-1533) Distcp should log to specified
location
Posted by "Senthil Subramanian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Senthil Subramanian resolved HADOOP-1533.
-----------------------------------------
Resolution: Fixed
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Assignee: Senthil Subramanian
> Attachments: HADOOP-1533_2.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1533) Distcp should log to specified
location
Posted by "Senthil Subramanian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Senthil Subramanian updated HADOOP-1533:
----------------------------------------
Attachment: HADOOP-1533_1.patch
distcp now logs errors to specified logDir as outputs from map and Reduce=None
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_1.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1533) Distcp should log to specified
location
Posted by "Senthil Subramanian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Senthil Subramanian updated HADOOP-1533:
----------------------------------------
Status: Patch Available (was: Open)
> Distcp should log to specified location
> ---------------------------------------
>
> Key: HADOOP-1533
> URL: https://issues.apache.org/jira/browse/HADOOP-1533
> Project: Hadoop
> Issue Type: Improvement
> Components: util
> Reporter: Senthil Subramanian
> Fix For: 0.14.0
>
> Attachments: HADOOP-1533_1.patch
>
>
> Distcp errors are now not logged anywhere (other that reporter.setStatus which gets overwritten). Add support to mention a log URI where the log files should go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.