You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ravi Gummadi (JIRA)" <ji...@apache.org> on 2009/10/09 10:20:31 UTC

[jira] Created: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

hadoop commands in streaming tasks are trying to write to tasktracker's log
---------------------------------------------------------------------------

                 Key: MAPREDUCE-1086
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: tasktracker
    Affects Versions: 0.20.1, 0.21.0, 0.22.0
            Reporter: Ravi Gummadi
            Assignee: Ravi Gummadi
             Fix For: 0.20.2


As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.

When streaming calls hadoop commands, it's trying to write to TaskTracker log file.

log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException:
/grid/0/hadoop/var/log/mapred/hadoop-mapred-tasktracker-gsbl60968.blue.ygrid.yahoo.com.log (Permission denied)
        at java.io.FileOutputStream.openAppend(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
        at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
        at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
        at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
        at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
        at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
        at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
        at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
        at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
        at org.apache.log4j.Logger.getLogger(Logger.java:105)
        at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
        at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
        at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated MAPREDUCE-1086:
------------------------------------

    Attachment: MR-1086-yhadoop20.patch

Attaching patch for yahoo-hadoop-0.20 branch.
ant test and ant test-patch passed on my local machine.

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: MR-1086-yhadoop20.patch, MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.2.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated MAPREDUCE-1086:
------------------------------------

    Attachment: MR-1086.v1.patch

Attaching new patch with the suggested changes.

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>         Attachments: MR-1086.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated MAPREDUCE-1086:
------------------------------------

    Release Note: This patch makes TT to set HADOOP_ROOT_LOGGER to INFO,TLA by default in the environment of taskjvm and its children.

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: MR-1086-yhadoop20.patch, MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.2.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766056#action_12766056 ] 

Hemanth Yamijala commented on MAPREDUCE-1086:
---------------------------------------------

Ravi, this looks mostly good.

I have very minor comments:

- I am thinking it makes sense to set the log parameters in HADOOP_CLIENT_OPTS than HADOOP_OPTS. This will make sure they get appended at the end and is more reliable. Further, I think we can assume that the streaming tasks only use the hadoop command line as a client.
- To be consistent with what is set in the scripts of Hadoop, I would recommend that the value of HADOOP_CLIENT_OPTS is enclosed in double quotes than single.
- We really don't need to convert taskId or log file size to Strings, as Java will do the conversion itself.
- In the test case, I suppose we can be a little more strict in verification. Can we check that the env contains 'INFO,TLA' as that's what we are setting. Likewise, we can also verify the log file size is set to a known value by setting it in the corresponding property in the Configuration.

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>         Attachments: MR-1086.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated MAPREDUCE-1086:
------------------------------------

    Attachment: MR-1086.patch

Attaching patch for trunk.

Please review and provide your comments.

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>         Attachments: MR-1086.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767843#action_12767843 ] 

Hudson commented on MAPREDUCE-1086:
-----------------------------------

Integrated in Hadoop-Mapreduce-trunk #118 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/118/])
    . Setup Hadoop logging environment for tasks to point to task related parameters. Contributed by Ravi Gummadi.


> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: MR-1086-yhadoop20.patch, MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.2.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767755#action_12767755 ] 

Hudson commented on MAPREDUCE-1086:
-----------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #87 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/87/])
    . Setup Hadoop logging environment for tasks to point to task related parameters. Contributed by Ravi Gummadi.


> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.2.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated MAPREDUCE-1086:
----------------------------------------

    Status: Open  (was: Patch Available)

Canceling patch to fix the test failure. The test passes locally on my machine as well. But I suspect the reason it is failing on Hudson *may* be due to the fact that the streaming commands are written without specifying a shell. Naturally, it would be useful to add some log statements, at a minimum to print the output of the streaming job.

Another issue could be that we are using getBytes() to convert from String to bytes and writing the command to a file. Why not use the more standard BufferedWriter ?

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>         Attachments: MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated MAPREDUCE-1086:
------------------------------------

    Status: Patch Available  (was: Open)

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>         Attachments: MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated MAPREDUCE-1086:
----------------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 0.20.2)
                   0.21.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I just committed this to trunk and branch 0.21. Thanks, Ravi !

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.2.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated MAPREDUCE-1086:
------------------------------------

    Attachment:     (was: MR-1086-yhadoop20.patch)

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.2.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated MAPREDUCE-1086:
------------------------------------

    Attachment: MR-1086.v1.2.patch

Test failure on Hudson was because of the shell specific   statement  " cat >& /dev/null ". Modified it to " cat > /dev/null 2>&1" in this new patch.

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>         Attachments: MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.2.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766436#action_12766436 ] 

Hemanth Yamijala commented on MAPREDUCE-1086:
---------------------------------------------

Seems fine. Though not really related to this patch, the place where the localFS is being initialized can avoid a catch and ignore of the IOException, as it is anyway going to die with a NullPointerException later. Since this is indeed an exceptional condition, I would rather we throw the IOException. Looking at code paths, that actually seems to be handled better in the child tasks than NullPointerException is.

Also can you update the documentation of the getVMEnvironment API to include the new parameters ?

Other than these changes +1. Please make the new patch with these changes Patch Available.

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>         Attachments: MR-1086.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767683#action_12767683 ] 

Hemanth Yamijala commented on MAPREDUCE-1086:
---------------------------------------------

The changes are fine. Appears my diagnosis for the test failure was not correct. Thanks for fixing the correct bug.

+1.

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>         Attachments: MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.2.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766483#action_12766483 ] 

Hadoop QA commented on MAPREDUCE-1086:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12422326/MR-1086.v1.1.patch
  against trunk revision 825469.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    -1 contrib tests.  The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/176/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/176/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/176/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/176/console

This message is automatically generated.

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>         Attachments: MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767818#action_12767818 ] 

Hemanth Yamijala commented on MAPREDUCE-1086:
---------------------------------------------

Verified the 20 patch. Looks fine.

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: MR-1086-yhadoop20.patch, MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.2.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767518#action_12767518 ] 

Hadoop QA commented on MAPREDUCE-1086:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12422569/MR-1086.v1.2.patch
  against trunk revision 826635.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/186/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/186/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/186/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/186/console

This message is automatically generated.

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>         Attachments: MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.2.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763908#action_12763908 ] 

Ravi Gummadi commented on MAPREDUCE-1086:
-----------------------------------------

This issue is only with LinuxTaskController.

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /grid/0/hadoop/var/log/mapred/hadoop-mapred-tasktracker-gsbl60968.blue.ygrid.yahoo.com.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated MAPREDUCE-1086:
------------------------------------

    Attachment: MR-1086.v1.1.patch

Attaching new patch with the suggested changes.

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>         Attachments: MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated MAPREDUCE-1086:
------------------------------------

    Attachment: MR-1086-yhadoop20.patch

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: MR-1086-yhadoop20.patch, MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.2.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated MAPREDUCE-1086:
----------------------------------------

    Description: 
As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.

When streaming calls hadoop commands, it's trying to write to TaskTracker log file.

log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException:
/a/b/tasktracker.log (Permission denied)
        at java.io.FileOutputStream.openAppend(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
        at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
        at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
        at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
        at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
        at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
        at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
        at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
        at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
        at org.apache.log4j.Logger.getLogger(Logger.java:105)
        at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
        at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
        at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

  was:
As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.

When streaming calls hadoop commands, it's trying to write to TaskTracker log file.

log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException:
/grid/0/hadoop/var/log/mapred/hadoop-mapred-tasktracker-gsbl60968.blue.ygrid.yahoo.com.log (Permission denied)
        at java.io.FileOutputStream.openAppend(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
        at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
        at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
        at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
        at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
        at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
        at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
        at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
        at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
        at org.apache.log4j.Logger.getLogger(Logger.java:105)
        at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
        at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
        at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].


> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1086) hadoop commands in streaming tasks are trying to write to tasktracker's log

Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated MAPREDUCE-1086:
------------------------------------

    Status: Patch Available  (was: Open)

Letting the new patch go through Hudson...

> hadoop commands in streaming tasks are trying to write to tasktracker's log
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1086
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.2
>
>         Attachments: MR-1086.patch, MR-1086.v1.1.patch, MR-1086.v1.2.patch, MR-1086.v1.patch
>
>
> As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.
> When streaming calls hadoop commands, it's trying to write to TaskTracker log file.
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException:
> /a/b/tasktracker.log (Permission denied)
>         at java.io.FileOutputStream.openAppend(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
>         at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
>         at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
>         at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
>         at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>         at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>         at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>         at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>         at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
>         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
>         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
>         at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>         at org.apache.log4j.Logger.getLogger(Logger.java:105)
>         at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>         at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
>         at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
>         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>         at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
> log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.