You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Aaron Morton (JIRA)" <ji...@apache.org> on 2011/08/19 06:16:27 UTC

[jira] [Created] (CASSANDRA-3061) Optionally skip log4j configuration

Optionally skip log4j configuration
-----------------------------------

                 Key: CASSANDRA-3061
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061
             Project: Cassandra
          Issue Type: Improvement
          Components: Core
    Affects Versions: 0.8.4
            Reporter: Aaron Morton
            Assignee: Aaron Morton
            Priority: Minor


from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8

When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. 

log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. 

Two examples...
http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html
http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html

Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. 

We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end.  

As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file...

2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties
2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally
2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
 
The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows 

<property>
  <name>mapred.child.java.opts</name>
  <value>-Xmx256m -Dcassandra.log4j.configure=false</value>
  <description>
    Tune your mapred jvm arguments for best performance. 
    Also see documentation from jvm vendor.
  </description>
</property>

It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone.  


Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages 
https://issues.apache.org/jira/browse/HADOOP-7556

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3061) Optionally skip log4j configuration

Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088463#comment-13088463 ] 

T Jake Luciani commented on CASSANDRA-3061:
-------------------------------------------

Yeah this is a pain.

I think it would be easier to check for log4j.defaultInitOverride=true (we set it in bin/cassandra) before we apply our log4j setup.  That way in hadoop it would not be loaded by default.


> Optionally skip log4j configuration
> -----------------------------------
>
>                 Key: CASSANDRA-3061
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>            Priority: Minor
>             Fix For: 0.8.5
>
>         Attachments: 0001-3061.patch
>
>
> from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8
> When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. 
> log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. 
> Two examples...
> http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html
> http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html
> Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. 
> We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end.  
> As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file...
> 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties
> 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally
> 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
>  
> The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows 
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx256m -Dcassandra.log4j.configure=false</value>
>   <description>
>     Tune your mapred jvm arguments for best performance. 
>     Also see documentation from jvm vendor.
>   </description>
> </property>
> It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone.  
> Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages 
> https://issues.apache.org/jira/browse/HADOOP-7556

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-3061) Optionally skip log4j configuration

Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

T Jake Luciani updated CASSANDRA-3061:
--------------------------------------

    Attachment: 3061_v2.txt

version that checks iflog4j.defaultInitOverride=true before executing our custom setup

> Optionally skip log4j configuration
> -----------------------------------
>
>                 Key: CASSANDRA-3061
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>            Priority: Minor
>             Fix For: 0.8.5
>
>         Attachments: 0001-3061.patch, 3061_v2.txt
>
>
> from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8
> When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. 
> log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. 
> Two examples...
> http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html
> http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html
> Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. 
> We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end.  
> As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file...
> 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties
> 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally
> 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
>  
> The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows 
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx256m -Dcassandra.log4j.configure=false</value>
>   <description>
>     Tune your mapred jvm arguments for best performance. 
>     Also see documentation from jvm vendor.
>   </description>
> </property>
> It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone.  
> Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages 
> https://issues.apache.org/jira/browse/HADOOP-7556

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3061) Optionally skip log4j configuration

Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092886#comment-13092886 ] 

T Jake Luciani commented on CASSANDRA-3061:
-------------------------------------------

Aaron any update here? this is working fine for me.

> Optionally skip log4j configuration
> -----------------------------------
>
>                 Key: CASSANDRA-3061
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>            Priority: Minor
>             Fix For: 0.8.5
>
>         Attachments: 0001-3061.patch, 3061_v2.txt
>
>
> from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8
> When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. 
> log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. 
> Two examples...
> http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html
> http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html
> Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. 
> We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end.  
> As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file...
> 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties
> 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally
> 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
>  
> The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows 
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx256m -Dcassandra.log4j.configure=false</value>
>   <description>
>     Tune your mapred jvm arguments for best performance. 
>     Also see documentation from jvm vendor.
>   </description>
> </property>
> It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone.  
> Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages 
> https://issues.apache.org/jira/browse/HADOOP-7556

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-3061) Optionally skip log4j configuration

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Morton updated CASSANDRA-3061:
------------------------------------

    Attachment: 0001-3061.patch

> Optionally skip log4j configuration
> -----------------------------------
>
>                 Key: CASSANDRA-3061
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>            Priority: Minor
>         Attachments: 0001-3061.patch
>
>
> from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8
> When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. 
> log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. 
> Two examples...
> http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html
> http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html
> Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. 
> We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end.  
> As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file...
> 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties
> 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally
> 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
>  
> The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows 
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx256m -Dcassandra.log4j.configure=false</value>
>   <description>
>     Tune your mapred jvm arguments for best performance. 
>     Also see documentation from jvm vendor.
>   </description>
> </property>
> It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone.  
> Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages 
> https://issues.apache.org/jira/browse/HADOOP-7556

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3061) Optionally skip log4j configuration

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094237#comment-13094237 ] 

Aaron Morton commented on CASSANDRA-3061:
-----------------------------------------

I tried to run it with the current cassandra-0.8 branch and brisk beta 2 but ran into this problem http://groups.google.com/group/brisk-users/browse_thread/thread/75c9f39d4c1859a9

What I can see looks good to me though. 

> Optionally skip log4j configuration
> -----------------------------------
>
>                 Key: CASSANDRA-3061
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>            Priority: Minor
>             Fix For: 0.8.5
>
>         Attachments: 0001-3061.patch, 3061_v2.txt
>
>
> from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8
> When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. 
> log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. 
> Two examples...
> http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html
> http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html
> Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. 
> We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end.  
> As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file...
> 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties
> 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally
> 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
>  
> The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows 
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx256m -Dcassandra.log4j.configure=false</value>
>   <description>
>     Tune your mapred jvm arguments for best performance. 
>     Also see documentation from jvm vendor.
>   </description>
> </property>
> It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone.  
> Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages 
> https://issues.apache.org/jira/browse/HADOOP-7556

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3061) Optionally skip log4j configuration

Posted by "Tomas Salfischberger (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094334#comment-13094334 ] 

Tomas Salfischberger commented on CASSANDRA-3061:
-------------------------------------------------

For me the log4j problem is solved with this patch

I ran into the same problem as Aaron did (which is fixed in CASSANDRA-3044), but worked around it by attaching a debugger and manually changing the endpoint to be correct. In Brisk there is another issue, because rpc_address is set to "0.0.0.0" which is not a valid endpoint to connect to. Also posted that here: http://www.datastax.com/support-forums/topic/building-brisk-from-source-to-update-cassandra-to-082?replies=1#post-409


> Optionally skip log4j configuration
> -----------------------------------
>
>                 Key: CASSANDRA-3061
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>            Priority: Minor
>             Fix For: 0.8.5
>
>         Attachments: 0001-3061.patch, 3061_v2.txt
>
>
> from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8
> When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. 
> log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. 
> Two examples...
> http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html
> http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html
> Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. 
> We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end.  
> As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file...
> 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties
> 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally
> 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
>  
> The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows 
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx256m -Dcassandra.log4j.configure=false</value>
>   <description>
>     Tune your mapred jvm arguments for best performance. 
>     Also see documentation from jvm vendor.
>   </description>
> </property>
> It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone.  
> Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages 
> https://issues.apache.org/jira/browse/HADOOP-7556

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-3061) Optionally skip log4j configuration

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-3061:
--------------------------------------

         Reviewer: tjake
    Fix Version/s: 0.8.5

> Optionally skip log4j configuration
> -----------------------------------
>
>                 Key: CASSANDRA-3061
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>            Priority: Minor
>             Fix For: 0.8.5
>
>         Attachments: 0001-3061.patch
>
>
> from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8
> When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. 
> log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. 
> Two examples...
> http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html
> http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html
> Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. 
> We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end.  
> As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file...
> 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties
> 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally
> 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
>  
> The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows 
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx256m -Dcassandra.log4j.configure=false</value>
>   <description>
>     Tune your mapred jvm arguments for best performance. 
>     Also see documentation from jvm vendor.
>   </description>
> </property>
> It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone.  
> Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages 
> https://issues.apache.org/jira/browse/HADOOP-7556

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3061) Optionally skip log4j configuration

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094823#comment-13094823 ] 

Hudson commented on CASSANDRA-3061:
-----------------------------------

Integrated in Cassandra-0.8 #307 (See [https://builds.apache.org/job/Cassandra-0.8/307/])
    Optionally skip log4j configuration
Patch by tjake; reviewed by Aaron Morton for CASSANDRA-3061

jake : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163736
Files : 
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/AbstractCassandraDaemon.java


> Optionally skip log4j configuration
> -----------------------------------
>
>                 Key: CASSANDRA-3061
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Aaron Morton
>            Assignee: T Jake Luciani
>            Priority: Minor
>             Fix For: 0.8.5
>
>         Attachments: 0001-3061.patch, 3061_v2.txt
>
>
> from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8
> When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. 
> log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. 
> Two examples...
> http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html
> http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html
> Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. 
> We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end.  
> As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file...
> 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties
> 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally
> 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
>  
> The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows 
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx256m -Dcassandra.log4j.configure=false</value>
>   <description>
>     Tune your mapred jvm arguments for best performance. 
>     Also see documentation from jvm vendor.
>   </description>
> </property>
> It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone.  
> Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages 
> https://issues.apache.org/jira/browse/HADOOP-7556

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3061) Optionally skip log4j configuration

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089090#comment-13089090 ] 

Aaron Morton commented on CASSANDRA-3061:
-----------------------------------------

Thanks, will try it out in the next few days. 

> Optionally skip log4j configuration
> -----------------------------------
>
>                 Key: CASSANDRA-3061
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>            Priority: Minor
>             Fix For: 0.8.5
>
>         Attachments: 0001-3061.patch, 3061_v2.txt
>
>
> from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8
> When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. 
> log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. 
> Two examples...
> http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html
> http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html
> Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. 
> We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end.  
> As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file...
> 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties
> 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally
> 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
>  
> The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows 
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx256m -Dcassandra.log4j.configure=false</value>
>   <description>
>     Tune your mapred jvm arguments for best performance. 
>     Also see documentation from jvm vendor.
>   </description>
> </property>
> It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone.  
> Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages 
> https://issues.apache.org/jira/browse/HADOOP-7556

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira