[jira] [Commented] (SQOOP-2834) Sqoop2: Integration: Limit debug log to only classes that we're interested in

["Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SQOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143546#comment-15143546 ] 

Jarek Jarcec Cecho commented on SQOOP-2834:
-------------------------------------------

I tried to set various loggers to {{INFO}} rather then {{DEBUG}} while not ignoring anything that seemed important. Whereas I'm sure that we can set more loggers to {{INFO}}, I'm fairly happy with the size I was able to get to. Here are results from my experiments when changing log levels and running {{AppendModeTest}}:

# Originating size is 30 MB
# Setting {{org.eclipse.jetty}} got us to 17 MB
# Setting {{org.apache.directory}} got us to 15 MB
# Setting {{org.apache.hadoop.ipc.Server}} got us to 14 MB
# Setting {{org.apache.hadoop.hdfs.DFSClient}} got us to 12 MB
# Setting {{org.apache.hadoop.hdfs.server.datanode.DataNode}} got us to 11 MB
# Setting {{org.apache.hadoop.security.SaslInputStream}} got us to 9.1MB
# Setting {{org.apache.hadoop.security.SaslRpcClient}} got us to 8.4 MB
# Setting {{org.apache.hadoop.ipc.Client}} got us to 7.3 MB
# Setting (rest of) {{org.apache.hadoop.hdfs}} got us to 6.9MB
# Setting {{org.apache.hadoop.conf.Configuration}} got us to 6.2MB

> Sqoop2: Integration: Limit debug log to only classes that we're interested in
> -----------------------------------------------------------------------------
>
>                 Key: SQOOP-2834
>                 URL: https://issues.apache.org/jira/browse/SQOOP-2834
>             Project: Sqoop
>          Issue Type: Bug
>            Reporter: Jarek Jarcec Cecho
>            Assignee: Jarek Jarcec Cecho
>             Fix For: 1.99.7
>
>
> To follow up on my recent patches that are helping with "debuggability" of pre-commit hook and our integration tests in general, I've looked into what exactly are we logging that size of our logs is more then 1GB per execution.
> Here is what I've done:
> # I've applied my patch from SQOOP-2832 to get log for one test only
> # I've run a magic that gives me classes that are responsible for logging:
> {code}
> cat test/target/surefire-reports/00000_org.apache.sqoop.integration.connector.hdfs.AppendModeTest.test.txt | sed -re "s/^.*\] ([A-Z]+)[ ]+([A-Za-z.]+) .*$/\1 \2/" | sort | uniq -c | sort -r > report
> {code}
> With a top results being:
> {code}
> 6927 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction
> 5783 DEBUG org.apache.hadoop.ipc.Client
> 5752 DEBUG org.apache.sqoop.repository.common.CommonRepositoryHandler
> 5750 DEBUG org.apache.hadoop.hdfs.DFSClient
> 4784 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode
> 4715 DEBUG org.eclipse.jetty.io.SelectorManager
> 4660 DEBUG org.eclipse.jetty.server.HttpConnection
> 4306 DEBUG org.apache.hadoop.security.UserGroupInformation
> 3489 DEBUG org.eclipse.jetty.io.WriteFlusher
> 2927 DEBUG org.eclipse.jetty.io.ChannelEndPoint
> 2846 DEBUG org.apache.hadoop.conf.Configuration
> 2830 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine
> 2357 DEBUG org.eclipse.jetty.io.AbstractConnection
> 2350 DEBUG org.eclipse.jetty.io.SelectChannelEndPoint
> 2343 DEBUG org.eclipse.jetty.server.HttpChannel
> 2332 DEBUG org.eclipse.jetty.servlet.ServletHandler
> 2309 INFO org.apache.sqoop.repository.JdbcRepositoryTransaction
> 16701 DEBUG org.apache.hadoop.security.SaslInputStream
> 14613 DEBUG org.eclipse.jetty.http.HttpParser
> 1426 
> 1175 DEBUG org.apache.sqoop.security.authorization.DefaultAuthorizationValidator
> 1168 DEBUG org.eclipse.jetty.server.handler.ContextHandler
> 1168 DEBUG org.eclipse.jetty.server.Server
> 1168 DEBUG org.eclipse.jetty.server.HttpChannelState
> 1034 DEBUG org.apache.hadoop.yarn.server.security.ApplicationACLsManager
> 10329 DEBUG org.apache.hadoop.ipc.Server
> {code}
> Based on that I would like to reconfigure certain classes to limit their logging to levels higher then {{DEBUG}} - jetty seems as a no-brainer and Hadoop ipc might be another good candidate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)