[jira] [Commented] (SQOOP-2832) Sqoop2: Precommit: Create log files for individual tests

["Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SQOOP-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142100#comment-15142100 ] 

Jarek Jarcec Cecho commented on SQOOP-2832:
-------------------------------------------

Here is my "poor's man solution" stab at the problem. It's well described in the patch, but for convenience:

{code}
 * Sqoop is running as much tests as possible inside one suite to safe time starting
 * miniclusters which is time consuming exercise (~40 seconds per single test class).
 * That however means that we have one output log file that recently grown to more then
 * 1GB in side and hence the usability has decreased.
 *
 * This listener will intercept each test and will reconfigure log4j to log directly
 * into files rather then to console (that would be forwarded by maven surefire plugin to
 * the normal log file). Each test will get it's own file which is easier for human to
 * read.
 *
 * We're using a counter to order log files per execution order rather then per name as
 * we can't guarantee log isolation entirely (e.g. some information relevant to test N
 * can be in log file for test N-1). It's easier to open previous log if you immediately
 * know what is the previous log.
{code}

I'm open to any other proposals - I just didn't find a different solution other then run one suite per test class which seems undesirable. I've put the new listener only to one suite for the time being to see how it will work on our jenkins infrastructure first.

> Sqoop2: Precommit: Create log files for individual tests
> --------------------------------------------------------
>
>                 Key: SQOOP-2832
>                 URL: https://issues.apache.org/jira/browse/SQOOP-2832
>             Project: Sqoop
>          Issue Type: Bug
>            Reporter: Jarek Jarcec Cecho
>            Assignee: Jarek Jarcec Cecho
>             Fix For: 1.99.7
>
>         Attachments: SQOOP-2832.patch
>
>
> I feel that debugging anything in our pre-commit hook is currently mission impossible from several reasons. One of the main ones is that we're generating one single log file that has [grown to 1GB in size|https://builds.apache.org/job/PreCommit-SQOOP-Build/2185/artifact/test/target/surefire-reports/] with all "test related" logs and I'm not even remotely able to find anything there.
> In normal case we would have one log per test class, but as we've refactored our integration tests to run multiple classes inside single Suite, we're no longer have that for free. We have done this change to cut the time it takes to run the integration tests - before we were initializing mini clusters for each class, which is extra ~40 seconds per class, so I don't think that reverting it would be reasonable.
> We should perhaps explore other ways how to get multiple log files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)