You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Jarek Cecho <ja...@apache.org> on 2016/02/11 02:25:02 UTC
Review Request 43467: SQOOP-2832: Sqoop2: Precommit: Create log files
for individual tests
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43467/
-----------------------------------------------------------
Review request for Sqoop and Jarek Cecho.
Bugs: SQOOP-2832
https://issues.apache.org/jira/browse/SQOOP-2832
Repository: sqoop-sqoop2
Description
-------
I feel that debugging anything in our pre-commit hook is currently mission impossible from several reasons. One of the main ones is that we're generating one single log file that has [grown to 1GB in size|https://builds.apache.org/job/PreCommit-SQOOP-Build/2185/artifact/test/target/surefire-reports/] with all "test related" logs and I'm not even remotely able to find anything there.
In normal case we would have one log per test class, but as we've refactored our integration tests to run multiple classes inside single Suite, we're no longer have that for free. We have done this change to cut the time it takes to run the integration tests - before we were initializing mini clusters for each class, which is extra ~40 seconds per class, so I don't think that reverting it would be reasonable.
We should perhaps explore other ways how to get multiple log files.
Diffs
-----
test/pom.xml 134bca1
test/src/main/java/org/apache/sqoop/test/testng/ReconfigureLogListener.java PRE-CREATION
test/src/test/resources/integration-tests-suite.xml 73e0a77
Diff: https://reviews.apache.org/r/43467/diff/
Testing
-------
Thanks,
Jarek Cecho
Re: Review Request 43467: SQOOP-2832: Sqoop2: Precommit: Create log
files for individual tests
Posted by Jarek Cecho <ja...@apache.org>.
> On Feb. 17, 2016, 3:42 a.m., Colin Ma wrote:
> > test/src/test/resources/integration-tests-suite.xml, line 25
> > <https://reviews.apache.org/r/43467/diff/2/?file=1240433#file1240433line25>
> >
> > How about add the listener for all suite like: connector-loading-tests-suite.xml and new-integration-tests-suite.xml, etc.
Yes, I wanted to get feedback first :) Let me do that real quick.
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43467/#review119407
-----------------------------------------------------------
On Feb. 17, 2016, 4:12 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43467/
> -----------------------------------------------------------
>
> (Updated Feb. 17, 2016, 4:12 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-2832
> https://issues.apache.org/jira/browse/SQOOP-2832
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I feel that debugging anything in our pre-commit hook is currently mission impossible from several reasons. One of the main ones is that we're generating one single log file that has [grown to 1GB in size|https://builds.apache.org/job/PreCommit-SQOOP-Build/2185/artifact/test/target/surefire-reports/] with all "test related" logs and I'm not even remotely able to find anything there.
>
> In normal case we would have one log per test class, but as we've refactored our integration tests to run multiple classes inside single Suite, we're no longer have that for free. We have done this change to cut the time it takes to run the integration tests - before we were initializing mini clusters for each class, which is extra ~40 seconds per class, so I don't think that reverting it would be reasonable.
>
> We should perhaps explore other ways how to get multiple log files.
>
>
> Diffs
> -----
>
> test/pom.xml 134bca1
> test/src/main/java/org/apache/sqoop/test/testng/ReconfigureLogListener.java PRE-CREATION
> test/src/test/resources/connector-loading-tests-suite.xml 02c1df3
> test/src/test/resources/hive-tests-suite.xml 3e2b328
> test/src/test/resources/integration-tests-suite.xml 73e0a77
> test/src/test/resources/new-integration-tests-suite.xml 96a1320
> test/src/test/resources/shell-tests-suite.xml e2c0b09
> test/src/test/resources/tools-tests-suite.xml fee2121
> test/src/test/resources/upgrade-tests-suite.xml 3318e67
>
> Diff: https://reviews.apache.org/r/43467/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 43467: SQOOP-2832: Sqoop2: Precommit: Create log
files for individual tests
Posted by Colin Ma <ju...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43467/#review119407
-----------------------------------------------------------
test/src/test/resources/integration-tests-suite.xml (line 25)
<https://reviews.apache.org/r/43467/#comment180762>
How about add the listener for all suite like: connector-loading-tests-suite.xml and new-integration-tests-suite.xml, etc.
- Colin Ma
On Feb. 11, 2016, 9:29 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43467/
> -----------------------------------------------------------
>
> (Updated Feb. 11, 2016, 9:29 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-2832
> https://issues.apache.org/jira/browse/SQOOP-2832
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I feel that debugging anything in our pre-commit hook is currently mission impossible from several reasons. One of the main ones is that we're generating one single log file that has [grown to 1GB in size|https://builds.apache.org/job/PreCommit-SQOOP-Build/2185/artifact/test/target/surefire-reports/] with all "test related" logs and I'm not even remotely able to find anything there.
>
> In normal case we would have one log per test class, but as we've refactored our integration tests to run multiple classes inside single Suite, we're no longer have that for free. We have done this change to cut the time it takes to run the integration tests - before we were initializing mini clusters for each class, which is extra ~40 seconds per class, so I don't think that reverting it would be reasonable.
>
> We should perhaps explore other ways how to get multiple log files.
>
>
> Diffs
> -----
>
> test/pom.xml 134bca1
> test/src/main/java/org/apache/sqoop/test/testng/ReconfigureLogListener.java PRE-CREATION
> test/src/test/resources/integration-tests-suite.xml 73e0a77
>
> Diff: https://reviews.apache.org/r/43467/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 43467: SQOOP-2832: Sqoop2: Precommit: Create log
files for individual tests
Posted by Colin Ma <ju...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43467/#review120083
-----------------------------------------------------------
Ship it!
Ship It!
- Colin Ma
On Feb. 19, 2016, 3:32 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43467/
> -----------------------------------------------------------
>
> (Updated Feb. 19, 2016, 3:32 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-2832
> https://issues.apache.org/jira/browse/SQOOP-2832
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I feel that debugging anything in our pre-commit hook is currently mission impossible from several reasons. One of the main ones is that we're generating one single log file that has [grown to 1GB in size|https://builds.apache.org/job/PreCommit-SQOOP-Build/2185/artifact/test/target/surefire-reports/] with all "test related" logs and I'm not even remotely able to find anything there.
>
> In normal case we would have one log per test class, but as we've refactored our integration tests to run multiple classes inside single Suite, we're no longer have that for free. We have done this change to cut the time it takes to run the integration tests - before we were initializing mini clusters for each class, which is extra ~40 seconds per class, so I don't think that reverting it would be reasonable.
>
> We should perhaps explore other ways how to get multiple log files.
>
>
> Diffs
> -----
>
> test/pom.xml 134bca1
> test/src/main/java/org/apache/sqoop/test/testng/ReconfigureLogListener.java PRE-CREATION
> test/src/test/resources/connector-loading-tests-suite.xml 02c1df3
> test/src/test/resources/hive-tests-suite.xml 3e2b328
> test/src/test/resources/integration-tests-suite.xml 73e0a77
> test/src/test/resources/new-integration-tests-suite.xml 96a1320
> test/src/test/resources/shell-tests-suite.xml e2c0b09
> test/src/test/resources/tools-tests-suite.xml fee2121
> test/src/test/resources/upgrade-tests-suite.xml 3318e67
>
> Diff: https://reviews.apache.org/r/43467/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 43467: SQOOP-2832: Sqoop2: Precommit: Create log
files for individual tests
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43467/
-----------------------------------------------------------
(Updated Feb. 19, 2016, 3:32 p.m.)
Review request for Sqoop and Jarek Cecho.
Bugs: SQOOP-2832
https://issues.apache.org/jira/browse/SQOOP-2832
Repository: sqoop-sqoop2
Description
-------
I feel that debugging anything in our pre-commit hook is currently mission impossible from several reasons. One of the main ones is that we're generating one single log file that has [grown to 1GB in size|https://builds.apache.org/job/PreCommit-SQOOP-Build/2185/artifact/test/target/surefire-reports/] with all "test related" logs and I'm not even remotely able to find anything there.
In normal case we would have one log per test class, but as we've refactored our integration tests to run multiple classes inside single Suite, we're no longer have that for free. We have done this change to cut the time it takes to run the integration tests - before we were initializing mini clusters for each class, which is extra ~40 seconds per class, so I don't think that reverting it would be reasonable.
We should perhaps explore other ways how to get multiple log files.
Diffs (updated)
-----
test/pom.xml 134bca1
test/src/main/java/org/apache/sqoop/test/testng/ReconfigureLogListener.java PRE-CREATION
test/src/test/resources/connector-loading-tests-suite.xml 02c1df3
test/src/test/resources/hive-tests-suite.xml 3e2b328
test/src/test/resources/integration-tests-suite.xml 73e0a77
test/src/test/resources/new-integration-tests-suite.xml 96a1320
test/src/test/resources/shell-tests-suite.xml e2c0b09
test/src/test/resources/tools-tests-suite.xml fee2121
test/src/test/resources/upgrade-tests-suite.xml 3318e67
Diff: https://reviews.apache.org/r/43467/diff/
Testing
-------
Thanks,
Jarek Cecho
Re: Review Request 43467: SQOOP-2832: Sqoop2: Precommit: Create log
files for individual tests
Posted by Jarek Cecho <ja...@apache.org>.
> On Feb. 18, 2016, 1:53 a.m., Colin Ma wrote:
> > test/src/main/java/org/apache/sqoop/test/testng/ReconfigureLogListener.java, line 99
> > <https://reviews.apache.org/r/43467/diff/3/?file=1252713#file1252713line99>
> >
> > Write a static field in non-static method cause the findbugs warning.
> > How about use System.currentTimeMillis() as the prefix?
I prefer the counter as this way we can keep the size of the number relatively small, whereas with timestimap it would have to be long number. I've added exception for ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD to the listener, so that findbugs doesn't worry about it anymore.
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43467/#review119569
-----------------------------------------------------------
On Feb. 19, 2016, 3:32 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43467/
> -----------------------------------------------------------
>
> (Updated Feb. 19, 2016, 3:32 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-2832
> https://issues.apache.org/jira/browse/SQOOP-2832
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I feel that debugging anything in our pre-commit hook is currently mission impossible from several reasons. One of the main ones is that we're generating one single log file that has [grown to 1GB in size|https://builds.apache.org/job/PreCommit-SQOOP-Build/2185/artifact/test/target/surefire-reports/] with all "test related" logs and I'm not even remotely able to find anything there.
>
> In normal case we would have one log per test class, but as we've refactored our integration tests to run multiple classes inside single Suite, we're no longer have that for free. We have done this change to cut the time it takes to run the integration tests - before we were initializing mini clusters for each class, which is extra ~40 seconds per class, so I don't think that reverting it would be reasonable.
>
> We should perhaps explore other ways how to get multiple log files.
>
>
> Diffs
> -----
>
> test/pom.xml 134bca1
> test/src/main/java/org/apache/sqoop/test/testng/ReconfigureLogListener.java PRE-CREATION
> test/src/test/resources/connector-loading-tests-suite.xml 02c1df3
> test/src/test/resources/hive-tests-suite.xml 3e2b328
> test/src/test/resources/integration-tests-suite.xml 73e0a77
> test/src/test/resources/new-integration-tests-suite.xml 96a1320
> test/src/test/resources/shell-tests-suite.xml e2c0b09
> test/src/test/resources/tools-tests-suite.xml fee2121
> test/src/test/resources/upgrade-tests-suite.xml 3318e67
>
> Diff: https://reviews.apache.org/r/43467/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 43467: SQOOP-2832: Sqoop2: Precommit: Create log
files for individual tests
Posted by Colin Ma <ju...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43467/#review119569
-----------------------------------------------------------
test/src/main/java/org/apache/sqoop/test/testng/ReconfigureLogListener.java (line 99)
<https://reviews.apache.org/r/43467/#comment180917>
Write a static field in non-static method cause the findbugs warning.
How about use System.currentTimeMillis() as the prefix?
- Colin Ma
On Feb. 17, 2016, 4:12 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43467/
> -----------------------------------------------------------
>
> (Updated Feb. 17, 2016, 4:12 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-2832
> https://issues.apache.org/jira/browse/SQOOP-2832
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I feel that debugging anything in our pre-commit hook is currently mission impossible from several reasons. One of the main ones is that we're generating one single log file that has [grown to 1GB in size|https://builds.apache.org/job/PreCommit-SQOOP-Build/2185/artifact/test/target/surefire-reports/] with all "test related" logs and I'm not even remotely able to find anything there.
>
> In normal case we would have one log per test class, but as we've refactored our integration tests to run multiple classes inside single Suite, we're no longer have that for free. We have done this change to cut the time it takes to run the integration tests - before we were initializing mini clusters for each class, which is extra ~40 seconds per class, so I don't think that reverting it would be reasonable.
>
> We should perhaps explore other ways how to get multiple log files.
>
>
> Diffs
> -----
>
> test/pom.xml 134bca1
> test/src/main/java/org/apache/sqoop/test/testng/ReconfigureLogListener.java PRE-CREATION
> test/src/test/resources/connector-loading-tests-suite.xml 02c1df3
> test/src/test/resources/hive-tests-suite.xml 3e2b328
> test/src/test/resources/integration-tests-suite.xml 73e0a77
> test/src/test/resources/new-integration-tests-suite.xml 96a1320
> test/src/test/resources/shell-tests-suite.xml e2c0b09
> test/src/test/resources/tools-tests-suite.xml fee2121
> test/src/test/resources/upgrade-tests-suite.xml 3318e67
>
> Diff: https://reviews.apache.org/r/43467/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 43467: SQOOP-2832: Sqoop2: Precommit: Create log
files for individual tests
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43467/
-----------------------------------------------------------
(Updated Feb. 17, 2016, 4:12 p.m.)
Review request for Sqoop and Jarek Cecho.
Bugs: SQOOP-2832
https://issues.apache.org/jira/browse/SQOOP-2832
Repository: sqoop-sqoop2
Description
-------
I feel that debugging anything in our pre-commit hook is currently mission impossible from several reasons. One of the main ones is that we're generating one single log file that has [grown to 1GB in size|https://builds.apache.org/job/PreCommit-SQOOP-Build/2185/artifact/test/target/surefire-reports/] with all "test related" logs and I'm not even remotely able to find anything there.
In normal case we would have one log per test class, but as we've refactored our integration tests to run multiple classes inside single Suite, we're no longer have that for free. We have done this change to cut the time it takes to run the integration tests - before we were initializing mini clusters for each class, which is extra ~40 seconds per class, so I don't think that reverting it would be reasonable.
We should perhaps explore other ways how to get multiple log files.
Diffs (updated)
-----
test/pom.xml 134bca1
test/src/main/java/org/apache/sqoop/test/testng/ReconfigureLogListener.java PRE-CREATION
test/src/test/resources/connector-loading-tests-suite.xml 02c1df3
test/src/test/resources/hive-tests-suite.xml 3e2b328
test/src/test/resources/integration-tests-suite.xml 73e0a77
test/src/test/resources/new-integration-tests-suite.xml 96a1320
test/src/test/resources/shell-tests-suite.xml e2c0b09
test/src/test/resources/tools-tests-suite.xml fee2121
test/src/test/resources/upgrade-tests-suite.xml 3318e67
Diff: https://reviews.apache.org/r/43467/diff/
Testing
-------
Thanks,
Jarek Cecho
Re: Review Request 43467: SQOOP-2832: Sqoop2: Precommit: Create log
files for individual tests
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43467/
-----------------------------------------------------------
(Updated Feb. 11, 2016, 9:29 p.m.)
Review request for Sqoop and Jarek Cecho.
Bugs: SQOOP-2832
https://issues.apache.org/jira/browse/SQOOP-2832
Repository: sqoop-sqoop2
Description
-------
I feel that debugging anything in our pre-commit hook is currently mission impossible from several reasons. One of the main ones is that we're generating one single log file that has [grown to 1GB in size|https://builds.apache.org/job/PreCommit-SQOOP-Build/2185/artifact/test/target/surefire-reports/] with all "test related" logs and I'm not even remotely able to find anything there.
In normal case we would have one log per test class, but as we've refactored our integration tests to run multiple classes inside single Suite, we're no longer have that for free. We have done this change to cut the time it takes to run the integration tests - before we were initializing mini clusters for each class, which is extra ~40 seconds per class, so I don't think that reverting it would be reasonable.
We should perhaps explore other ways how to get multiple log files.
Diffs (updated)
-----
test/pom.xml 134bca1
test/src/main/java/org/apache/sqoop/test/testng/ReconfigureLogListener.java PRE-CREATION
test/src/test/resources/integration-tests-suite.xml 73e0a77
Diff: https://reviews.apache.org/r/43467/diff/
Testing
-------
Thanks,
Jarek Cecho