You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2012/06/05 01:15:22 UTC
[jira] [Created] (HIVE-3085) make parallel tests work
Namit Jain created HIVE-3085:
--------------------------------
Summary: make parallel tests work
Key: HIVE-3085
URL: https://issues.apache.org/jira/browse/HIVE-3085
Project: Hive
Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
I was trying to run the tests using the instructions above.
I was able to run them using a single machine (parallelism of 4 in ~2 hours).
The conf. file is as follows: .hive_ptest.conf
{
"qfile_hosts": [
["root@<MC>", 4]
],
"other_hosts": [
["root@<MC>", 1]
],
"master_base_path": "/data/users/tmp",
"host_base_path": "/data/users/hivetests",
"java_home": "/usr/local/jdk-6u24-64"
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3085) make parallel tests work
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-3085:
-----------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed. Thanks Shuai
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch, hive.3085.2.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3085) make parallel tests work
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-3085:
-----------------------------
Attachment: hive.3085.2.patch
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch, hive.3085.2.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3085) make parallel tests work
Posted by "Shuai Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shuai Ding reassigned HIVE-3085:
--------------------------------
Assignee: Shuai Ding (was: Namit Jain)
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396159#comment-13396159 ]
Carl Steinbach commented on HIVE-3085:
--------------------------------------
@Namit: Yeah, I think it's a problem, but the changes made in HIVE-3092 should insulate us from it. You'll turn up a bunch of links if you try searching for "ivy concurrent build". However, I also think it's possible to work around this problem by publishing the Hive JARs to a filesystem repository that's local to each build, and then making sure that this local repository takes precedence in the list of resolvers in ivy/ivysettings.xml.
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch, hive.3085.2.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Shuai Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293124#comment-13293124 ]
Shuai Ding commented on HIVE-3085:
----------------------------------
https://reviews.facebook.net/D3585
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch, hive.3085.2.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293055#comment-13293055 ]
Edward Capriolo commented on HIVE-3085:
---------------------------------------
Do the 'parallel' tests still require a shared NFS mount? A while back someone told me I did not need NFS anymore because hadoop 'give me Big Datas'. Really though this shared NFS mount destroys the utility of this toolkit for me.
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396098#comment-13396098 ]
Namit Jain commented on HIVE-3085:
----------------------------------
@Carl, isn't there a problem in having hive*jar in ~/ivy2 ?
I am not sure, but will it create a problem if I am trying to compile 2 different versions of hive on my machine concurrently ?
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch, hive.3085.2.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293191#comment-13293191 ]
Carl Steinbach commented on HIVE-3085:
--------------------------------------
@Namit: I filed two followup tickets: HIVE-3116 and HIVE-3117. Thanks.
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch, hive.3085.2.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3085) make parallel tests work
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-3085:
-----------------------------
Attachment: hive.3085.1.patch
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.3085.1.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3085) make parallel tests work
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-3085:
-----------------------------
Status: Patch Available (was: Open)
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.3085.1.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289052#comment-13289052 ]
Namit Jain commented on HIVE-3085:
----------------------------------
It ran fine for me
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.3085.1.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Travis Crawford (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289001#comment-13289001 ]
Travis Crawford commented on HIVE-3085:
---------------------------------------
Does {{script_broken_pipe1.q}} pass in parallel mode? I've been looking at HIVE-3084 for a couple hours now and am curious if parallel mode casuses it to complete successfully. It passes when run by itself, but regularly fails as part of the full test suite.
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.3085.1.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293925#comment-13293925 ]
Hudson commented on HIVE-3085:
------------------------------
Integrated in Hive-trunk-h0.21 #1482 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1482/])
HIVE-3085 make parallel tests work
(Shuai Ding via namit) (Revision 1349398)
Result = SUCCESS
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349398
Files :
* /hive/trunk/testutils/ptest/hivetest.py
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch, hive.3085.2.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293700#comment-13293700 ]
Namit Jain commented on HIVE-3085:
----------------------------------
+1
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch, hive.3085.2.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293176#comment-13293176 ]
Carl Steinbach commented on HIVE-3085:
--------------------------------------
@Namit
bq. Won't it be better to not populate hive*jar in ivy in our local builds?
I don't think we can do that and also continue to list inter-subproject dependencies in the ivy.xml files. The basic reason why this doesn't work correctly for Hive is that the build is still not using Ivy correctly. More specifically, we're manually specifying the order in which subprojects are built instead of letting Ivy determine the order through dependency analysis.
bq. Or, what is the downside of removing 'hive.jar' from .ivy2 and then running ant clean package.
The main downside is that the user may have configured ivy.cache.dir to be something other than ~/.ivy2, so to be sure that you're deleting the right files you have to get the value of ${ivy.cache.dir} (which didn't seem that straightforward the last time I looked at it: http://ant.apache.org/ivy/history/2.0.0/use/cleancache.html).
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch, hive.3085.2.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Shuai Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292969#comment-13292969 ]
Shuai Ding commented on HIVE-3085:
----------------------------------
https://reviews.facebook.net/D3585
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Shuai Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293289#comment-13293289 ]
Shuai Ding commented on HIVE-3085:
----------------------------------
https://reviews.facebook.net/D3585
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch, hive.3085.2.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Shuai Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293117#comment-13293117 ]
Shuai Ding commented on HIVE-3085:
----------------------------------
https://reviews.facebook.net/D3585
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch, hive.3085.2.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288984#comment-13288984 ]
Namit Jain commented on HIVE-3085:
----------------------------------
This is a script change used to run tests in parallel.
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.3085.1.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3085) make parallel tests work
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-3085:
---------------------------------
Component/s: Testing Infrastructure
I added some comments to the review. Thanks.
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293095#comment-13293095 ]
Namit Jain commented on HIVE-3085:
----------------------------------
Right now, the tests does require a shared mount if you want to run on multiple machines.
This is good, if we dont want to compile across all the machines.
Having said that, I am also planning to use it on my machine only, and this should still help to finish the tests in about 1.5 hours.
In that case, I was able to use local disk on my machine.
This can be further optimized. Some of them are:
1. Run the other tests in parallel with TestCliDriver and TestNegativeCliDriver
2. Run the tests that need super-user privilege in parallel with TestCliDriver and TestNegativeCliDriver
Filed https://issues.apache.org/jira/browse/HIVE-3111 for that.
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch, hive.3085.2.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290565#comment-13290565 ]
Kevin Wilfong commented on HIVE-3085:
-------------------------------------
The tests testDatabaseLocation and testSynchronized are failing for all test classes that extend TestHiveMetaStore when I try to run the script.
I'm running in a Linux environment using the .hive_ptest.conf in the description.
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.3085.1.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3085) make parallel tests work
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293121#comment-13293121 ]
Namit Jain commented on HIVE-3085:
----------------------------------
@Carl, I had a question about ant very-clean package.
That takes a very long time (~20 min.) since we are downloading so many jar files.
Won't it be better to not populate hive*jar in ivy in our local builds ?
Then, ant clean package can run much faster.
Or, what is the downside of removing '*hive*.jar' from .ivy2 and then running ant clean package.
Other jars rarely change, but this saves ~10 min. in compile time.
> make parallel tests work
> ------------------------
>
> Key: HIVE-3085
> URL: https://issues.apache.org/jira/browse/HIVE-3085
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Shuai Ding
> Attachments: hive.3085.1.patch, hive.3085.2.patch
>
>
> https://cwiki.apache.org/Hive/unit-test-parallel-execution.html
> I was trying to run the tests using the instructions above.
> I was able to run them using a single machine (parallelism of 4 in ~2 hours).
> The conf. file is as follows: .hive_ptest.conf
> {
> "qfile_hosts": [
> ["root@<MC>", 4]
> ],
> "other_hosts": [
> ["root@<MC>", 1]
> ],
> "master_base_path": "/data/users/tmp",
> "host_base_path": "/data/users/hivetests",
> "java_home": "/usr/local/jdk-6u24-64"
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira