You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org> on 2010/08/10 03:15:16 UTC
[jira] Created: (HIVE-1523) ql tests no longer work in miniMR mode
ql tests no longer work in miniMR mode
--------------------------------------
Key: HIVE-1523
URL: https://issues.apache.org/jira/browse/HIVE-1523
Project: Hadoop Hive
Issue Type: Bug
Components: Build Infrastructure
Reporter: Joydeep Sen Sarma
as per title. here's the first exception i see:
2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1523) ql tests no longer work in miniMR
mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901687#action_12901687 ]
Joydeep Sen Sarma commented on HIVE-1523:
-----------------------------------------
can someone review/commit this? i don't think i am going to make more changes to this.
will work on long regression framework separately.
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch, hive-1523.2.patch, hive-1523.3.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1523) ql tests no longer work in miniMR mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-1523:
------------------------------------
Status: Patch Available (was: Open)
Assignee: Joydeep Sen Sarma
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1523) ql tests no longer work in miniMR mode
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1523:
-----------------------------
Status: Open (was: Patch Available)
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch, hive-1523.2.patch, hive-1523.3.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1523) ql tests no longer work in miniMR
mode
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902558#action_12902558 ]
Namit Jain commented on HIVE-1523:
----------------------------------
+1
Running tests now - will file a new jira for the ability to run unit tests in parallel in miniMr mode
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch, hive-1523.2.patch, hive-1523.3.patch, hive-1523.4.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1523) ql tests no longer work in miniMR
mode
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900018#action_12900018 ]
John Sichi commented on HIVE-1523:
----------------------------------
+1. Last time I talked to Ning about this, my take was that we should be able to re-run any subset of tests in either mode (without needing test codegen for it), but for now we can just get things working again this way.
Some candidates for existing tests to adding to the minimr suite:
jsichi-mac:clientpositive jsichi$ grep reducer *.q
bucket1.q:set hive.exec.reducers.max = 200;
bucket2.q:set hive.exec.reducers.max = 1;
bucket3.q:set hive.exec.reducers.max = 1;
bucket4.q:set hive.exec.reducers.max = 1;
bucketmapjoin6.q:set hive.exec.reducers.max=1;
disable_merge_for_bucketing.q:set hive.exec.reducers.max = 1;
reduce_deduplicate.q:set hive.exec.reducers.max = 1;
sample10.q:set hive.exec.reducers.max=4;
smb_mapjoin_6.q:set hive.exec.reducers.max = 1;
smb_mapjoin_7.q:set hive.exec.reducers.max = 1;
smb_mapjoin_8.q:set hive.exec.reducers.max = 1;
smb_mapjoin_8.q:set hive.exec.reducers.max = 1;
udaf_percentile_approx.q:set hive.exec.reducers.max=4
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1523) ql tests no longer work in miniMR mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-1523:
------------------------------------
Attachment: hive-1523.3.patch
small change - fix 0.20 version match to pick the right jetty version.
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch, hive-1523.2.patch, hive-1523.3.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1523) ql tests no longer work in miniMR mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-1523:
------------------------------------
Attachment: hive-1523.2.patch
with modified list of minimr tests:
+ <property name="minimr.query.files" value="input16_cc.q,scriptfile1.q,bucket1.q,bucket2.q,bucket3.q,bucket4.q,bucketmapjoin6.q,disable_merge_for_bucketing.q,reduce_deduplicate.q,smb_mapjoin_6.q,smb_mapjoin_7.q,join1.q,groupby2.q"/>
i took the ones that worked from John's list. also added a couple of tests that had 'add jar' and 'add file' commands (since their interaction with real cluster is quite different).
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch, hive-1523.2.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HIVE-1523) ql tests no longer work in
miniMR mode
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900018#action_12900018 ]
John Sichi edited comment on HIVE-1523 at 8/18/10 4:30 PM:
-----------------------------------------------------------
+1. Last time I talked to Ning about this, my take was that we should be able to re-run any subset of tests in either mode (without needing test codegen for it), but for now we can just get things working again this way.
Some candidates for existing tests to add in to the minimr suite:
jsichi-mac:clientpositive jsichi$ grep reducer *.q
bucket1.q:set hive.exec.reducers.max = 200;
bucket2.q:set hive.exec.reducers.max = 1;
bucket3.q:set hive.exec.reducers.max = 1;
bucket4.q:set hive.exec.reducers.max = 1;
bucketmapjoin6.q:set hive.exec.reducers.max=1;
disable_merge_for_bucketing.q:set hive.exec.reducers.max = 1;
reduce_deduplicate.q:set hive.exec.reducers.max = 1;
sample10.q:set hive.exec.reducers.max=4;
smb_mapjoin_6.q:set hive.exec.reducers.max = 1;
smb_mapjoin_7.q:set hive.exec.reducers.max = 1;
smb_mapjoin_8.q:set hive.exec.reducers.max = 1;
smb_mapjoin_8.q:set hive.exec.reducers.max = 1;
udaf_percentile_approx.q:set hive.exec.reducers.max=4
was (Author: jvs):
+1. Last time I talked to Ning about this, my take was that we should be able to re-run any subset of tests in either mode (without needing test codegen for it), but for now we can just get things working again this way.
Some candidates for existing tests to adding to the minimr suite:
jsichi-mac:clientpositive jsichi$ grep reducer *.q
bucket1.q:set hive.exec.reducers.max = 200;
bucket2.q:set hive.exec.reducers.max = 1;
bucket3.q:set hive.exec.reducers.max = 1;
bucket4.q:set hive.exec.reducers.max = 1;
bucketmapjoin6.q:set hive.exec.reducers.max=1;
disable_merge_for_bucketing.q:set hive.exec.reducers.max = 1;
reduce_deduplicate.q:set hive.exec.reducers.max = 1;
sample10.q:set hive.exec.reducers.max=4;
smb_mapjoin_6.q:set hive.exec.reducers.max = 1;
smb_mapjoin_7.q:set hive.exec.reducers.max = 1;
smb_mapjoin_8.q:set hive.exec.reducers.max = 1;
smb_mapjoin_8.q:set hive.exec.reducers.max = 1;
udaf_percentile_approx.q:set hive.exec.reducers.max=4
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1523) ql tests no longer work in miniMR
mode
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900141#action_12900141 ]
John Sichi commented on HIVE-1523:
----------------------------------
Yeah, shortreg/longreg split would be good. The challenge is to keep longreg healthy since breakages don't get caught with every checkin, so we'll need
(a) automation to run it constantly and report failures
(b) people to actually fix failures in a timely fashion
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch, hive-1523.2.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1523) ql tests no longer work in miniMR
mode
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902025#action_12902025 ]
Namit Jain commented on HIVE-1523:
----------------------------------
The tests in miniMR will run twice, and the log files may get over-written.
They should be excluded from non-miniMR.
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch, hive-1523.2.patch, hive-1523.3.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1523) ql tests no longer work in miniMR mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-1523:
------------------------------------
Status: Patch Available (was: Open)
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch, hive-1523.2.patch, hive-1523.3.patch, hive-1523.4.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1523) ql tests no longer work in miniMR mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-1523:
------------------------------------
Attachment: hive-1523.1.patch
fixed minimr test mode.
enabled a couple of queries to always run (additionally) in minimr mode (like hbase-handler tests) when running standard tests. we should probably expand this to a larger number of queries (especially those requiring multiple reducers). i don't have good insight into this part - if people have ideas - we can expand the list easily.
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1523) ql tests no longer work in miniMR mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-1523:
------------------------------------
Attachment: hive-1523.4.patch
- added exclude tests - minimr tests are excluded from regular clientpositive tests
- did some subtle changes in how fs.default.name and mapred.job.tracker are specified to allow testing against external hadoop clusters
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch, hive-1523.2.patch, hive-1523.3.patch, hive-1523.4.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1523) ql tests no longer work in miniMR
mode
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901971#action_12901971 ]
Namit Jain commented on HIVE-1523:
----------------------------------
I will take a look
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch, hive-1523.2.patch, hive-1523.3.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1523) ql tests no longer work in miniMR
mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902560#action_12902560 ]
Joydeep Sen Sarma commented on HIVE-1523:
-----------------------------------------
there's already a jira on running tests in parallel. i think i can cover it there itself.
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch, hive-1523.2.patch, hive-1523.3.patch, hive-1523.4.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1523) ql tests no longer work in miniMR mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-1523:
------------------------------------
Component/s: Query Processor
(was: Build Infrastructure)
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1523) ql tests no longer work in miniMR
mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900048#action_12900048 ]
Joydeep Sen Sarma commented on HIVE-1523:
-----------------------------------------
i am running through the above qfiles and see what executes successfully on minimr (because many dont).
one concern is the length of the tests. i think we need to divide our tests into a short and long regression. otherwise development cycle is severely impacted if everything has to be tested on every iteration.
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1523) ql tests no longer work in miniMR mode
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1523:
-----------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
Committed. Thanks Joy
> ql tests no longer work in miniMR mode
> --------------------------------------
>
> Key: HIVE-1523
> URL: https://issues.apache.org/jira/browse/HIVE-1523
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-1523.1.patch, hive-1523.2.patch, hive-1523.3.patch, hive-1523.4.patch
>
>
> as per title. here's the first exception i see:
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(743)) - Got exception: java.io.FileNotFoun\
> dException File file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> 2010-08-09 18:05:11,259 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(746)) - java.io.FileNotFoundException: Fil\
> e file:/build/ql/test/data/warehouse/dest_j1 does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:136)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:677)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.