You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Richard Lee (JIRA)" <ji...@apache.org> on 2009/02/28 00:43:15 UTC
[jira] Created: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
External table definitions should be allowed outside of Warehouse Filesystem
----------------------------------------------------------------------------
Key: HIVE-316
URL: https://issues.apache.org/jira/browse/HIVE-316
Project: Hadoop Hive
Issue Type: Improvement
Components: Metastore
Reporter: Richard Lee
I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680559#action_12680559 ]
Namit Jain commented on HIVE-316:
---------------------------------
Can you add a test - I am not sure whether it will work with 1 hdfs and 1 external table. Can you try that with your changes ?
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Attachments: hive-external-filesystems.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Richard Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Lee updated HIVE-316:
-----------------------------
Attachment: hive-external-filesystems5.diff
Ok. I did the svn adds and collapsed both tests into a single test which drops the table that it creates. (the second test just ads functionality onto the first test) and took a new diff... this time not appending to the previous diff.
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Richard Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683947#action_12683947 ]
Richard Lee commented on HIVE-316:
----------------------------------
Namit, I re-checked out trunk and applied my patch to it so that i have a clean tree... i then ran the test 4 times in a row with the following:
ant test -Dtest.include=TestCliDriver -Dqfile=external_table_join.q
Could not reproduce the error to debug. Were you merging multiple changes? Let me know if there are any other steps that I can try to reproduce the issue.
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Richard Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769290#action_12769290 ]
Richard Lee commented on HIVE-316:
----------------------------------
I've upgraded to hive 0.4.0 and the issue seems resolved.
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681575#action_12681575 ]
Prasad Chakka commented on HIVE-316:
------------------------------------
The latest patch contains two diffs for each file. Also single diff should have all the files including the testcase and its result.
Also add the both test files and the outputs using svn add and then take the diff.
can you drop the new tables that you create at the end of the testcase so that if there are any 'show tables' test cases will not fail.
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Richard Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Lee updated HIVE-316:
-----------------------------
Status: Open (was: Patch Available)
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Richard Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Lee updated HIVE-316:
-----------------------------
Attachment: hive-external-filesystems2.diff
external_table1.q.out
external_table1.q
Attached are the test query script for creating an external table on a filesystem other than the warehouse filesystem. There's a dirty hard coded path in the test... but i'm not sure what to do about it.
The hive-external-filesystems2.diff file contains the latest diff, which includes getFs() throwing a MetaException instead of just logging and returning null.
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, hive-external-filesystems.diff, hive-external-filesystems2.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683554#action_12683554 ]
Namit Jain commented on HIVE-316:
---------------------------------
While I was running tests for merging, I got the following error:
Note that this happened while running tests for the second time. I have not debugged it, but I suspect it might have something to do with the directory already present (created the first time).
[junit] Begin query: external_table_join.q
[junit] Loading data to table srcpart partition {ds=2008-04-08, hr=11}
[junit] OK
[junit] Loading data to table srcpart partition {ds=2008-04-08, hr=12}
[junit] OK
[junit] Loading data to table srcpart partition {ds=2008-04-09, hr=11}
[junit] OK
[junit] Loading data to table srcpart partition {ds=2008-04-09, hr=12}
[junit] OK
[junit] Loading data to table srcbucket
[junit] OK
[junit] Loading data to table srcbucket
[junit] OK
[junit] Loading data to table src
[junit] OK
[junit] Exception: Client Execution failed with error code = 9
[junit] junit.framework.AssertionFailedError: Client Execution failed with error code = 9
[junit] at junit.framework.Assert.fail(Assert.java:47)
[junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_external_table_join(TestCliDriver.java:705)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at junit.framework.TestCase.runTest(TestCase.java:154)
[junit] at junit.framework.TestCase.runBare(TestCase.java:127)
[junit] at junit.framework.TestResult$1.protect(TestResult.java:106)
[junit] at junit.framework.TestResult.runProtected(TestResult.java:124)
[junit] at junit.framework.TestResult.run(TestResult.java:109)
[junit] at junit.framework.TestCase.run(TestCase.java:118)
[junit] at junit.framework.TestSuite.runTest(TestSuite.java:208)
[junit] at junit.framework.TestSuite.run(TestSuite.java:203)
[junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:297)
[junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:672)
[junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:567)
[junit] Hive history file=/data/users/njain/hive_commit/trunk/ql/../build/ql/tmp/hive_job_log_njain_200903181609_1260283057.txt
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Richard Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Lee updated HIVE-316:
-----------------------------
Attachment: hive-external-filesystems3.diff
external_table_join.q
Here's a join unit test and a new diff that's needed for the test to pass. Essentially, the TestFileSystem referenced in the test needs to be in the classpath of the test hadoop job... so i modified the build-common to create a test-classes.jar instead of just a test-udf.jar.
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Thusoo reassigned HIVE-316:
----------------------------------
Assignee: Richard Lee
Made Richard a contributor and assigned the bug to him.
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: hive-external-filesystems.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Richard Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688363#action_12688363 ]
Richard Lee commented on HIVE-316:
----------------------------------
So the hard coded path seems to be a red herring.
in test-common.xml there are two modes in which the junit tests are run... If a specific class is specified, the test passes. If no class is specified, all tests are run and the test fails. Looking at the resulting .out file, I see that it was unable to figure out how to handle tmpfs:// ... so the SET fs.testfs.impl = X at the top of the .q file was ignored.
It looks like some form of race condition on the job conf around setting file system handlers... I don't think that this problem is specific to my test case.
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Richard Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683564#action_12683564 ]
Richard Lee commented on HIVE-316:
----------------------------------
I was afraid that this might happen. Is there a way to only "Delete If Exists" from within a .q file?
dfs -rmr seems to fail if the directory doesn't actually exist.
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Richard Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Lee resolved HIVE-316.
------------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Fix For: 0.4.0
>
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Richard Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Lee updated HIVE-316:
-----------------------------
Attachment: hive-external-filesystems.diff
Attached is a diff from trunk rev 751874.
This should allow users to specify an absolute path outside of the hive warehouse FileSystem when creating external tables.
Relative paths are still resolved relative to the warehouse's FileSystem.
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Attachments: hive-external-filesystems.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677604#action_12677604 ]
Prasad Chakka commented on HIVE-316:
------------------------------------
There is an EXTERNAL TABLE feature where you can give the location of the table such as file:// during the creation of the table. Can you use that? If not, how else would you want to use it?
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Richard Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677605#action_12677605 ]
Richard Lee commented on HIVE-316:
----------------------------------
If you specify a file:// URI for the external table definition, it eventually gets passed to Warehouse.mkdirs(). That explicitly calls fs.mkdirs() which will throw an exception if the URI is outside of the FileSystem specified in the hive configuration. If i had specified a file:// URI in the hive configuration that'd work, but i don't want ALL of my hive operations to be done on my external storage.
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683628#action_12683628 ]
Namit Jain commented on HIVE-316:
---------------------------------
No, there is nothing like 'Delete If Exists'
You can invoke a shell script from .q file
try
!<shell_script_name>
In the script, you can delete the directory and ignore the error if any
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12685374#action_12685374 ]
Namit Jain commented on HIVE-316:
---------------------------------
No, I was only merging your change - I did a svn up and reran the about query a few times, and it succeeded.
I was only merging your patch - not sure, why I got the error in the first place.
Running all tests again
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681900#action_12681900 ]
Prasad Chakka commented on HIVE-316:
------------------------------------
looks fine to me. +1
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688331#action_12688331 ]
Namit Jain commented on HIVE-316:
---------------------------------
you will get the error if you run all the tests
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688598#action_12688598 ]
Prasad Chakka commented on HIVE-316:
------------------------------------
in either mode tests are run sequentially so i don't think race condition is a problem. i will take a look.
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680567#action_12680567 ]
Prasad Chakka commented on HIVE-316:
------------------------------------
Warehouse.java: 86, 94
you should guard against null return from getFs() or make getFs throw an exception. I would prefer an exception since it is an error if Hive doesn't know what is the file system of a file.
otherwise it looks good, but adding the test that Namit suggested would be great.
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Attachments: hive-external-filesystems.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Richard Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Lee updated HIVE-316:
-----------------------------
Status: Patch Available (was: Open)
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, external_table_join.q, hive-external-filesystems.diff, hive-external-filesystems2.diff, hive-external-filesystems3.diff, hive-external-filesystems5.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-316) External table definitions should be
allowed outside of Warehouse Filesystem
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681122#action_12681122 ]
Namit Jain commented on HIVE-316:
---------------------------------
Can you add a test which joins a hdfs table (standard hive table) with another external table ?
> External table definitions should be allowed outside of Warehouse Filesystem
> ----------------------------------------------------------------------------
>
> Key: HIVE-316
> URL: https://issues.apache.org/jira/browse/HIVE-316
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Richard Lee
> Assignee: Richard Lee
> Attachments: external_table1.q, external_table1.q.out, hive-external-filesystems.diff, hive-external-filesystems2.diff
>
>
> I have a situation where I have hive's datastore pointed at an hdfs, but would like to create an external table on data accessable from an outside data storage solution exported via nfs.
> Presently, Warehouse.java aggregates only a single FileSystem object which limits all tables, both internal and external to being relative to the URl specified in the hive configuration. I feel like the Warehouse code should prefer to use the configured warehouse URI for non-absolute Paths, but honor paths outside of the Warehouse; particularly when they are defined in external tables.
> I was going to implement this by adding a Map of FileSystem objects to the Warehouse object. This map gets populated with FileSystem objects when operations cannot be performed by either the warehouse FS, or any other FS object in the map. I am not sure what impact this change would have on hive overall... or if this is the only place that this change would need to be made.
> Please advise.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.