You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Thiruvel Thirumoolan (JIRA)" <ji...@apache.org> on 2011/03/08 20:14:59 UTC
[jira] Created: (HIVE-2032) create database does not honour
warehouse.dir in dbproperties
create database does not honour warehouse.dir in dbproperties
-------------------------------------------------------------
Key: HIVE-2032
URL: https://issues.apache.org/jira/browse/HIVE-2032
Project: Hive
Issue Type: Bug
Components: Clients
Affects Versions: 0.7.0, 0.8.0
Reporter: Thiruvel Thirumoolan
Assignee: Thiruvel Thirumoolan
Fix For: 0.8.0
# create database db with dbproperties ('hive.metastore.warehouse.dir' = 'loc');
The above command does not set location of 'db' to 'loc'. It instead creates 'db.db' under the warehouse directory configured in hive-site.xml of CLI. Looks conflicting with HIVE-1820's expectation. If scratch dir is specified here, that is honoured.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-2032) create database does not honour
warehouse.dir in dbproperties
Posted by "Thiruvel Thirumoolan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvel Thirumoolan updated HIVE-2032:
---------------------------------------
Attachment: DatabaseLocation.patch
Preliminary patch. Test cases are added.
> create database does not honour warehouse.dir in dbproperties
> -------------------------------------------------------------
>
> Key: HIVE-2032
> URL: https://issues.apache.org/jira/browse/HIVE-2032
> Project: Hive
> Issue Type: Bug
> Components: Clients
> Affects Versions: 0.7.0, 0.8.0
> Reporter: Thiruvel Thirumoolan
> Assignee: Thiruvel Thirumoolan
> Fix For: 0.8.0
>
> Attachments: DatabaseLocation.patch
>
>
> # create database db with dbproperties ('hive.metastore.warehouse.dir' = 'loc');
> The above command does not set location of 'db' to 'loc'. It instead creates 'db.db' under the warehouse directory configured in hive-site.xml of CLI. Looks conflicting with HIVE-1820's expectation. If scratch dir is specified here, that is honoured.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2032) create database does not honour
warehouse.dir in dbproperties
Posted by "Thiruvel Thirumoolan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015925#comment-13015925 ]
Thiruvel Thirumoolan commented on HIVE-2032:
--------------------------------------------
> Use cases make sense. But, drop database would remove tables only from new location.
If I am not wrong, drop db succeeds only if all tables under it are dropped.
> Thiruvel, did you get a chance to test this? Because, your changes in patch does not look complete. Changes should be propagated to PersistentManager through ObjectStore.
Will take a look.
> create database does not honour warehouse.dir in dbproperties
> -------------------------------------------------------------
>
> Key: HIVE-2032
> URL: https://issues.apache.org/jira/browse/HIVE-2032
> Project: Hive
> Issue Type: Bug
> Components: Clients
> Affects Versions: 0.7.0, 0.8.0
> Reporter: Thiruvel Thirumoolan
> Assignee: Thiruvel Thirumoolan
> Fix For: 0.8.0
>
> Attachments: DatabaseLocation.patch
>
>
> # create database db with dbproperties ('hive.metastore.warehouse.dir' = 'loc');
> The above command does not set location of 'db' to 'loc'. It instead creates 'db.db' under the warehouse directory configured in hive-site.xml of CLI. Looks conflicting with HIVE-1820's expectation. If scratch dir is specified here, that is honoured.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2032) create database does not honour
warehouse.dir in dbproperties
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015918#comment-13015918 ]
Amareshwari Sriramadasu commented on HIVE-2032:
-----------------------------------------------
Use cases make sense. But, drop database would remove tables only from new location.
Does anyone have the idea why alter Database currently allows to change only DB properties? Namit/Ning?
bq.Post altering, new tables will be created under new location. Old tables' have the fully qualified location in metadata and they should continue to work as before.
Thiruvel, did you get a chance to test this? Because, your changes in patch does not look complete. Changes should be propagated to PersistentManager through ObjectStore.
> create database does not honour warehouse.dir in dbproperties
> -------------------------------------------------------------
>
> Key: HIVE-2032
> URL: https://issues.apache.org/jira/browse/HIVE-2032
> Project: Hive
> Issue Type: Bug
> Components: Clients
> Affects Versions: 0.7.0, 0.8.0
> Reporter: Thiruvel Thirumoolan
> Assignee: Thiruvel Thirumoolan
> Fix For: 0.8.0
>
> Attachments: DatabaseLocation.patch
>
>
> # create database db with dbproperties ('hive.metastore.warehouse.dir' = 'loc');
> The above command does not set location of 'db' to 'loc'. It instead creates 'db.db' under the warehouse directory configured in hive-site.xml of CLI. Looks conflicting with HIVE-1820's expectation. If scratch dir is specified here, that is honoured.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2032) create database does not honour
warehouse.dir in dbproperties
Posted by "Thiruvel Thirumoolan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvel Thirumoolan resolved HIVE-2032.
----------------------------------------
Resolution: Duplicate
HIVE-1537 will address specifying a location for a db.
> create database does not honour warehouse.dir in dbproperties
> -------------------------------------------------------------
>
> Key: HIVE-2032
> URL: https://issues.apache.org/jira/browse/HIVE-2032
> Project: Hive
> Issue Type: Bug
> Components: Clients
> Affects Versions: 0.7.0, 0.8.0
> Reporter: Thiruvel Thirumoolan
> Assignee: Thiruvel Thirumoolan
> Fix For: 0.8.0
>
> Attachments: DatabaseLocation.patch
>
>
> # create database db with dbproperties ('hive.metastore.warehouse.dir' = 'loc');
> The above command does not set location of 'db' to 'loc'. It instead creates 'db.db' under the warehouse directory configured in hive-site.xml of CLI. Looks conflicting with HIVE-1820's expectation. If scratch dir is specified here, that is honoured.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2032) create database does not honour
warehouse.dir in dbproperties
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017277#comment-13017277 ]
Ning Zhang commented on HIVE-2032:
----------------------------------
Thiruvel, HIVE-1820 (actually the subtask of it HIVE-1836) is not intended to define the database LOCATION. The database metastore schema already contains a LOCATION field, but it was not currently exposed through the CREATE DATABASE syntax. That's what HIVE-1537 is about to solve. HIVE-1836 is to introduce dbproperties of generic key-value pairs. The dbproperties can be used/interpreted by application-level pre-execution hooks. We currently do not require the warehouse.dir (if it is one of the keys) must be consistent with the LOCATION field, and I think probably we don't need to keep 2 piece of information with the same semantics.
If you want to change the database location URL I think probably a better way is to change the CREATE DATABASE syntax to add location (HIVE-1537) and any table created under that database should honor the database location property. What do you think?
> create database does not honour warehouse.dir in dbproperties
> -------------------------------------------------------------
>
> Key: HIVE-2032
> URL: https://issues.apache.org/jira/browse/HIVE-2032
> Project: Hive
> Issue Type: Bug
> Components: Clients
> Affects Versions: 0.7.0, 0.8.0
> Reporter: Thiruvel Thirumoolan
> Assignee: Thiruvel Thirumoolan
> Fix For: 0.8.0
>
> Attachments: DatabaseLocation.patch
>
>
> # create database db with dbproperties ('hive.metastore.warehouse.dir' = 'loc');
> The above command does not set location of 'db' to 'loc'. It instead creates 'db.db' under the warehouse directory configured in hive-site.xml of CLI. Looks conflicting with HIVE-1820's expectation. If scratch dir is specified here, that is honoured.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2032) create database does not honour
warehouse.dir in dbproperties
Posted by "Thiruvel Thirumoolan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015818#comment-13015818 ]
Thiruvel Thirumoolan commented on HIVE-2032:
--------------------------------------------
@Amareshwari
Post altering, new tables will be created under new location. Old tables' have the fully qualified location in metadata and they should continue to work as before.
The reasons I went with alter location are:
1. Allow migration to happen if one would like to reorganize or if quota runs out. Not sure how many folks have this situation.
2. One can migrate new tables to another DFS cluster (existing cluster becoming full).
3. One can migrate between file systems if they have sufficient use cases.
Do you think these are valid use cases?
> create database does not honour warehouse.dir in dbproperties
> -------------------------------------------------------------
>
> Key: HIVE-2032
> URL: https://issues.apache.org/jira/browse/HIVE-2032
> Project: Hive
> Issue Type: Bug
> Components: Clients
> Affects Versions: 0.7.0, 0.8.0
> Reporter: Thiruvel Thirumoolan
> Assignee: Thiruvel Thirumoolan
> Fix For: 0.8.0
>
> Attachments: DatabaseLocation.patch
>
>
> # create database db with dbproperties ('hive.metastore.warehouse.dir' = 'loc');
> The above command does not set location of 'db' to 'loc'. It instead creates 'db.db' under the warehouse directory configured in hive-site.xml of CLI. Looks conflicting with HIVE-1820's expectation. If scratch dir is specified here, that is honoured.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2032) create database does not honour
warehouse.dir in dbproperties
Posted by "Thiruvel Thirumoolan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015951#comment-13015951 ]
Thiruvel Thirumoolan commented on HIVE-2032:
--------------------------------------------
HIVE-1537 is supporting LOCATION in create database command
HIVE-675 - first comment is consistent with HIVE-1537
HIVE-1820 is about warehouse location through DBPROPERTIES
My approach on this JIRA so far was based on HIVE-1820.
What is the suggested way to specify location during create database?
@Carl, @Ning, @He Yongqiang - suggestions?
> create database does not honour warehouse.dir in dbproperties
> -------------------------------------------------------------
>
> Key: HIVE-2032
> URL: https://issues.apache.org/jira/browse/HIVE-2032
> Project: Hive
> Issue Type: Bug
> Components: Clients
> Affects Versions: 0.7.0, 0.8.0
> Reporter: Thiruvel Thirumoolan
> Assignee: Thiruvel Thirumoolan
> Fix For: 0.8.0
>
> Attachments: DatabaseLocation.patch
>
>
> # create database db with dbproperties ('hive.metastore.warehouse.dir' = 'loc');
> The above command does not set location of 'db' to 'loc'. It instead creates 'db.db' under the warehouse directory configured in hive-site.xml of CLI. Looks conflicting with HIVE-1820's expectation. If scratch dir is specified here, that is honoured.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2032) create database does not honour
warehouse.dir in dbproperties
Posted by "Thiruvel Thirumoolan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017419#comment-13017419 ]
Thiruvel Thirumoolan commented on HIVE-2032:
--------------------------------------------
I prefer HIVE-1537 approach. I got confused by the test queries and found 1537 only later.
Thanks for the clarification.
> create database does not honour warehouse.dir in dbproperties
> -------------------------------------------------------------
>
> Key: HIVE-2032
> URL: https://issues.apache.org/jira/browse/HIVE-2032
> Project: Hive
> Issue Type: Bug
> Components: Clients
> Affects Versions: 0.7.0, 0.8.0
> Reporter: Thiruvel Thirumoolan
> Assignee: Thiruvel Thirumoolan
> Fix For: 0.8.0
>
> Attachments: DatabaseLocation.patch
>
>
> # create database db with dbproperties ('hive.metastore.warehouse.dir' = 'loc');
> The above command does not set location of 'db' to 'loc'. It instead creates 'db.db' under the warehouse directory configured in hive-site.xml of CLI. Looks conflicting with HIVE-1820's expectation. If scratch dir is specified here, that is honoured.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2032) create database does not honour
warehouse.dir in dbproperties
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017363#comment-13017363 ]
Amareshwari Sriramadasu commented on HIVE-2032:
-----------------------------------------------
I agree with Ning. There should not be two piece of information with same semantics. Location should be added to create database and alter database if it can be altered. Also, We should document that dbproperties are interpreted at application level.
> create database does not honour warehouse.dir in dbproperties
> -------------------------------------------------------------
>
> Key: HIVE-2032
> URL: https://issues.apache.org/jira/browse/HIVE-2032
> Project: Hive
> Issue Type: Bug
> Components: Clients
> Affects Versions: 0.7.0, 0.8.0
> Reporter: Thiruvel Thirumoolan
> Assignee: Thiruvel Thirumoolan
> Fix For: 0.8.0
>
> Attachments: DatabaseLocation.patch
>
>
> # create database db with dbproperties ('hive.metastore.warehouse.dir' = 'loc');
> The above command does not set location of 'db' to 'loc'. It instead creates 'db.db' under the warehouse directory configured in hive-site.xml of CLI. Looks conflicting with HIVE-1820's expectation. If scratch dir is specified here, that is honoured.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2032) create database does not honour
warehouse.dir in dbproperties
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014458#comment-13014458 ]
Amareshwari Sriramadasu commented on HIVE-2032:
-----------------------------------------------
I see changes for alter Database also to change the location URI. What are semantics of alter database for changing location uri? Is it going to create the tables under the new location after altering? If so, what happens to earlier tables. I see that location uri is accessed only in create db and drop db. So, I think we should not allow altering the location uri. Shall we throw an error if it altered?
Changes for create DB look fine. Can you update TestHiveMetaStore.testDatabase() to test the same?
> create database does not honour warehouse.dir in dbproperties
> -------------------------------------------------------------
>
> Key: HIVE-2032
> URL: https://issues.apache.org/jira/browse/HIVE-2032
> Project: Hive
> Issue Type: Bug
> Components: Clients
> Affects Versions: 0.7.0, 0.8.0
> Reporter: Thiruvel Thirumoolan
> Assignee: Thiruvel Thirumoolan
> Fix For: 0.8.0
>
> Attachments: DatabaseLocation.patch
>
>
> # create database db with dbproperties ('hive.metastore.warehouse.dir' = 'loc');
> The above command does not set location of 'db' to 'loc'. It instead creates 'db.db' under the warehouse directory configured in hive-site.xml of CLI. Looks conflicting with HIVE-1820's expectation. If scratch dir is specified here, that is honoured.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira