You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Raghotham Murthy (JIRA)" <ji...@apache.org> on 2008/12/08 13:34:44 UTC
[jira] Created: (HIVE-132) Show table and describe results should
be read via FetchTask
Show table and describe results should be read via FetchTask
------------------------------------------------------------
Key: HIVE-132
URL: https://issues.apache.org/jira/browse/HIVE-132
Project: Hadoop Hive
Issue Type: New Feature
Components: Query Processor
Affects Versions: 0.20.0
Reporter: Raghotham Murthy
Assignee: Raghotham Murthy
Priority: Minor
Fix For: 0.20.0
Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-132) Show table and describe results should
be read via FetchTask
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-132:
----------------------------------
Attachment: hive-132.2.patch
In this patch:
1. added support for show partitions
2. describe extended is also supported via FetchTask - the detailed information becomes an additional row in the output.
3. added comments
4. captured new outputs dor
> Show table and describe results should be read via FetchTask
> ------------------------------------------------------------
>
> Key: HIVE-132
> URL: https://issues.apache.org/jira/browse/HIVE-132
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.2.0
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: hive-132.1.patch, hive-132.2.patch
>
>
> Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
> Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-132) Show table and describe results should
be read via FetchTask
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658024#action_12658024 ]
Namit Jain commented on HIVE-132:
---------------------------------
1. The same change needs to be done for showParitions also in DDLTask
2. Remove all the context specific code from Driver.fetch() - fetch only needs to invoke fetchtask now.
3. Can you add more comments ?
> Show table and describe results should be read via FetchTask
> ------------------------------------------------------------
>
> Key: HIVE-132
> URL: https://issues.apache.org/jira/browse/HIVE-132
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.2.0
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: hive-132.1.patch
>
>
> Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
> Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-132) Show table and describe results should
be read via FetchTask
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658033#action_12658033 ]
Raghotham Murthy commented on HIVE-132:
---------------------------------------
@Namit
1. I was just concentrating on getting show tables and describe working first. Will add show partitions next.
2. I cant remove context specific code from Driver.fetch() since 'extended' option returns free-form results. It might be a good idea to return structured results for extended as well as explain statements.
3. Will add more comments.
> Show table and describe results should be read via FetchTask
> ------------------------------------------------------------
>
> Key: HIVE-132
> URL: https://issues.apache.org/jira/browse/HIVE-132
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.2.0
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: hive-132.1.patch
>
>
> Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
> Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-132) Show table and describe results should
be read via FetchTask
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688972#action_12688972 ]
Prasad Chakka commented on HIVE-132:
------------------------------------
TestNegativeCliDriver.vm:74
dailed reappears?
I don't understand why the function getSerDe() is needed. The return value is being used to serialize partition names etc. Why?
> Show table and describe results should be read via FetchTask
> ------------------------------------------------------------
>
> Key: HIVE-132
> URL: https://issues.apache.org/jira/browse/HIVE-132
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.2.0
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: hive-132.1.patch, hive-132.2.patch, hive-132.3.patch
>
>
> Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
> Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-132) Show table and describe results should
be read via FetchTask
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660848#action_12660848 ]
Raghotham Murthy commented on HIVE-132:
---------------------------------------
The table Information is actually another row with the same schema: column-name = "Table Information", column-type = <the extended information>, column-description = null. Without this, its not possible to use the FetchTask to get to the extended data. I also added an additional empty row (all columns are null). I could hack it such that that a ':' is printed after "Table Information" and write out the actual information in another row. But I cant get rid of the extra tab since I have to print all rows using the same schema. I think its better to figure out a schema for the extended information rather than hacking this more to make it look better in unstructured form.
DDLSemanticAnalyzer: when i set the null string to empty string '\N' was being displayed. So, I set it to space instead. I was not sure what to change to make empty string work.
> Show table and describe results should be read via FetchTask
> ------------------------------------------------------------
>
> Key: HIVE-132
> URL: https://issues.apache.org/jira/browse/HIVE-132
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.2.0
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: hive-132.1.patch, hive-132.2.patch
>
>
> Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
> Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-132) Show table and describe results should
be read via FetchTask
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660770#action_12660770 ]
Ashish Thusoo commented on HIVE-132:
------------------------------------
A few nitpicks:
1. Why did we get rid of : in the explain plan output "Table Information:" has been replaced by "Table Information" without a new line. Also there seems to be extra space/tab after the column metadata is printed.
Thanks for fixing dailed to failed :).. that has been there for so long... :))
For the general code:
1. In DDLSemanticAnalyzer.java in the function createFetchTask setSerializationNullFormat is "" - should this be a null string instead or a ""
Otherwise it looks good to me.
> Show table and describe results should be read via FetchTask
> ------------------------------------------------------------
>
> Key: HIVE-132
> URL: https://issues.apache.org/jira/browse/HIVE-132
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.2.0
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: hive-132.1.patch, hive-132.2.patch
>
>
> Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
> Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-132) Show table and describe results should
be read via FetchTask
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-132:
----------------------------------
Affects Version/s: 0.2.0
Status: Patch Available (was: Open)
> Show table and describe results should be read via FetchTask
> ------------------------------------------------------------
>
> Key: HIVE-132
> URL: https://issues.apache.org/jira/browse/HIVE-132
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.2.0
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: hive-132.1.patch
>
>
> Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
> Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-132) Show table and describe results should
be read via FetchTask
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688971#action_12688971 ]
Namit Jain commented on HIVE-132:
---------------------------------
Havent looked at the changes in detail yet: some observations:
1. Use LazySerDe instead of Dynamic SerDe for the schema
2. input20.q.out seems to be garbled - can you resolve that ?
> Show table and describe results should be read via FetchTask
> ------------------------------------------------------------
>
> Key: HIVE-132
> URL: https://issues.apache.org/jira/browse/HIVE-132
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.2.0
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: hive-132.1.patch, hive-132.2.patch, hive-132.3.patch
>
>
> Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
> Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-132) Show table and describe results should
be read via FetchTask
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-132:
----------------------------------
Attachment: hive-132.1.patch
Changed show tables and describe table to return results via FetchTask. 'extended' option results in bypassing the FetchTask for the extra information.
For JDBC we need to potentially add to the syntax of show tables and describe tables to return more columns (albeit with dummy values). We need the following columns:
For show tables:
NAME_SPACE
TAB_NAME
For describe (on a set of table names):
NAME_SPACE
TAB_NAME
COL_NAME
DATA_TYPE
DATA_LEN
DATA_PREC
DATA_SCALE
> Show table and describe results should be read via FetchTask
> ------------------------------------------------------------
>
> Key: HIVE-132
> URL: https://issues.apache.org/jira/browse/HIVE-132
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.2.0
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: hive-132.1.patch
>
>
> Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
> Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-132) Show table and describe results should
be read via FetchTask
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-132:
----------------------------
Resolution: Fixed
Fix Version/s: (was: 0.2.0)
0.3.0
Release Note: HIVE-132. Show table and describe results to be read via FetchTask. (Raghotham Murthy via zshao)
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed revision 758933. Thanks Raghu!
> Show table and describe results should be read via FetchTask
> ------------------------------------------------------------
>
> Key: HIVE-132
> URL: https://issues.apache.org/jira/browse/HIVE-132
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.2.0
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Priority: Minor
> Fix For: 0.3.0
>
> Attachments: hive-132.1.patch, hive-132.2.patch, hive-132.3.patch, hive-132.4.patch
>
>
> Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
> Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-132) Show table and describe results should
be read via FetchTask
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688854#action_12688854 ]
Namit Jain commented on HIVE-132:
---------------------------------
Raghu, can you refresh and resubmit the patch - I will take a look once immediately
> Show table and describe results should be read via FetchTask
> ------------------------------------------------------------
>
> Key: HIVE-132
> URL: https://issues.apache.org/jira/browse/HIVE-132
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.2.0
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: hive-132.1.patch, hive-132.2.patch
>
>
> Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
> Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-132) Show table and describe results should
be read via FetchTask
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-132:
----------------------------------
Attachment: hive-132.3.patch
Regenerating patch. The following changes happen to the output of show tables, describe, describe extended and describe extended partition.
show tables:
The list of tables will be printed out one table in one row
describe:
The schema of the output of describe will be tab-separated <string, string, string>, irrespective of whether columns have comments or not.
describe extended:
Schema is the same as describe. Right now detailed table information is printed out as a blob in a new line. It has been changed to be printed in the same line as the string 'Detailed Table Information'.
describe extended partition:
Same as describe extended for Detailed Partition Information
> Show table and describe results should be read via FetchTask
> ------------------------------------------------------------
>
> Key: HIVE-132
> URL: https://issues.apache.org/jira/browse/HIVE-132
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.2.0
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: hive-132.1.patch, hive-132.2.patch, hive-132.3.patch
>
>
> Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
> Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-132) Show table and describe results should
be read via FetchTask
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-132:
----------------------------------
Attachment: hive-132.4.patch
Removed SerDe from DDLTask and writing out tab-separated values directly. Using LazySimpleSerDe in FetchTask. Regenerated test outputs.
> Show table and describe results should be read via FetchTask
> ------------------------------------------------------------
>
> Key: HIVE-132
> URL: https://issues.apache.org/jira/browse/HIVE-132
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.2.0
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: hive-132.1.patch, hive-132.2.patch, hive-132.3.patch, hive-132.4.patch
>
>
> Right now there is a different code path in the Driver for show tables/describe etc. By adding the FetchTask the code paths will be merged. In addition, the results of these statements will be readable via JDBC.
> Going forward, we should provide SQL access to our metastore similar to access provided by Oracle or mysql to their catalog tables. In this jira, I will just add schema information to show tables and describe queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.