You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Raghotham Murthy (JIRA)" <ji...@apache.org> on 2009/06/29 21:26:47 UTC
[jira] Created: (HIVE-590) Pass type information in genFileSinkPlan
Pass type information in genFileSinkPlan
----------------------------------------
Key: HIVE-590
URL: https://issues.apache.org/jira/browse/HIVE-590
Project: Hadoop Hive
Issue Type: Bug
Components: Query Processor
Reporter: Raghotham Murthy
Assignee: Namit Jain
Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
{code}
struct ResultSchema {
// column names, types, comments
1: list<hive_metastore.FieldSchema> fieldSchemas,
// delimiters etc
2: map<string, string> properties
}
{code}
Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-590) Pass type information in
genFileSinkPlan
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726268#action_12726268 ]
Raghotham Murthy commented on HIVE-590:
---------------------------------------
changes look good for the most part. a few comments:
* we should provide an additional method in HiveInterface.java which returns a SerDe given a Schema. If we implement it once - we can reuse it in all the tests and in HiveResultSet.java instead of duplicating the serde creation logic everywhere (including the dsp.setProperty() calls). We can choose to do it in a later jira.
* use standard name for get method: public boolean isUseJSONSerialize() {
* includes unnecessary changes to test results
> Pass type information in genFileSinkPlan
> ----------------------------------------
>
> Key: HIVE-590
> URL: https://issues.apache.org/jira/browse/HIVE-590
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Raghotham Murthy
> Assignee: Namit Jain
> Attachments: hive.590.1.patch
>
>
> Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
> Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
> {code}
> struct ResultSchema {
> // column names, types, comments
> 1: list<hive_metastore.FieldSchema> fieldSchemas,
> // delimiters etc
> 2: map<string, string> properties
> }
> {code}
> Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-590) Pass type information in
genFileSinkPlan
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726569#action_12726569 ]
Prasad Chakka commented on HIVE-590:
------------------------------------
I don't think the new Schema object should be in metastore as it is not going to be used by it. There is no corresponding object in metadb. It will be used only by HiveServer so it should be in the thrift interface of HiveServer. Namit and I had a chat about it and he agreed to move it there.
> Pass type information in genFileSinkPlan
> ----------------------------------------
>
> Key: HIVE-590
> URL: https://issues.apache.org/jira/browse/HIVE-590
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Raghotham Murthy
> Assignee: Namit Jain
> Fix For: 0.4.0
>
> Attachments: hive.590.1.patch, hive.590.2.patch
>
>
> Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
> Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
> {code}
> struct ResultSchema {
> // column names, types, comments
> 1: list<hive_metastore.FieldSchema> fieldSchemas,
> // delimiters etc
> 2: map<string, string> properties
> }
> {code}
> Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-590) Pass type information in genFileSinkPlan
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-590:
----------------------------
Attachment: hive.590.2.patch
resolved conflicts with 136
> Pass type information in genFileSinkPlan
> ----------------------------------------
>
> Key: HIVE-590
> URL: https://issues.apache.org/jira/browse/HIVE-590
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Raghotham Murthy
> Assignee: Namit Jain
> Attachments: hive.590.1.patch, hive.590.2.patch
>
>
> Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
> Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
> {code}
> struct ResultSchema {
> // column names, types, comments
> 1: list<hive_metastore.FieldSchema> fieldSchemas,
> // delimiters etc
> 2: map<string, string> properties
> }
> {code}
> Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-590) Pass type information in
genFileSinkPlan
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725347#action_12725347 ]
Namit Jain commented on HIVE-590:
---------------------------------
This has the following requirements:
1. Pass the type info. in getFileSinkPlan
2. Enhance LazySerde to use json to serialize for complex types under certain conditions
3. Return a new thrift type instead of string from Driver.getSchema()
> Pass type information in genFileSinkPlan
> ----------------------------------------
>
> Key: HIVE-590
> URL: https://issues.apache.org/jira/browse/HIVE-590
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Raghotham Murthy
> Assignee: Namit Jain
>
> Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
> Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
> {code}
> struct ResultSchema {
> // column names, types, comments
> 1: list<hive_metastore.FieldSchema> fieldSchemas,
> // delimiters etc
> 2: map<string, string> properties
> }
> {code}
> Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-590) Pass type information in genFileSinkPlan
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-590:
----------------------------
Attachment: hive.590.1.patch
> Pass type information in genFileSinkPlan
> ----------------------------------------
>
> Key: HIVE-590
> URL: https://issues.apache.org/jira/browse/HIVE-590
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Raghotham Murthy
> Assignee: Namit Jain
> Attachments: hive.590.1.patch
>
>
> Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
> Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
> {code}
> struct ResultSchema {
> // column names, types, comments
> 1: list<hive_metastore.FieldSchema> fieldSchemas,
> // delimiters etc
> 2: map<string, string> properties
> }
> {code}
> Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-590) Pass type information in genFileSinkPlan
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-590:
----------------------------
Status: Patch Available (was: Open)
> Pass type information in genFileSinkPlan
> ----------------------------------------
>
> Key: HIVE-590
> URL: https://issues.apache.org/jira/browse/HIVE-590
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Raghotham Murthy
> Assignee: Namit Jain
> Attachments: hive.590.1.patch
>
>
> Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
> Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
> {code}
> struct ResultSchema {
> // column names, types, comments
> 1: list<hive_metastore.FieldSchema> fieldSchemas,
> // delimiters etc
> 2: map<string, string> properties
> }
> {code}
> Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-590) Pass type information in
genFileSinkPlan
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726279#action_12726279 ]
Namit Jain commented on HIVE-590:
---------------------------------
2. I am already using a standard name - it was generated by eclipse
> Pass type information in genFileSinkPlan
> ----------------------------------------
>
> Key: HIVE-590
> URL: https://issues.apache.org/jira/browse/HIVE-590
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Raghotham Murthy
> Assignee: Namit Jain
> Attachments: hive.590.1.patch
>
>
> Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
> Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
> {code}
> struct ResultSchema {
> // column names, types, comments
> 1: list<hive_metastore.FieldSchema> fieldSchemas,
> // delimiters etc
> 2: map<string, string> properties
> }
> {code}
> Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-590) Pass type information in
genFileSinkPlan
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726688#action_12726688 ]
Raghotham Murthy commented on HIVE-590:
---------------------------------------
Moving discussion to HIVE-604.
> Pass type information in genFileSinkPlan
> ----------------------------------------
>
> Key: HIVE-590
> URL: https://issues.apache.org/jira/browse/HIVE-590
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Raghotham Murthy
> Assignee: Namit Jain
> Fix For: 0.4.0
>
> Attachments: hive.590.1.patch, hive.590.2.patch
>
>
> Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
> Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
> {code}
> struct ResultSchema {
> // column names, types, comments
> 1: list<hive_metastore.FieldSchema> fieldSchemas,
> // delimiters etc
> 2: map<string, string> properties
> }
> {code}
> Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-590) Pass type information in
genFileSinkPlan
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726572#action_12726572 ]
Prasad Chakka commented on HIVE-590:
------------------------------------
IMO, HiveServer should return a Table object for the results set instead of this new object.
> Pass type information in genFileSinkPlan
> ----------------------------------------
>
> Key: HIVE-590
> URL: https://issues.apache.org/jira/browse/HIVE-590
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Raghotham Murthy
> Assignee: Namit Jain
> Fix For: 0.4.0
>
> Attachments: hive.590.1.patch, hive.590.2.patch
>
>
> Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
> Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
> {code}
> struct ResultSchema {
> // column names, types, comments
> 1: list<hive_metastore.FieldSchema> fieldSchemas,
> // delimiters etc
> 2: map<string, string> properties
> }
> {code}
> Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-590) Pass type information in genFileSinkPlan
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-590:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
Release Note:
Pass type information in genFileSinkPlan and improve getSchema call in Driver. (Namit Jain via rmurthy)
Status: Resolved (was: Patch Available)
Committed. Thanks Namit!
> Pass type information in genFileSinkPlan
> ----------------------------------------
>
> Key: HIVE-590
> URL: https://issues.apache.org/jira/browse/HIVE-590
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Raghotham Murthy
> Assignee: Namit Jain
> Fix For: 0.4.0
>
> Attachments: hive.590.1.patch, hive.590.2.patch
>
>
> Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
> Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
> {code}
> struct ResultSchema {
> // column names, types, comments
> 1: list<hive_metastore.FieldSchema> fieldSchemas,
> // delimiters etc
> 2: map<string, string> properties
> }
> {code}
> Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-590) Pass type information in
genFileSinkPlan
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726277#action_12726277 ]
Namit Jain commented on HIVE-590:
---------------------------------
1. Will do in a follow-up
2. will do
3. All those results have changed - since the plan has changed. There is no extra stuff
> Pass type information in genFileSinkPlan
> ----------------------------------------
>
> Key: HIVE-590
> URL: https://issues.apache.org/jira/browse/HIVE-590
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Raghotham Murthy
> Assignee: Namit Jain
> Attachments: hive.590.1.patch
>
>
> Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
> Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
> {code}
> struct ResultSchema {
> // column names, types, comments
> 1: list<hive_metastore.FieldSchema> fieldSchemas,
> // delimiters etc
> 2: map<string, string> properties
> }
> {code}
> Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-590) Pass type information in
genFileSinkPlan
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726576#action_12726576 ]
Raghotham Murthy commented on HIVE-590:
---------------------------------------
In most cases, HiveServer is called to get information about SELECT queries which dont create tables. Also, these transient query results dont have partitionkeys, bucketcols, sortcols etc. So a Table object seems to be an overkill.
I do agree that we should support a getTableObject or some such metastore api call which returns the Table object for a table in the warehouse. This call can then be used instead of doing a 'describe extended' query on a table and parsing that blob that gets returned.
> Pass type information in genFileSinkPlan
> ----------------------------------------
>
> Key: HIVE-590
> URL: https://issues.apache.org/jira/browse/HIVE-590
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Raghotham Murthy
> Assignee: Namit Jain
> Fix For: 0.4.0
>
> Attachments: hive.590.1.patch, hive.590.2.patch
>
>
> Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
> Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
> {code}
> struct ResultSchema {
> // column names, types, comments
> 1: list<hive_metastore.FieldSchema> fieldSchemas,
> // delimiters etc
> 2: map<string, string> properties
> }
> {code}
> Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-590) Pass type information in
genFileSinkPlan
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726586#action_12726586 ]
Prasad Chakka commented on HIVE-590:
------------------------------------
well, there is get_table() call that returns Table object. You can use that one.
> Pass type information in genFileSinkPlan
> ----------------------------------------
>
> Key: HIVE-590
> URL: https://issues.apache.org/jira/browse/HIVE-590
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Raghotham Murthy
> Assignee: Namit Jain
> Fix For: 0.4.0
>
> Attachments: hive.590.1.patch, hive.590.2.patch
>
>
> Right now only column names are being passed between semanticanalyzer and fetchtask. Once type information is passed, we can use LazySerDe to serialize the data (into json) in Driver.getResults().
> Driver.getSchema() should then return a new thrift type ResultSchema instead of String:
> {code}
> struct ResultSchema {
> // column names, types, comments
> 1: list<hive_metastore.FieldSchema> fieldSchemas,
> // delimiters etc
> 2: map<string, string> properties
> }
> {code}
> Once this is done, the jdbc client can instantiate a simplified serde from the ResultSchema and parse the query results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.