You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Eric Hwang (JIRA)" <ji...@apache.org> on 2009/08/13 03:29:14 UTC
[jira] Created: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Move parsing of multiple queries from CLI into Hive parser
----------------------------------------------------------
Key: HIVE-753
URL: https://issues.apache.org/jira/browse/HIVE-753
Project: Hadoop Hive
Issue Type: Sub-task
Reporter: Eric Hwang
Assignee: Eric Hwang
Priority: Minor
CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742938#action_12742938 ]
Raghotham Murthy commented on HIVE-753:
---------------------------------------
I spoke to Eric about this. His changes dont handle SET statements. So, if there are SET statements between two queries, the parser will error out. Also, it is not clear what the behavior should be if there are multiple SELECT statements (no INSERT). The patch currently silently just picks up the schema of the last query. If that is the case, should we even run the intermediate SELECT queries? (answer: potentially, that they may have side effects). Maybe we should return a list of ResultSets, one per statement. Or, error out if there are multiple SELECT statements submitted at once.
> Move parsing of multiple queries from CLI into Hive parser
> ----------------------------------------------------------
>
> Key: HIVE-753
> URL: https://issues.apache.org/jira/browse/HIVE-753
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Eric Hwang
> Assignee: Eric Hwang
> Priority: Minor
> Attachments: HIVE-753.1.patch
>
>
> CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Posted by "Eric Hwang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742999#action_12742999 ]
Eric Hwang commented on HIVE-753:
---------------------------------
@ Ashish - If you are referring to the Driver's original getPlan() method, I do not believe there are any callers of that method.
@ Raghu - Returning a list of ResultSets would also work. I do not believe that we should enforce whether or not the caller should be able to run multiple SELECT statements. If we were to add this ability to either error out or silence intermediate SELECT statements without side effects, then the SemanticAnalyzer will need to be able to classify statments w/ and w/o side effects.
Ideally, I think that the CLI driver should have its own grammar that extends Hive's grammar, allowing it to strip out CLI specific commands before passing it to Hive. Hive should only have to understand sequences of Hive queries. Any caller of this driver should be responsible for doing its own parsing and stripping out its own specific commands.
> Move parsing of multiple queries from CLI into Hive parser
> ----------------------------------------------------------
>
> Key: HIVE-753
> URL: https://issues.apache.org/jira/browse/HIVE-753
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Eric Hwang
> Assignee: Eric Hwang
> Priority: Minor
> Attachments: HIVE-753.1.patch
>
>
> CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743494#action_12743494 ]
Ashish Thusoo commented on HIVE-753:
------------------------------------
Raghu/Edward,
Does this sound reasonable? Does it address your concerns?
> Move parsing of multiple queries from CLI into Hive parser
> ----------------------------------------------------------
>
> Key: HIVE-753
> URL: https://issues.apache.org/jira/browse/HIVE-753
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Eric Hwang
> Assignee: Eric Hwang
> Priority: Minor
> Attachments: HIVE-753.1.patch
>
>
> CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-753:
----------------------------
Status: Open (was: Patch Available)
Instead of the current approach, this whole thing should be made extensible.
CliDriver submits the list of statements to the server - which breaks it into a list of statements and returns them back to the CliDriver -
The CliDriver still needs some logic to see if the statement needs special execution (for example '!') - if no, it parses and executes the
statement on the server.
This way, the HWI server (or any other client) does not need any special processing - SET should be added in the grammar.
> Move parsing of multiple queries from CLI into Hive parser
> ----------------------------------------------------------
>
> Key: HIVE-753
> URL: https://issues.apache.org/jira/browse/HIVE-753
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Eric Hwang
> Assignee: Eric Hwang
> Priority: Minor
> Attachments: HIVE-753.1.patch
>
>
> CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Posted by "Eric Hwang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744186#action_12744186 ]
Eric Hwang commented on HIVE-753:
---------------------------------
I have linked this issue to a new JIRA for SET commands (HIVE-761). I'll see if I can add the list of ResultSets, but this may be delayed as I have some more pressing issues to attend to this week. If I can't get around to it, I'll file another JIRA for the list of ResultSets.
> Move parsing of multiple queries from CLI into Hive parser
> ----------------------------------------------------------
>
> Key: HIVE-753
> URL: https://issues.apache.org/jira/browse/HIVE-753
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Eric Hwang
> Assignee: Eric Hwang
> Priority: Minor
> Attachments: HIVE-753.1.patch
>
>
> CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743532#action_12743532 ]
Edward Capriolo commented on HIVE-753:
--------------------------------------
Ashish,
Yes. I am cool with the changes. You are correct the client can still execute in the same manner and a smart client could batch up the statements.
> Move parsing of multiple queries from CLI into Hive parser
> ----------------------------------------------------------
>
> Key: HIVE-753
> URL: https://issues.apache.org/jira/browse/HIVE-753
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Eric Hwang
> Assignee: Eric Hwang
> Priority: Minor
> Attachments: HIVE-753.1.patch
>
>
> CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Posted by "Eric Hwang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Hwang updated HIVE-753:
----------------------------
Status: Patch Available (was: Open)
> Move parsing of multiple queries from CLI into Hive parser
> ----------------------------------------------------------
>
> Key: HIVE-753
> URL: https://issues.apache.org/jira/browse/HIVE-753
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Eric Hwang
> Assignee: Eric Hwang
> Priority: Minor
> Attachments: HIVE-753.1.patch
>
>
> CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742926#action_12742926 ]
Ashish Thusoo commented on HIVE-753:
------------------------------------
Eric,
Changes look fine to me. I have one question about who uses getQueryPlan()?
> Move parsing of multiple queries from CLI into Hive parser
> ----------------------------------------------------------
>
> Key: HIVE-753
> URL: https://issues.apache.org/jira/browse/HIVE-753
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Eric Hwang
> Assignee: Eric Hwang
> Priority: Minor
> Attachments: HIVE-753.1.patch
>
>
> CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743002#action_12743002 ]
Edward Capriolo commented on HIVE-753:
--------------------------------------
Eric,
I see what you are saying, consider a query like this
{noformat}
select * from table;
! ls ;
dfs -ls / ;
select * from table;
{noformat}
For the results to come back in order entered the query processor and CliDriver would have to shift control back and forth during the sequence. Otherwise the results might come back like.
{noformat}
! ls ;
dfs -ls / ;
select * from table;
select * from table;
{noformat}
I realize that ! and dfs are quick helpers and not part of HiveQL. So I agree with you the CLI Driver should strip out commands, but it also should be able to insert the results back into the result stream in the correct order, or we agree that they will be executed before or after the QL. How should it work?
> Move parsing of multiple queries from CLI into Hive parser
> ----------------------------------------------------------
>
> Key: HIVE-753
> URL: https://issues.apache.org/jira/browse/HIVE-753
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Eric Hwang
> Assignee: Eric Hwang
> Priority: Minor
> Attachments: HIVE-753.1.patch
>
>
> CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743008#action_12743008 ]
Ashish Thusoo commented on HIVE-753:
------------------------------------
Actually Eric and I had a long conversation about this yesterday. His immediate use case was for odbc to be able to support a sequence of queries and not necessarily set functions. We could eventually push in the set functions as well thorough odbc but that is more of a good to have at this point I think.
Regarding this particular change, if there is client specific syntax e.g the ! command or the dfs command, I feel that should be handled in the clients and not in the driver grammar unless we state that those features are not client specific. All this means is that it would be the responsibility of the client to carve out the correct sequence of hiveql statements to pass to the driver. It is for that reason the cli driver continues to send one statement at a time to Driver in this patch. The ordering issue would not arise at all in the above example as the client will process one statement at a time and only pass the selects to the driver. A more advanced client could figure out the contiguous set of such statements and then pass that over to the Driver but that is more of an optimization.
The sets would have to be eventually moved into the grammar as a number of query side optimizations are handled through those statements. Thoughts?
> Move parsing of multiple queries from CLI into Hive parser
> ----------------------------------------------------------
>
> Key: HIVE-753
> URL: https://issues.apache.org/jira/browse/HIVE-753
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Eric Hwang
> Assignee: Eric Hwang
> Priority: Minor
> Attachments: HIVE-753.1.patch
>
>
> CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742941#action_12742941 ]
Edward Capriolo commented on HIVE-753:
--------------------------------------
Raghotham's points are taken. Other the SET and ADD FILE, things like
'! ls ' or 'DFS' need to be taken into account. DFS can be handled on the server. '! 's cannot' so we have to filter those out.
> Move parsing of multiple queries from CLI into Hive parser
> ----------------------------------------------------------
>
> Key: HIVE-753
> URL: https://issues.apache.org/jira/browse/HIVE-753
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Eric Hwang
> Assignee: Eric Hwang
> Priority: Minor
> Attachments: HIVE-753.1.patch
>
>
> CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743888#action_12743888 ]
Raghotham Murthy commented on HIVE-753:
---------------------------------------
I am ok with not supporting SET commands in the grammar for now (Eric can you file a jira for that?). But, its probably better to return a list of ResultSets, one per query statement.
> Move parsing of multiple queries from CLI into Hive parser
> ----------------------------------------------------------
>
> Key: HIVE-753
> URL: https://issues.apache.org/jira/browse/HIVE-753
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Eric Hwang
> Assignee: Eric Hwang
> Priority: Minor
> Attachments: HIVE-753.1.patch
>
>
> CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-753) Move parsing of multiple queries from
CLI into Hive parser
Posted by "Eric Hwang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Hwang updated HIVE-753:
----------------------------
Attachment: HIVE-753.1.patch
Tests were added to JDBC to test functionality.
All tests pass.
> Move parsing of multiple queries from CLI into Hive parser
> ----------------------------------------------------------
>
> Key: HIVE-753
> URL: https://issues.apache.org/jira/browse/HIVE-753
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Eric Hwang
> Assignee: Eric Hwang
> Priority: Minor
> Attachments: HIVE-753.1.patch
>
>
> CLI breaks down queries containing multiple statements and submits them to the Hive query parser one by one. This functionality should be added to the Hive language grammar so that other clients may also submit multiple queries. It will also make it easier to handle comments in Hive queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.