You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Jeremy Huylebroeck (JIRA)" <ji...@apache.org> on 2008/10/23 02:32:44 UTC
[jira] Created: (HADOOP-4501) CLI not supporting multiline queries
from an input file
CLI not supporting multiline queries from an input file
-------------------------------------------------------
Key: HADOOP-4501
URL: https://issues.apache.org/jira/browse/HADOOP-4501
Project: Hadoop Core
Issue Type: Improvement
Components: contrib/hive
Affects Versions: 0.19.0, 0.19.1
Reporter: Jeremy Huylebroeck
Priority: Trivial
Fix For: 0.19.0, 0.19.1
calling "hive -f filename" allows to pass HQL queries from a file.
but the queries can't be on multiple lines as we would expect by reading Hive documentation.
a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4501) CLI not supporting multiline
queries from an input file
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642212#action_12642212 ]
Prasad Chakka commented on HADOOP-4501:
---------------------------------------
also, mysql client assume that a newline is a word break. so 'select\n* from tabA;' is a valid query in mysql but not in the code above or in main() function. I think we do want to make it as similar to mysql client as possible.
> CLI not supporting multiline queries from an input file
> -------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt, patch-trunk.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4501) CLI not supporting multiline queries
from an input file
Posted by "Jeremy Huylebroeck (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Huylebroeck updated HADOOP-4501:
---------------------------------------
Fix Version/s: (was: 0.19.1)
(was: 0.19.0)
0.17.3
Affects Version/s: (was: 0.19.1)
(was: 0.19.0)
0.17.3
Status: Patch Available (was: Open)
patch for CliDriver created against 0.17 code available on the facebook mirror.
http://mirror.facebook.com/facebook/hive/
> CLI not supporting multiline queries from an input file
> -------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4501) CLI not supporting multiline queries
from an input file
Posted by "Jeremy Huylebroeck (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Huylebroeck updated HADOOP-4501:
---------------------------------------
Attachment: patch2-0.17.txt
proposed new patch for 0.17:
- supports comments properly (lines started with "--")
- keeps empty lines etc... so error position is closer to reality, easier to debug.
- shows the number of line offset in the INFO log to ease debugging of the error position (not very pretty implementation but it helps)
- some cleanup
> CLI not supporting multiline queries from an input file
> -------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt, patch-trunk.txt, patch2-0.17.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4501) [Hive] CLI not supporting multiline
queries from an input file
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644820#action_12644820 ]
Ashish Thusoo commented on HADOOP-4501:
---------------------------------------
+1
Jeremy, the patch looks good to me. Thanks for the contribution.
One thing though.
1. The latest patch is generated from the directory that contains CliDriver.java so I think the hadoop QA will fail as it expects the patch to be generated from the root of the source tree.
Prasad,
is your issue resolved?
> [Hive] CLI not supporting multiline queries from an input file
> --------------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt, patch-trunk.txt, patch2-0.17.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4501) CLI not supporting multiline
queries from an input file
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642211#action_12642211 ]
Prasad Chakka commented on HADOOP-4501:
---------------------------------------
The patch looks good except for trimming that Ashish pointed out. there is some code in CliDriver.main() which does similar thing but little differently. Could you make the two uniform so that multi-line query behaves the same both from a file and from command line?
> CLI not supporting multiline queries from an input file
> -------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt, patch-trunk.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4501) [Hive] CLI not supporting multiline
queries from an input file
Posted by "Jeremy Huylebroeck (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644452#action_12644452 ]
Jeremy Huylebroeck commented on HADOOP-4501:
--------------------------------------------
your example is actually working for me in 0.17.3, at least with the patched version.
I remember observing something similar when using 0.19, but only for some time. But I was changing the code and playing with multiple combinaisons of hive/hadoop versions at the time. can you try the new patch?
> [Hive] CLI not supporting multiline queries from an input file
> --------------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt, patch-trunk.txt, patch2-0.17.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4501) CLI not supporting multiline
queries from an input file
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642207#action_12642207 ]
Ashish Thusoo commented on HADOOP-4501:
---------------------------------------
Hi Jeremy,
Thanks a bunch for the patch.
One comment though - it would be great if the patch is able to pass all the empty lines around to the parser for debuggability purposes. Otherwise, it will be impossible for the users to figure out where the error happened in a statement.
you bring up a good point about processLine calling the trim function. We should move that to the checks in processCmd and not trim the line in processLine and we should pass the command as is to qp.run(cmd) call in processCmd.
What do think?
> CLI not supporting multiline queries from an input file
> -------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt, patch-trunk.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4501) [Hive] CLI not supporting multiline
queries from an input file
Posted by "Jeremy Huylebroeck (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Huylebroeck updated HADOOP-4501:
---------------------------------------
Summary: [Hive] CLI not supporting multiline queries from an input file (was: CLI not supporting multiline queries from an input file)
> [Hive] CLI not supporting multiline queries from an input file
> --------------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt, patch-trunk.txt, patch2-0.17.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4501) CLI not supporting multiline
queries from an input file
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644451#action_12644451 ]
Prasad Chakka commented on HADOOP-4501:
---------------------------------------
oh the problem i mentioned only if you are using interactive shell. so first line enter select and on second prompt enter a.* from tabA a;
> CLI not supporting multiline queries from an input file
> -------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt, patch-trunk.txt, patch2-0.17.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4501) [Hive] CLI not supporting multiline
queries from an input file
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644826#action_12644826 ]
Prasad Chakka commented on HADOOP-4501:
---------------------------------------
yeah, my issue is resolved but see the comments above.
> [Hive] CLI not supporting multiline queries from an input file
> --------------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt, patch-trunk.txt, patch2-0.17.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-4501) CLI not supporting
multiline queries from an input file
Posted by "Jeremy Huylebroeck (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642023#action_12642023 ]
mimizone edited comment on HADOOP-4501 at 10/22/08 5:46 PM:
----------------------------------------------------------------------
uploaded patch against 0.17 hive code
(hive code available at http://mirror.facebook.com/facebook/hive/)
was (Author: mimizone):
patch against 0.17 available at http://mirror.facebook.com/facebook/hive/
> CLI not supporting multiline queries from an input file
> -------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4501) CLI not supporting multiline queries
from an input file
Posted by "Jeremy Huylebroeck (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Huylebroeck updated HADOOP-4501:
---------------------------------------
Attachment: patch-trunk.txt
uploaded a patch for CliDriver for hadoop trunk
> CLI not supporting multiline queries from an input file
> -------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt, patch-trunk.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4501) CLI not supporting multiline queries
from an input file
Posted by "Jeremy Huylebroeck (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Huylebroeck updated HADOOP-4501:
---------------------------------------
Comment: was deleted
> CLI not supporting multiline queries from an input file
> -------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4501) CLI not supporting multiline queries
from an input file
Posted by "Jeremy Huylebroeck (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Huylebroeck updated HADOOP-4501:
---------------------------------------
Attachment: patch-0.17.txt
patch against 0.17 available at http://mirror.facebook.com/facebook/hive/
> CLI not supporting multiline queries from an input file
> -------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4501) [Hive] CLI not supporting multiline
queries from an input file
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644818#action_12644818 ]
Prasad Chakka commented on HADOOP-4501:
---------------------------------------
Hi Jeremy,
Couple of issues
1) Should line 166 be in inside the if (m.matches()) condition?
2) line 185: i think if loop gets executed even if the last line doesn't end with a ';'
3) I think processLine should be used instead of processCmd. Otherwise the code can't handle cmds like 'select a.* from tabA a; select \nb.* from tabB b'
4) also, could you update the patch with new code available for 0.17. There have been some changes that make the commands case insensitive.
It will be good if you can make similar changes to code that handles interactive commands (in main()). If not we can do that in a separate JIRA as well.
> [Hive] CLI not supporting multiline queries from an input file
> --------------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt, patch-trunk.txt, patch2-0.17.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4501) CLI not supporting multiline
queries from an input file
Posted by "Jeremy Huylebroeck (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644449#action_12644449 ]
Jeremy Huylebroeck commented on HADOOP-4501:
--------------------------------------------
Prasad, I don't observe the issue with something like 'select\n* tabl;'
I do all the time this in my hql files without problems.
do you have an actual example?
> CLI not supporting multiline queries from an input file
> -------------------------------------------------------
>
> Key: HADOOP-4501
> URL: https://issues.apache.org/jira/browse/HADOOP-4501
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hive
> Affects Versions: 0.17.3
> Reporter: Jeremy Huylebroeck
> Priority: Trivial
> Fix For: 0.17.3
>
> Attachments: patch-0.17.txt, patch-trunk.txt, patch2-0.17.txt
>
>
> calling "hive -f filename" allows to pass HQL queries from a file.
> but the queries can't be on multiple lines as we would expect by reading Hive documentation.
> a proposed fix is to modify CliDriver.processReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.