You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "kalyan ram (Created) (JIRA)" <ji...@apache.org> on 2011/10/04 16:48:34 UTC
[jira] [Created] (HIVE-2478) Support dry run option in hive
Support dry run option in hive
------------------------------
Key: HIVE-2478
URL: https://issues.apache.org/jira/browse/HIVE-2478
Project: Hive
Issue Type: Improvement
Components: Configuration
Reporter: kalyan ram
Priority: Minor
Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "John Sichi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146571#comment-13146571 ]
John Sichi commented on HIVE-2478:
----------------------------------
If we're going to add this, it would be best to generalize it so that you can choose what phase to stop after, e.g.
hive.exec.dryrun={off,parse,analyze,plan}
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "John Sichi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172639#comment-13172639 ]
John Sichi commented on HIVE-2478:
----------------------------------
Could you abandon D921 and create a new review request with --jira so that they get linked properly from the start?
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "Sushanth Sowmyan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146396#comment-13146396 ]
Sushanth Sowmyan commented on HIVE-2478:
----------------------------------------
Hi,
I have a question - does EXPLAIN execute any user-specified code? I'm faced with a environment issue where I don't want to execute any user-specified code (UDFs,etc) or run anything, merely to syntax check. Would EXPLAIN still fit my need?
Thanks!
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178966#comment-13178966 ]
Phabricator commented on HIVE-2478:
-----------------------------------
khorgath has commented on the revision "HIVE-2478 [jira] Support dry run option in hive".
REVISION DETAIL
https://reviews.facebook.net/D927
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Assignee: Sushanth Sowmyan
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch, HIVE-2478.D927.1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "Sushanth Sowmyan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172644#comment-13172644 ]
Sushanth Sowmyan commented on HIVE-2478:
----------------------------------------
Ok, trying.
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "Sushanth Sowmyan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172656#comment-13172656 ]
Sushanth Sowmyan commented on HIVE-2478:
----------------------------------------
That worked - thanks!
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch, HIVE-2478.D927.1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2478) Support dry run option in hive
Posted by "kalyan ram (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
kalyan ram updated HIVE-2478:
-----------------------------
Affects Version/s: 0.9.0
Status: Patch Available (was: Open)
Adding a sample patch <HIVE-2478-1.patch> for this improvement. Let me know if this is sufficent enough for the feature.
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2478) Support dry run option in hive
Posted by "Sushanth Sowmyan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sushanth Sowmyan updated HIVE-2478:
-----------------------------------
Attachment: HIVE-2478-3.patch
Patch with {off,parse,analyze,plan} change detailed above.
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "Sushanth Sowmyan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172651#comment-13172651 ]
Sushanth Sowmyan commented on HIVE-2478:
----------------------------------------
I'm not quite sure how to abandon D921 yet (looking through arcanist manpage)
But I generated again with --jira, and asked it to create a new revision - it created it at D927 : https://reviews.facebook.net/D927
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch, HIVE-2478.D927.1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "kalyan ram (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120890#comment-13120890 ]
kalyan ram commented on HIVE-2478:
----------------------------------
Yes we can use EXPLAIN as well. In MySql older versions for some queries having sub queries <explain> was executing the sub queries so I was thinking on those terms. So in Hive is it that we never execute the query if EXPLAIN is present ? This feature might still be useful in remote cases where you can't go and change your queries. For example if you want to check if your queries will work against a new version of hive we can set this property and check without modifying the query.
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178965#comment-13178965 ]
Phabricator commented on HIVE-2478:
-----------------------------------
khorgath has commented on the revision "HIVE-2478 [jira] Support dry run option in hive".
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:135-140 PLAN is not explicitly used anywhere in code, because we need to only check 3 times to establish the 4 states.
The first two checks happen in the compile part :
First check is for PARSE, if it's PARSE, it's weeded out and if not, it continues.
Second check is for ANALYZE, if it's that, it's weeded out and if not, it continues.
The third check is after the compile, and thus, done for whether it's not-OFF(because both the PARSE and ANALYZE returns return here too, and we weed out all the non-OFF cases there.)
I think I agree that an empty string would serve better than OFF, looking to change that.
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:558 Ok, fixing along with trying to make OFF an empty string in the next patch.
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:423 Good spot, changing.
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:452 Yup. Without this, it does, because it tries to read .getFields() on the schema. Adding comment.
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:479-488 Oh, ok - I wasn't entirely sure of that, given the LOG.info proclaiming semantic analysis was complete before the validation. Would the plan validation not count as part of the plan phase?
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:944 Disagree. PARSE, ANALYZE and PLAN all come up to here, and need to be rejected. !OFF is the correct check here.
ql/src/test/queries/clientnegative/dryrun_bad_fetch_serde.q:1 Ok, doing so.
REVISION DETAIL
https://reviews.facebook.net/D927
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Assignee: Sushanth Sowmyan
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch, HIVE-2478.D927.1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179239#comment-13179239 ]
Phabricator commented on HIVE-2478:
-----------------------------------
khorgath has commented on the revision "HIVE-2478 [jira] Support dry run option in hive".
INLINE COMMENTS
ql/src/test/queries/clientnegative/dryrun_bad_fetch_serde.q:1 Hmm.. ok, this is funny - if I try adding comments (using double hyphen) to the .q script, I get parse failures from the testcase runner. And a number of other .q tests don't have comments either, so I don't see what format I can use to comment.
How do I add comments here?
-- comment : does not work
# comment : does not work
// comment : does not work
/* comment */ : does not work
REVISION DETAIL
https://reviews.facebook.net/D927
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Assignee: Sushanth Sowmyan
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch, HIVE-2478.D927.1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2478) Support dry run option in hive
Posted by "Sushanth Sowmyan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sushanth Sowmyan updated HIVE-2478:
-----------------------------------
Attachment: HIVE-2478-2.patch
Would this modification of Kalyan's patch do the trick then?
(Tests not added yet, will do so if intent is okay)
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2478) Support dry run option in hive
Posted by "John Sichi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-2478:
-----------------------------
Status: Open (was: Patch Available)
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "Sushanth Sowmyan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146575#comment-13146575 ]
Sushanth Sowmyan commented on HIVE-2478:
----------------------------------------
I like that idea! Ok, will look into it.
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "Sushanth Sowmyan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172542#comment-13172542 ]
Sushanth Sowmyan commented on HIVE-2478:
----------------------------------------
Added phabricator review at https://reviews.facebook.net/D921 (Not familiar enough with the new process to know if I'm doing it right - please correct if needed - Thanks!)
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "John Sichi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121114#comment-13121114 ]
John Sichi commented on HIVE-2478:
----------------------------------
Currently, Hive does not execute anything as part of EXPLAIN.
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2478) Support dry run option in hive
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2478:
------------------------------
Attachment: HIVE-2478.D927.1.patch
khorgath requested code review of "HIVE-2478 [jira] Support dry run option in hive".
Reviewers: JIRA
Adding a hive.dry.run parameter that takes the following values : {off,parse,analyze,plan}
They have the following behaviours:
off : Nothing special, run query as expected
parse : Return after parse finishes (acts as a syntax check)
analyze : Return after semantic analysis finishes
plan : Return after plan generation.
Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D927
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/test/results/clientnegative/dryrun_bad_parse.q.out
ql/src/test/results/clientnegative/dryrun_nonexistant_table.q.out
ql/src/test/results/clientnegative/dryrun_bad_fetch_serde.q.out
ql/src/test/results/clientpositive/dryrun4.q.out
ql/src/test/results/clientpositive/dryrun_nonexistant_table_parse_ok.q.out
ql/src/test/results/clientpositive/dryrun_bad_fetch_serde_analyze_ok.q.out
ql/src/test/results/clientpositive/dryrun1.q.out
ql/src/test/results/clientpositive/dryrun2.q.out
ql/src/test/results/clientpositive/dryrun3.q.out
ql/src/test/queries/clientnegative/dryrun_bad_fetch_serde.q
ql/src/test/queries/clientnegative/dryrun_nonexistant_table.q
ql/src/test/queries/clientnegative/dryrun_bad_parse.q
ql/src/test/queries/clientpositive/dryrun_nonexistant_table_parse_ok.q
ql/src/test/queries/clientpositive/dryrun_bad_fetch_serde_analyze_ok.q
ql/src/test/queries/clientpositive/dryrun1.q
ql/src/test/queries/clientpositive/dryrun2.q
ql/src/test/queries/clientpositive/dryrun3.q
ql/src/test/queries/clientpositive/dryrun4.q
ql/src/java/org/apache/hadoop/hive/ql/Driver.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/1935/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch, HIVE-2478.D927.1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "John Sichi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120272#comment-13120272 ]
John Sichi commented on HIVE-2478:
----------------------------------
Can't you already use EXPLAIN for this purpose?
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "John Sichi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172652#comment-13172652 ]
John Sichi commented on HIVE-2478:
----------------------------------
You can abandon it from the Phabricator UI...use the dropdown at the bottom of the page and then click the button.
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch, HIVE-2478.D927.1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2478) Support dry run option in hive
Posted by "John Sichi (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi reassigned HIVE-2478:
--------------------------------
Assignee: Sushanth Sowmyan
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Assignee: Sushanth Sowmyan
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch, HIVE-2478.D927.1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175012#comment-13175012 ]
Phabricator commented on HIVE-2478:
-----------------------------------
ashutoshc has requested changes to the revision "HIVE-2478 [jira] Support dry run option in hive".
INLINE COMMENTS
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:558 Name the property as hive.exec.dryrun. Also, add it in hive-default.xml.template
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:135-140 There seems to be a confusion about these different mode means. You have not used PLAN anywhere. Also, 'off' is confusing. I don't think there is any need of OFF. Allow three values: PARSE, ANALYZE & PLAN. and then match these against the user provided value. In hiveConf, default value returned is empty string "".
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:423 Use, conf.getVar() instead.
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:452 Without thus does it result in NPE later? Because returned schema is empty object. It will be good to add that in comment, if that is so.
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:479-488 I think you want to end this phase after sem.validate().
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:944 This should be DryMode.PLAN
ql/src/test/queries/clientnegative/dryrun_bad_fetch_serde.q:1 Please add some comments at the top of script, explaining which phase should succeed and which should fail in these tests. You can also come up with better filenaemes for these tests which explains the intent.
This applies for all tests
REVISION DETAIL
https://reviews.facebook.net/D927
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Assignee: Sushanth Sowmyan
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch, HIVE-2478.D927.1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2478) Support dry run option in hive
Posted by "kalyan ram (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
kalyan ram updated HIVE-2478:
-----------------------------
Attachment: HIVE-2478-1.patch
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "John Sichi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146486#comment-13146486 ]
John Sichi commented on HIVE-2478:
----------------------------------
Yes, it does execute user-defined code, e.g. for resolving the types of UDF invocations. Without that, you'd have to stop immediately after ANTLR parsing (pure syntax check, no semantic analysis).
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Priority: Minor
> Attachments: HIVE-2478-1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183428#comment-13183428 ]
Phabricator commented on HIVE-2478:
-----------------------------------
ashutoshc has commented on the revision "HIVE-2478 [jira] Support dry run option in hive".
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:479-488 Analysis would be complete but validation won't. I think when some wants a dryrun till analysis they want to know that whether query is semantically valid or not and not just whether semantic analysis phase could be completed successfully.
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:944 Didn't get. How come Parse, Analyze get here? I was assuming execution returns if dry mode is set to one of these modes. Also, now that we have agreed to get rid of OFF, this needs to be modified anyways. If you still plan not to use PLAN, then I think that should be removed from enum, since else it sounds confusing to me.
ql/src/test/queries/clientnegative/dryrun_bad_fetch_serde.q:1 Try:
-- comment
Note don't put ; at the end of line.
REVISION DETAIL
https://reviews.facebook.net/D927
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Assignee: Sushanth Sowmyan
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch, HIVE-2478.D927.1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2478) Support dry run option in hive
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172657#comment-13172657 ]
Phabricator commented on HIVE-2478:
-----------------------------------
khorgath has added reviewers to the revision "HIVE-2478 [jira] Support dry run option in hive".
Added Reviewers: jsichi
REVISION DETAIL
https://reviews.facebook.net/D927
> Support dry run option in hive
> ------------------------------
>
> Key: HIVE-2478
> URL: https://issues.apache.org/jira/browse/HIVE-2478
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 0.9.0
> Reporter: kalyan ram
> Assignee: Sushanth Sowmyan
> Priority: Minor
> Attachments: HIVE-2478-1.patch, HIVE-2478-2.patch, HIVE-2478-3.patch, HIVE-2478.D927.1.patch
>
>
> Hive currently doesn't support a dry run option. For some complex queries we just want to verify the query syntax initally before running it. A dry run option where just the parsing is done without actual execution is a good option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira