You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Siying Dong (JIRA)" <ji...@apache.org> on 2011/08/17 21:27:28 UTC
[jira] [Created] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Local Mode can be more aggressive if LIMIT optimization is on
-------------------------------------------------------------
Key: HIVE-2385
URL: https://issues.apache.org/jira/browse/HIVE-2385
Project: Hive
Issue Type: Improvement
Reporter: Siying Dong
Priority: Minor
Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siying Dong updated HIVE-2385:
------------------------------
Status: Patch Available (was: Open)
> Local Mode can be more aggressive if LIMIT optimization is on
> -------------------------------------------------------------
>
> Key: HIVE-2385
> URL: https://issues.apache.org/jira/browse/HIVE-2385
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Assignee: Siying Dong
> Priority: Minor
> Attachments: HIVE-2385.1.patch, HIVE-2385.2.patch
>
>
> Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086539#comment-13086539 ]
Siying Dong commented on HIVE-2385:
-----------------------------------
I don't know why but I can't create review board using this patch.
> Local Mode can be more aggressive if LIMIT optimization is on
> -------------------------------------------------------------
>
> Key: HIVE-2385
> URL: https://issues.apache.org/jira/browse/HIVE-2385
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Assignee: Siying Dong
> Priority: Minor
> Attachments: HIVE-2385.1.patch
>
>
> Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091333#comment-13091333 ]
Carl Steinbach commented on HIVE-2385:
--------------------------------------
@Siying: autolocal1 now passes. +1. Will commit if full tests pass.
> Local Mode can be more aggressive if LIMIT optimization is on
> -------------------------------------------------------------
>
> Key: HIVE-2385
> URL: https://issues.apache.org/jira/browse/HIVE-2385
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Assignee: Siying Dong
> Priority: Minor
> Attachments: HIVE-2385.1.patch, HIVE-2385.2.patch
>
>
> Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092112#comment-13092112 ]
Hudson commented on HIVE-2385:
------------------------------
Integrated in Hive-trunk-h0.21 #918 (See [https://builds.apache.org/job/Hive-trunk-h0.21/918/])
HIVE-2385. Local Mode can be more aggressive if LIMIT optimization is on (Siying Dong via cws)
cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162202
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> Local Mode can be more aggressive if LIMIT optimization is on
> -------------------------------------------------------------
>
> Key: HIVE-2385
> URL: https://issues.apache.org/jira/browse/HIVE-2385
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Siying Dong
> Assignee: Siying Dong
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: HIVE-2385.1.patch, HIVE-2385.2.patch
>
>
> Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2385:
---------------------------------
Component/s: Query Processor
Fix Version/s: 0.8.0
> Local Mode can be more aggressive if LIMIT optimization is on
> -------------------------------------------------------------
>
> Key: HIVE-2385
> URL: https://issues.apache.org/jira/browse/HIVE-2385
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Siying Dong
> Assignee: Siying Dong
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: HIVE-2385.1.patch, HIVE-2385.2.patch
>
>
> Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siying Dong updated HIVE-2385:
------------------------------
Status: Patch Available (was: Open)
> Local Mode can be more aggressive if LIMIT optimization is on
> -------------------------------------------------------------
>
> Key: HIVE-2385
> URL: https://issues.apache.org/jira/browse/HIVE-2385
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Priority: Minor
> Attachments: HIVE-2385.1.patch
>
>
> Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siying Dong updated HIVE-2385:
------------------------------
Attachment: HIVE-2385.2.patch
Fix the bug and it passes autolocal1.q. I'm running the whole test suites now.
> Local Mode can be more aggressive if LIMIT optimization is on
> -------------------------------------------------------------
>
> Key: HIVE-2385
> URL: https://issues.apache.org/jira/browse/HIVE-2385
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Assignee: Siying Dong
> Priority: Minor
> Attachments: HIVE-2385.1.patch, HIVE-2385.2.patch
>
>
> Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siying Dong updated HIVE-2385:
------------------------------
Attachment: HIVE-2385.1.patch
Further estimate input for LIMIT when deciding local mode. Also fix a bug (won't cause wrong result) of the LIMIT optimization.
> Local Mode can be more aggressive if LIMIT optimization is on
> -------------------------------------------------------------
>
> Key: HIVE-2385
> URL: https://issues.apache.org/jira/browse/HIVE-2385
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Priority: Minor
> Attachments: HIVE-2385.1.patch
>
>
> Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siying Dong reassigned HIVE-2385:
---------------------------------
Assignee: Siying Dong
> Local Mode can be more aggressive if LIMIT optimization is on
> -------------------------------------------------------------
>
> Key: HIVE-2385
> URL: https://issues.apache.org/jira/browse/HIVE-2385
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Assignee: Siying Dong
> Priority: Minor
> Attachments: HIVE-2385.1.patch
>
>
> Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090397#comment-13090397 ]
Siying Dong commented on HIVE-2385:
-----------------------------------
It passed all the tests.
> Local Mode can be more aggressive if LIMIT optimization is on
> -------------------------------------------------------------
>
> Key: HIVE-2385
> URL: https://issues.apache.org/jira/browse/HIVE-2385
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Assignee: Siying Dong
> Priority: Minor
> Attachments: HIVE-2385.1.patch, HIVE-2385.2.patch
>
>
> Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2385:
---------------------------------
Status: Open (was: Patch Available)
@Siying: I get a test failure in TestNegativeCliDriver.autlocal1. Can you please take a look? Thanks.
> Local Mode can be more aggressive if LIMIT optimization is on
> -------------------------------------------------------------
>
> Key: HIVE-2385
> URL: https://issues.apache.org/jira/browse/HIVE-2385
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Assignee: Siying Dong
> Priority: Minor
> Attachments: HIVE-2385.1.patch
>
>
> Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2385:
---------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks Siying!
> Local Mode can be more aggressive if LIMIT optimization is on
> -------------------------------------------------------------
>
> Key: HIVE-2385
> URL: https://issues.apache.org/jira/browse/HIVE-2385
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Siying Dong
> Assignee: Siying Dong
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: HIVE-2385.1.patch, HIVE-2385.2.patch
>
>
> Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2385) Local Mode can be more aggressive if
LIMIT optimization is on
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091305#comment-13091305 ]
Siying Dong commented on HIVE-2385:
-----------------------------------
@Carl, are you still seeing tests failing?
> Local Mode can be more aggressive if LIMIT optimization is on
> -------------------------------------------------------------
>
> Key: HIVE-2385
> URL: https://issues.apache.org/jira/browse/HIVE-2385
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Assignee: Siying Dong
> Priority: Minor
> Attachments: HIVE-2385.1.patch, HIVE-2385.2.patch
>
>
> Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira