You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2010/12/22 22:47:01 UTC
[jira] Created: (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Revive partition filtering in the Hive MetaStore
------------------------------------------------
Key: HIVE-1862
URL: https://issues.apache.org/jira/browse/HIVE-1862
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.7.0
Reporter: Devaraj Das
Fix For: 0.7.0
HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980572#action_12980572 ]
Mac Yang commented on HIVE-1862:
--------------------------------
Bingo!
Ran the test with metastore in local mode and was able to reproduce the table not found error. It appears that the bug gets tickled when metastore is running in local mode, but not (at least not easily) in thrift server mode.
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979911#action_12979911 ]
Mac Yang commented on HIVE-1862:
--------------------------------
Ran the test again with the full dataset (~300k partitions in jdo2 table) and 10 concurrent processes. It ran for 30 hours without error.
The test was run against a remote metastore server, with datanucleus 2.1.1 and the partition filtering code in place (pre HIVE-1853). MySQL version is 5.1.46.
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mac Yang updated HIVE-1862:
---------------------------
Attachment: HIVE-1862.3.patch.txt
Incorporated Paul's feedback
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Assignee: Mac Yang
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, HIVE-1862.3.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Yang updated HIVE-1862:
----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed. Thanks Mac! Cool use of string manipulation. Hopefully, we'll find a workaround for those escaped partition names soon..
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Assignee: Mac Yang
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, HIVE-1862.3.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mac Yang updated HIVE-1862:
---------------------------
Attachment: HIVE-1862.1.patch.txt
Datanucleus 2.0.3 does not support the get() method on Collection, which the
partition filtering code depends on in order to retrieve the value for a
particular partition and use it for filtering.
The submitted patch is a quick work around. It uses the substring() function
to extract the partition value out of the partitionName field, and thus
eliminates the need of the get() method.
However, this approach does not work if the partition value contains special
characters. This is because the partitionName has the special characters
escaped. Hence the partition value generated using the substring() approach is
also in the escaped form. Here is the list of special characters for reference
purpose, '"', '#', '%', '\'', '*', '/', ':', '=', '?', '\\', '\u007F', '{',
']'
While this solution is incomplete, I am hoping this submission will trigger more
suggestions and ideas.
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1862:
-----------------------------
Attachment: qry-sch.Z
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977663#action_12977663 ]
Namit Jain commented on HIVE-1862:
----------------------------------
I have attached the test case which I was able to run to reproduce the problem with the higher jdo version.
Note that this is not deterministic, and may take some time to error out
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1862:
-----------------------------
Attachment: runqry
qry
invoke_runqry.sh
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1862:
---------------------------------
Labels: JDO PartitionPruner (was: )
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Assignee: Mac Yang
> Labels: JDO, PartitionPruner
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, HIVE-1862.3.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981409#action_12981409 ]
Mac Yang commented on HIVE-1862:
--------------------------------
"A quick note about the patch. The work around is implemented in ExpressionTree.java. The rest of the patch just added back old code that was removed as part of HIVE-1853."
This was a comment I made on 4/Jan/11. It's still mostly true. However, I did make some additional minor changes due to other commits in the trunk since then.
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Yang reassigned HIVE-1862:
-------------------------------
Assignee: Mac Yang
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Assignee: Mac Yang
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1862:
-----------------------------
Status: Open (was: Patch Available)
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979936#action_12979936 ]
Mac Yang commented on HIVE-1862:
--------------------------------
I did not set those parameters and just picked up from hive-default.xml,
datanucleus.transactionIsolation=read-committed
datanucleus.connectionPoolingType=DBCP
Will try the test with repeatable-read and no connection pooling and see if that makes a difference.
I'm curious why you run without connection pooling. Was it because the metadata server was running in local mode or was there some issue with DBCP?
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980894#action_12980894 ]
Mac Yang commented on HIVE-1862:
--------------------------------
Given that the issue does not seem to impact thrift server mode metastore server, I would like to propose the following approach,
- Make datanucleus version configurable at build time (similar to hadoop.version). Use 2.0.3 as the default
- Put back the code in listMPartitionsByFilter
With those changes, the behavior of a default build will be what it is today, but people also has the option to use the partition filtering function.
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983514#action_12983514 ]
Paul Yang commented on HIVE-1862:
---------------------------------
* There may be a potential bug with the string index logic:
{code}
172 String keyEqual = FileUtils.escapePathName(keyName) + "=";
173 int keyEqualLength = keyEqual.length();
174 String valString;
175 // partitionname ==> (key=value/)*(key=value)
176 if (partitionColumnCount == 1) {
177 valString = "partitionName.substring(partitionName.indexOf(\"" + keyEqual + "\")+" + keyEqualLength + ")";
178 }
179 else {
180 valString = "partitionName.substring(partitionName.indexOf(\"" + keyEqual + "\")+" + keyEqualLength + ").substring(0, partitionName.substring(partitionName.indexOf(\"" + keyEqual + "\")+" + keyEqualLength + ").indexOf(\"/\"))";
181 }
{code}
Say the partition name was ds=1/hr=2 and we want to extract the value of hr. Since there is no tailing '/', what will .indexOf(\"/\"))" return?
* Can you elaborate further on why it can't handle escaped partition names? It seems that if we escape the value that we are searching for, instead of unescaping the partition name, we should be able to handle that case.
If escaped partition names can't be handled, can we revert the code inside get_partition_names_ps() / get_partitions_ps() to return the correct, but slow results?
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12974409#action_12974409 ]
Ashutosh Chauhan commented on HIVE-1862:
----------------------------------------
Namit,
Since HIVE-1853 is marked fixed and resolved can you attach the test scripts here and instructions on how to reproduce it.
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983908#action_12983908 ]
Paul Yang commented on HIVE-1862:
---------------------------------
+1 looks good. Will commit if tests pass.
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Assignee: Mac Yang
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, HIVE-1862.3.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983598#action_12983598 ]
Mac Yang commented on HIVE-1862:
--------------------------------
Paul, thanks for the comment.
- I will add a check to catch the case where it's the last element in the path
- Escaping the value does not work for the "like" operator, for example, the pattern "p.*3" would match values like "p13". However, it would also match value "p1#" since it got truned into "p1%23"
- I will leave get_partition_names_ps()/get_partitions_ps() as the way they are in the trunk in the next patch
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mac Yang updated HIVE-1862:
---------------------------
Status: Patch Available (was: Open)
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Assignee: Mac Yang
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, HIVE-1862.3.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978874#action_12978874 ]
Mac Yang commented on HIVE-1862:
--------------------------------
Namit, thanks for posting the test.
I have run the test with partial data (36k out of 302k partitions for the jdo2 table). With six concurrent processes, the test ran for more than a day without error. Will try again after finish loading the full data set.
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12974404#action_12974404 ]
Ashutosh Chauhan commented on HIVE-1862:
----------------------------------------
I can see four possible routes going forward:
1) Fish out exact issue by coming up with a test case and submit it to jdo community and ask for a fix in the library.
Assuming 1) may take longer and is unbounded, remaining on older stable jdo version presents following possibilities.
2) Rewrite the jdo query to not make use of Collection.get() which only exists in newer jdo version and thereby required a jdo lib upgrade.
3) Refactor the partition pruning code from ql to metastore so that metastore returns a filtered list of partition
4) If 2) is not possible then, then push equality predicate to MySQL and do filtering for other predicates using 3) in Metastore.
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980202#action_12980202 ]
John Sichi commented on HIVE-1862:
----------------------------------
Many of our clients connect to the DB directly (not through the metastore Thrift server). As far as I know, there are no issues with DBCP.
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mac Yang updated HIVE-1862:
---------------------------
Status: Patch Available (was: Open)
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mac Yang updated HIVE-1862:
---------------------------
Attachment: HIVE-1862.2.patch.txt
Regenerate patch based on current trunk
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981088#action_12981088 ]
Namit Jain commented on HIVE-1862:
----------------------------------
Can you generate the patch - I am getting some conflicts while applying the patch
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977583#action_12977583 ]
Mac Yang commented on HIVE-1862:
--------------------------------
A quick note about the patch. The work around is implemented in ExpressionTree.java. The rest of the patch just added back old code that was removed as part of HIVE-1853.
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980946#action_12980946 ]
Namit Jain commented on HIVE-1862:
----------------------------------
I think the problem exists in the thrift server mode also, it is just more difficult to reproduce.
Can you run 200 concurrent clients, and see if the problem appears ?
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981992#action_12981992 ]
Paul Yang commented on HIVE-1862:
---------------------------------
I'll take a look..
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980599#action_12980599 ]
Namit Jain commented on HIVE-1862:
----------------------------------
cool
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mac Yang updated HIVE-1862:
---------------------------
Status: Patch Available (was: Open)
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981087#action_12981087 ]
Mac Yang commented on HIVE-1862:
--------------------------------
@Namit, certainly, but I don't have a set up ready to go for that scale, so it will take some time.
In the mean time, it would be great to get some feedback on the patch to see if the approach looks reasonable.
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1862) Revive partition filtering in the Hive
MetaStore
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Yang updated HIVE-1862:
----------------------------
Status: Open (was: Patch Available)
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979926#action_12979926 ]
John Sichi commented on HIVE-1862:
----------------------------------
I checked our server; it's MySQL 5.0.84. Also, we set the following hive-site.xml properties:
datanucleus.transactionIsolation=repeatable-read
datanucleus.valuegeneration.transactionIsolation=repeatable-read
datanucleus.connectionPoolingType=none
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the
Hive MetaStore
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981887#action_12981887 ]
Devaraj Das commented on HIVE-1862:
-----------------------------------
Folks, any update on this one? Does it serve our purpose with the limitation that partition values can't have special characters. If so, could it be reviewed. Thanks!
> Revive partition filtering in the Hive MetaStore
> ------------------------------------------------
>
> Key: HIVE-1862
> URL: https://issues.apache.org/jira/browse/HIVE-1862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Devaraj Das
> Fix For: 0.7.0
>
> Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry
>
>
> HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.