[jira] [Commented] (HIVE-11229) Mutation API: Coordinator communication with meta store should be optional

["Hive QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-11229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14622972#comment-14622972 ] 

Hive QA commented on HIVE-11229:
--------------------------------



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744722/HIVE-11229.1.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9160 tests executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4575/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4575/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4575/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12744722 - PreCommit-HIVE-TRUNK-Build

> Mutation API: Coordinator communication with meta store should be optional
> --------------------------------------------------------------------------
>
>                 Key: HIVE-11229
>                 URL: https://issues.apache.org/jira/browse/HIVE-11229
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>    Affects Versions: 2.0.0
>            Reporter: Elliot West
>            Assignee: Elliot West
>              Labels: streaming_api
>         Attachments: HIVE-11229.1.patch
>
>
> [~ekoifman] raised a theoretical issue with the streaming mutation API (HIVE-10165) where worker nodes operating in a distributed cluster might overwhelm a meta store while trying to obtain partition locks. Although this does not happen in practice (see HIVE-11228), the API does communicate with the meta store in this manner to obtain partition paths and create new partitions. Therefore the issue described does in fact exist in the current implementation, albeit in a different code path. I’d like to make such communication optional like so:
> * When the user chooses not to create partitions on demand, no meta store connection will be created in the {{MutationCoordinators}}. Additionally, partition paths will be resolved using {{org.apache.hadoop.hive.metastore.Warehouse.getPartitionPath(Path, LinkedHashMap<String, String>)}} which should be suitable so long as standard Hive partition layouts are followed.
> * If the user does choose to create partitions on demand then the system will operate as is does currently; using the meta store to both issue {{add_partition}} events and look up partition meta data.
> * The documentation will be updated to describe these behaviours and outline alternative approaches to collecting affected partition names and creating partitions in a less intensive manner.
> Side note for follow up: The parameter names {{tblName}} and {{dbName}} seem to be the wrong way around on the method {{org.apache.hadoop.hive.metastore.IMetaStoreClient.getPartition(String, String, List<String>)}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)