[jira] [Commented] (HIVE-12856) LLAP: update (add/remove) the UDFs available in LLAP when they are changed (refresh periodically)

["Hive QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136462#comment-15136462 ] 

Hive QA commented on HIVE-12856:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12786557/HIVE-12856.patch

{color:green}SUCCESS:{color} +1 due to 21 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 86 failed/errored test(s), 10028 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_constprog_dpp
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llapdecider
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_lvj_mapjoin
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mapjoin_decimal
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_bmj_schema_evolution
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_fsstat
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_insert_overwrite_local_directory_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_result_complex
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_multi_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_schema_evolution
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_self_join
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_decimal
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_dynamic_partition
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_group_by
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_join_part_col_char
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInEmbed.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInEmbed
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterPartition
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterTable
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterViewParititon
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testColumnStatistics
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testComplexTable
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testComplexTypeApi
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testConcurrentMetastores
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDBOwner
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDBOwnerChange
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDatabase
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDatabaseLocation
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDatabaseLocationWithPermissionProblems
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDropTable
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testFilterLastPartition
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testFilterSinglePartition
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testFunctionWithResources
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testGetConfigValue
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testGetTableObjects
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testListPartitionNames
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testListPartitions
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testNameMethods
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartition
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartitionFilter
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testRenamePartition
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testRetriableClientWithConnLifetime
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testSimpleFunction
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testSimpleTable
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testSimpleTypeApi
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testStatsFastTrivial
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testSynchronized
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTableDatabase
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTableFilter
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testValidateTableCols
org.apache.hadoop.hive.metastore.TestHiveMetaTool.testExecuteJDOQL
org.apache.hadoop.hive.metastore.TestHiveMetaTool.testListFSRoot
org.apache.hadoop.hive.metastore.TestHiveMetaTool.testUpdateFSRootLocation
org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testSimpleFunction
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSimpleFunction
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testSimpleFunction
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testSimpleFunction
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6898/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6898/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6898/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 86 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12786557 - PreCommit-HIVE-TRUNK-Build

> LLAP: update (add/remove) the UDFs available in LLAP when they are changed (refresh periodically)
> -------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-12856
>                 URL: https://issues.apache.org/jira/browse/HIVE-12856
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: HIVE-12856.nogen.patch, HIVE-12856.patch
>
>
> I don't think re-querying the functions is going to scale, and the sessions obviously cannot notify all LLAP clusters of every change. We should add global versioning to metastore functions to track changes, and then possibly add a notification mechanism, potentially thru ZK to avoid overloading the metastore itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)