[jira] [Commented] (HIVE-12631) LLAP: support ORC ACID tables

["Hive QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-12631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071621#comment-16071621 ] 

Hive QA commented on HIVE-12631:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12875408/HIVE-12631.11.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5863/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5863/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5863/

Messages:
{noformat}
**** This message was trimmed, see log for full details ****
error: ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkPartitionPruningSinkOperator.java: patch does not apply
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/plan/AlterDatabaseDesc.java:21
error: ql/src/java/org/apache/hadoop/hive/ql/plan/AlterDatabaseDesc.java: patch does not apply
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java:28
error: ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java: patch does not apply
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/plan/CreateFunctionDesc.java:22
error: ql/src/java/org/apache/hadoop/hive/ql/plan/CreateFunctionDesc.java: patch does not apply
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/plan/DropFunctionDesc.java:19
error: ql/src/java/org/apache/hadoop/hive/ql/plan/DropFunctionDesc.java: patch does not apply
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/plan/RenamePartitionDesc.java:17
error: ql/src/java/org/apache/hadoop/hive/ql/plan/RenamePartitionDesc.java: patch does not apply
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java:22
error: ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java: patch does not apply
error: patch failed: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/AbstractTestParquetDirect.java:27
error: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/AbstractTestParquetDirect.java: patch does not apply
error: patch failed: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestDataWritableWriter.java:16
error: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestDataWritableWriter.java: patch does not apply
error: patch failed: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRowGroupFilter.java:21
error: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRowGroupFilter.java: patch does not apply
error: patch failed: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java:106
error: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java: patch does not apply
error: patch failed: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java:31
error: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java: patch does not apply
error: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java: No such file or directory
error: patch failed: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/read/TestParquetFilterPredicate.java:33
error: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/read/TestParquetFilterPredicate.java: patch does not apply
error: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/TestParquetTimestampConverter.java: No such file or directory
error: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/TestNanoTimeUtils.java: No such file or directory
error: ql/src/test/queries/clientnegative/parquet_alter_part_table_drop_columns.q: already exists in working directory
error: ql/src/test/queries/clientnegative/parquet_int96_alter_invalid_timezone.q: No such file or directory
error: ql/src/test/queries/clientnegative/parquet_int96_create_invalid_timezone.q: No such file or directory
error: patch failed: ql/src/test/queries/clientpositive/concat_op.q:24
error: ql/src/test/queries/clientpositive/concat_op.q: patch does not apply
error: patch failed: ql/src/test/queries/clientpositive/correlationoptimizer14.q:2
error: ql/src/test/queries/clientpositive/correlationoptimizer14.q: patch does not apply
error: patch failed: ql/src/test/queries/clientpositive/groupby_distinct_samekey.q:1
error: ql/src/test/queries/clientpositive/groupby_distinct_samekey.q: patch does not apply
error: patch failed: ql/src/test/queries/clientpositive/input20.q:7
error: ql/src/test/queries/clientpositive/input20.q: patch does not apply
error: patch failed: ql/src/test/queries/clientpositive/input33.q:7
error: ql/src/test/queries/clientpositive/input33.q: patch does not apply
error: patch failed: ql/src/test/queries/clientpositive/input3_limit.q:7
error: ql/src/test/queries/clientpositive/input3_limit.q: patch does not apply
error: ql/src/test/queries/clientpositive/llap_smb.q: already exists in working directory
error: ql/src/test/queries/clientpositive/parquet_int96_timestamp.q: No such file or directory
error: ql/src/test/queries/clientpositive/parquet_predicate_pushdown_2.q: already exists in working directory
error: ql/src/test/queries/clientpositive/parquet_timestamp_conversion.q: No such file or directory
error: patch failed: ql/src/test/queries/clientpositive/ppd2.q:2
error: ql/src/test/queries/clientpositive/ppd2.q: patch does not apply
error: patch failed: ql/src/test/queries/clientpositive/reduce_deduplicate_extended.q:2
error: ql/src/test/queries/clientpositive/reduce_deduplicate_extended.q: patch does not apply
error: patch failed: ql/src/test/queries/clientpositive/subquery_in.q:74
error: ql/src/test/queries/clientpositive/subquery_in.q: patch does not apply
error: patch failed: ql/src/test/queries/clientpositive/truncate_column_buckets.q:11
error: ql/src/test/queries/clientpositive/truncate_column_buckets.q: patch does not apply
error: patch failed: ql/src/test/results/clientnegative/authorization_uri_export.q.out:9
error: ql/src/test/results/clientnegative/authorization_uri_export.q.out: patch does not apply
error: ql/src/test/results/clientnegative/parquet_alter_part_table_drop_columns.q.out: already exists in working directory
error: ql/src/test/results/clientnegative/parquet_int96_alter_invalid_timezone.q.out: No such file or directory
error: ql/src/test/results/clientnegative/parquet_int96_create_invalid_timezone.q.out: No such file or directory
error: patch failed: ql/src/test/results/clientpositive/auto_join0.q.out:1
error: ql/src/test/results/clientpositive/auto_join0.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/auto_join15.q.out:15
error: ql/src/test/results/clientpositive/auto_join15.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/auto_join20.q.out:15
error: ql/src/test/results/clientpositive/auto_join20.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/auto_join31.q.out:21
error: ql/src/test/results/clientpositive/auto_join31.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/cbo_rp_auto_join0.q.out:1
error: ql/src/test/results/clientpositive/cbo_rp_auto_join0.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/concat_op.q.out:160
error: ql/src/test/results/clientpositive/concat_op.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/constprog_partitioner.q.out:107
error: ql/src/test/results/clientpositive/constprog_partitioner.q.out: patch does not apply
error: ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out: No such file or directory
error: patch failed: ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out:24
error: ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/identity_project_remove_skip.q.out:17
error: ql/src/test/results/clientpositive/identity_project_remove_skip.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/input20.q.out:11
error: ql/src/test/results/clientpositive/input20.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/input33.q.out:11
error: ql/src/test/results/clientpositive/input33.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/input3_limit.q.out:31
error: ql/src/test/results/clientpositive/input3_limit.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/limit_pushdown_negative.q.out:145
error: ql/src/test/results/clientpositive/limit_pushdown_negative.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/auto_join0.q.out:1
error: ql/src/test/results/clientpositive/llap/auto_join0.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/auto_join30.q.out:23
error: ql/src/test/results/clientpositive/llap/auto_join30.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/explainuser_1.q.out:209
error: ql/src/test/results/clientpositive/llap/explainuser_1.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/identity_project_remove_skip.q.out:25
error: ql/src/test/results/clientpositive/llap/identity_project_remove_skip.q.out: patch does not apply
error: ql/src/test/results/clientpositive/llap/llap_smb.q.out: already exists in working directory
error: patch failed: ql/src/test/results/clientpositive/llap/mrr.q.out:1293
error: ql/src/test/results/clientpositive/llap/mrr.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/subquery_exists.q.out:50
error: ql/src/test/results/clientpositive/llap/subquery_exists.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/subquery_in.q.out:164
error: ql/src/test/results/clientpositive/llap/subquery_in.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/subquery_multi.q.out:2216
error: ql/src/test/results/clientpositive/llap/subquery_multi.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/subquery_notin.q.out:341
error: ql/src/test/results/clientpositive/llap/subquery_notin.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/subquery_scalar.q.out:2300
error: ql/src/test/results/clientpositive/llap/subquery_scalar.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/subquery_select.q.out:1825
error: ql/src/test/results/clientpositive/llap/subquery_select.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/subquery_views.q.out:157
error: ql/src/test/results/clientpositive/llap/subquery_views.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/sysdb.q.out:130
error: ql/src/test/results/clientpositive/llap/sysdb.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/tez_join_tests.q.out:13
error: ql/src/test/results/clientpositive/llap/tez_join_tests.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/tez_joins_explain.q.out:13
error: ql/src/test/results/clientpositive/llap/tez_joins_explain.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/vector_join30.q.out:39
error: ql/src/test/results/clientpositive/llap/vector_join30.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out:402
error: ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/llap/vector_number_compare_projection.q.out:118
error: ql/src/test/results/clientpositive/llap/vector_number_compare_projection.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/multi_insert_gby2.q.out:30
error: ql/src/test/results/clientpositive/multi_insert_gby2.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/multi_insert_gby3.q.out:38
error: ql/src/test/results/clientpositive/multi_insert_gby3.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out:305
error: ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out: patch does not apply
error: ql/src/test/results/clientpositive/parquet_predicate_pushdown_2.q.out: already exists in working directory
error: ql/src/test/results/clientpositive/parquet_timestamp_conversion.q.out: No such file or directory
error: patch failed: ql/src/test/results/clientpositive/perf/query16.q.out:1
error: ql/src/test/results/clientpositive/perf/query16.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/perf/query94.q.out:1
error: ql/src/test/results/clientpositive/perf/query94.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/ppd2.q.out:66
error: ql/src/test/results/clientpositive/ppd2.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/ppd_join4.q.out:46
error: ql/src/test/results/clientpositive/ppd_join4.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/auto_join0.q.out:1
error: ql/src/test/results/clientpositive/spark/auto_join0.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/auto_join15.q.out:24
error: ql/src/test/results/clientpositive/spark/auto_join15.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/auto_join20.q.out:66
error: ql/src/test/results/clientpositive/spark/auto_join20.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/auto_join30.q.out:22
error: ql/src/test/results/clientpositive/spark/auto_join30.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/auto_join31.q.out:28
error: ql/src/test/results/clientpositive/spark/auto_join31.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out:122
error: ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out:28
error: ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/identity_project_remove_skip.q.out:24
error: ql/src/test/results/clientpositive/spark/identity_project_remove_skip.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/multi_insert_gby2.q.out:39
error: ql/src/test/results/clientpositive/spark/multi_insert_gby2.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out:47
error: ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out: patch does not apply
error: ql/src/test/results/clientpositive/spark/parquet_int96_timestamp.q.out: No such file or directory
error: patch failed: ql/src/test/results/clientpositive/spark/spark_multi_insert_parallel_orderby.q.out:263
error: ql/src/test/results/clientpositive/spark/spark_multi_insert_parallel_orderby.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/subquery_exists.q.out:47
error: ql/src/test/results/clientpositive/spark/subquery_exists.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/subquery_in.q.out:155
error: ql/src/test/results/clientpositive/spark/subquery_in.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/tez_join_tests.q.out:12
error: ql/src/test/results/clientpositive/spark/tez_join_tests.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/tez_joins_explain.q.out:12
error: ql/src/test/results/clientpositive/spark/tez_joins_explain.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/truncate_column_buckets.q.out:19
error: ql/src/test/results/clientpositive/spark/truncate_column_buckets.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out:354
error: ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/subquery_exists.q.out:40
error: ql/src/test/results/clientpositive/subquery_exists.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/subquery_exists_having.q.out:74
error: ql/src/test/results/clientpositive/subquery_exists_having.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/subquery_in_having.q.out:1776
error: ql/src/test/results/clientpositive/subquery_in_having.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/subquery_notexists.q.out:29
error: ql/src/test/results/clientpositive/subquery_notexists.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/subquery_notexists_having.q.out:72
error: ql/src/test/results/clientpositive/subquery_notexists_having.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/subquery_notin_having.q.out:286
error: ql/src/test/results/clientpositive/subquery_notin_having.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out:62
error: ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/truncate_column_buckets.q.out:19
error: ql/src/test/results/clientpositive/truncate_column_buckets.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out:361
error: ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out: patch does not apply
error: patch failed: ql/src/test/results/clientpositive/vector_tablesample_rows.q.out:238
error: ql/src/test/results/clientpositive/vector_tablesample_rows.q.out: patch does not apply
error: patch failed: spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java:87
error: spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java: patch does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12875408 - PreCommit-HIVE-Build

> LLAP: support ORC ACID tables
> -----------------------------
>
>                 Key: HIVE-12631
>                 URL: https://issues.apache.org/jira/browse/HIVE-12631
>             Project: Hive
>          Issue Type: Bug
>          Components: llap, Transactions
>            Reporter: Sergey Shelukhin
>            Assignee: Teddy Choi
>         Attachments: HIVE-12631.10.patch, HIVE-12631.10.patch, HIVE-12631.11.patch, HIVE-12631.1.patch, HIVE-12631.2.patch, HIVE-12631.3.patch, HIVE-12631.4.patch, HIVE-12631.5.patch, HIVE-12631.6.patch, HIVE-12631.7.patch, HIVE-12631.8.patch, HIVE-12631.8.patch, HIVE-12631.9.patch
>
>
> LLAP uses a completely separate read path in ORC to allow for caching and parallelization of reads and processing. This path does not support ACID. As far as I remember ACID logic is embedded inside ORC format; we need to refactor it to be on top of some interface, if practical; or just port it to LLAP read path.
> Another consideration is how the logic will work with cache. The cache is currently low-level (CB-level in ORC), so we could just use it to read bases and deltas (deltas should be cached with higher priority) and merge as usual. We could also cache merged representation in future.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)