You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "ludlows (via GitHub)" <gi...@apache.org> on 2026/08/07 17:11:27 UTC
[PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
ludlows opened a new pull request, #6685:
URL: https://github.com/apache/hive/pull/6685
### What changes were proposed in this pull request?
here we want to check the columns in the `ON` clause of `MERGE INTO` syntax without being assigned table names
in case of copy on write table giving wrong results.
so, here we directly do not allow the unresolved columns in the `ON` clause to pass the check point.
the related issus is https://issues.apache.org/jira/si/jira.issueviews:issue-html/HIVE-29783/HIVE-29783.html .
The SQL used to reproduce the issue is like
```sql
-- source table
CREATE TABLE src (a INT, b INT) STORED AS ORC;
INSERT INTO src VALUES (9, 10);
-- target table cow
CREATE TABLE tgt_cow (a INT, b INT) STORED BY iceberg STORED AS ORC
TBLPROPERTIES (
'write.merge.mode' = 'copy-on-write'
);
INSERT INTO tgt_cow VALUES (9, 3), (2, 3);
-- target table mor
CREATE TABLE tgt_mor (a INT, b INT) STORED BY iceberg STORED AS ORC
TBLPROPERTIES (
'write.merge.mode' = 'merge-on-read'
);
INSERT INTO tgt_mor VALUES (9, 3), (2, 3);
-- merge into mor
MERGE INTO tgt_mor
USING src
ON a = src.a
WHEN MATCHED THEN UPDATE SET b = src.b;
-- Expected: SemanticException: Column a Found in more than One Tables/Subqueries
-- merge into cow
MERGE INTO tgt_cow
USING src
ON a = src.a
WHEN MATCHED THEN UPDATE SET b = src.b;
-- Not expected : Executes successfully without throwing an exception.
-- correctness checking after `merge` sql
SELECT * FROM testcow
-- returns (9, 10), (2, 10)
-- The row (2, 3) was incorrectly updated to (2, 10) .
```
### Why are the changes needed?
copy-on-write has a different write behavior with that of merge-on-read.
if we do not check the unresolved columns in the `ON` clause (make it a little more strict), the wrong logical plan is executed and produces wrong results.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
both negative and positive test cases are added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "ludlows (via GitHub)" <gi...@apache.org>.
ludlows commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-5831972909
https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-6685/6/tests/
this pr failed to pass the following check
```sql
alter table ice_orc COMPACT 'major' and wait where company_id=100 or dept_id in (1,2)
```
@kasakrisz @Aggarwal-Raghav do you have any suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "deniskuzZ (via GitHub) via gitbox" <gi...@hive.apache.org>.
deniskuzZ commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4468522561
##########
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergMerge.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.List;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.mr.TestHelper;
+import org.apache.iceberg.mr.hive.test.utils.HiveIcebergTestUtils;
+import org.apache.iceberg.relocated.com.google.common.base.Throwables;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+/**
+ * Tests Merge Related SQL features.
+ */
+public class TestHiveIcebergMerge extends HiveIcebergStorageHandlerWithEngineBase {
+
+ static final Schema SCHEMA = new Schema(
+ optional(1, "a", Types.IntegerType.get()),
+ optional(2, "b", Types.IntegerType.get(), "This is first name")
+ );
+
+ static final List<Record> TGT_RECORDS = TestHelper.RecordsBuilder.newInstance(SCHEMA)
+ .add(0, 1)
+ .add(9, 9)
+ .build();
+
+ @Test
+ public void testMergeIntoOnClauseColumnsNoAssignedTables() {
+ testTables.createTable(shell, "tgt", SCHEMA,
+ PartitionSpec.unpartitioned(), fileFormat, TGT_RECORDS, formatVersion);
+ shell.executeStatement("alter table default.tgt set tblproperties('write.merge.mode'='copy-on-write')");
+ shell.executeStatement("drop table if exists default.src");
+ shell.executeStatement("create table default.src (a int, b int) stored as orc");
+ shell.executeStatement("use default");
+ String sql = "MERGE INTO tgt using src on a= src.a when matched then update set b=10";
+ try {
+ shell.executeStatement(sql);
+ Assert.assertFalse(true); // place we should not touch
Review Comment:
The AssertionError from assertFalse(true) is caught by the catch (Throwable) below. The test still fails, but on the instanceof SemanticException check, with a misleading message. Please use Assert.assertThrows(...) (JUnit 4.13.2 is available) and assert on the root cause.
Since this is a parse-time check, a .q file under iceberg-handler/src/test/queries/negative would be cheaper than running the parameterized HiveIcebergStorageHandlerWithEngineBase matrix. Please also cover an unqualified source-only column on COW, and that MOR behavior is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-5230920951
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6685) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6685&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6685&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6685&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6685&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6685&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6685)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-5828974825
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6685) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6685&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6685&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6685&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6685&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6685&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6685)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4007683241
##########
ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java:
##########
@@ -1114,19 +1114,19 @@ public void testQuotedIdentifier() throws Exception {
"`d?*de e` decimal(5,2)," +
"vc varchar(128)) clustered by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true')");
runStatementOnDriver("create table " + src + "(gh int, j decimal(5,2), k varchar(128))");
- runStatementOnDriver("merge into " + target + " as `d/8` using " + src + " as `a/b` on i=gh " +
+ runStatementOnDriver("merge into " + target + " as `d/8` using " + src + " as `a/b` on `d/8`.i=`a/b`.gh " +
"\nwhen matched and i > 5 then delete " +
"\nwhen matched then update set vc='blah' " +
"\nwhen not matched then insert values(1,2.1,'baz')");
- runStatementOnDriver("merge into " + target + " as `d/8` using " + src + " as `a/b` on i=gh " +
+ runStatementOnDriver("merge into " + target + " as `d/8` using " + src + " as `a/b` on `d/8`.i=`a/b`.gh " +
Review Comment:
This seems to be a breaking change! For acid table, users will have to change their query scripts. Maybe we can enforce the check only when its COW?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "deniskuzZ (via GitHub) via gitbox" <gi...@hive.apache.org>.
deniskuzZ commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4468496831
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java:
##########
@@ -131,7 +133,21 @@ WHEN NOT MATCHED THEN INSERT VALUES (source.a2, source.b2)
.sourceName(sourceName)
.sourceAlias(getSourceAlias(source, sourceName))
.onClauseAsText(onClauseAsText);
-
+
+ OnClauseAnalyzer oca = new OnClauseAnalyzer(onClause, targetTable, targetAlias,
Review Comment:
Moving oca.analyze() out of the TOK_INSERT branch means it now runs for every MERGE, including UPDATE/DELETE-only ones on ACID/MOR tables. Its > 2 table refs IllegalArgumentException can fire where it didn’t before. Please keep it scoped to where it’s needed, or lazily evaluate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "deniskuzZ (via GitHub) via gitbox" <gi...@hive.apache.org>.
deniskuzZ commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4468418279
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java:
##########
@@ -131,7 +133,21 @@ WHEN NOT MATCHED THEN INSERT VALUES (source.a2, source.b2)
.sourceName(sourceName)
.sourceAlias(getSourceAlias(source, sourceName))
.onClauseAsText(onClauseAsText);
-
+
+ OnClauseAnalyzer oca = new OnClauseAnalyzer(onClause, targetTable, targetAlias,
+ conf, onClauseAsText);
+ oca.analyze();
+
+ boolean copyOnWriteMode = false;
+ HiveStorageHandler storageHandler = targetTable.getStorageHandler();
+ if (storageHandler != null) {
+ copyOnWriteMode = storageHandler.shouldOverwrite(targetTable, Context.Operation.MERGE);
Review Comment:
i am not sure it's a proper place for that. COW has dedicated rewriters:
- CopyOnWriteUpdateRewriter
- CopyOnWriteMergeRewriter
- CopyOnWriteDeleteRewriter
however, if @kasakrisz thinks it's ok - fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-6200578459
LGTM +1 (non-binding)
The changes I've suggested has been incorporated.
I can't see the CI outcome here, not sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4007683241
##########
ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java:
##########
@@ -1114,19 +1114,19 @@ public void testQuotedIdentifier() throws Exception {
"`d?*de e` decimal(5,2)," +
"vc varchar(128)) clustered by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true')");
runStatementOnDriver("create table " + src + "(gh int, j decimal(5,2), k varchar(128))");
- runStatementOnDriver("merge into " + target + " as `d/8` using " + src + " as `a/b` on i=gh " +
+ runStatementOnDriver("merge into " + target + " as `d/8` using " + src + " as `a/b` on `d/8`.i=`a/b`.gh " +
"\nwhen matched and i > 5 then delete " +
"\nwhen matched then update set vc='blah' " +
"\nwhen not matched then insert values(1,2.1,'baz')");
- runStatementOnDriver("merge into " + target + " as `d/8` using " + src + " as `a/b` on i=gh " +
+ runStatementOnDriver("merge into " + target + " as `d/8` using " + src + " as `a/b` on `d/8`.i=`a/b`.gh " +
Review Comment:
This seems to be a breaking change! For acid tables users will have to change the query. Maybe we can enforce the check only when its COW?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "deniskuzZ (via GitHub) via gitbox" <gi...@hive.apache.org>.
deniskuzZ commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4468418279
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java:
##########
@@ -131,7 +133,21 @@ WHEN NOT MATCHED THEN INSERT VALUES (source.a2, source.b2)
.sourceName(sourceName)
.sourceAlias(getSourceAlias(source, sourceName))
.onClauseAsText(onClauseAsText);
-
+
+ OnClauseAnalyzer oca = new OnClauseAnalyzer(onClause, targetTable, targetAlias,
+ conf, onClauseAsText);
+ oca.analyze();
+
+ boolean copyOnWriteMode = false;
+ HiveStorageHandler storageHandler = targetTable.getStorageHandler();
+ if (storageHandler != null) {
+ copyOnWriteMode = storageHandler.shouldOverwrite(targetTable, Context.Operation.MERGE);
Review Comment:
+1 to @kasakrisz’s option 1.
The wrong result comes from the COW rewrite: CopyOnWriteMergeRewriter.handleSource projects target columns with the t__ prefix, and replaceColumnRefsWithTargetPrefix rewrites only qualified tgt. refs. So an unqualified a resolves to src.a, ON a = src.a becomes src.a = src.a, and every target row matches. MOR doesn’t hit this because its join exposes a on both sides, so RowResolver raises the ambiguity error.
**Could we move the guard into the COW path?**
OnClauseAnalyzer.handleUnresolvedColumns already identifies which unqualified columns belong to the target. Pass those via MergeStatement and fail only for them, e.g. with ErrorMsg.AMBIGUOUS_COLUMN. That gives the MOR parity HIVE-29783 asks for, and oca.analyze() can stay scoped to the insert branch as before, so ACID/MOR paths are untouched.
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java:
##########
@@ -131,7 +133,21 @@ WHEN NOT MATCHED THEN INSERT VALUES (source.a2, source.b2)
.sourceName(sourceName)
.sourceAlias(getSourceAlias(source, sourceName))
.onClauseAsText(onClauseAsText);
-
+
+ OnClauseAnalyzer oca = new OnClauseAnalyzer(onClause, targetTable, targetAlias,
Review Comment:
Moving oca.analyze() out of the TOK_INSERT branch means it now runs for every MERGE, including UPDATE/DELETE-only ones on ACID/MOR tables. Its > 2 table refs IllegalArgumentException can fire where it didn’t before. Please keep it scoped to where it’s needed, or lazily evaluate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-5653380312
cc @Aggarwal-Raghav, @kasakrisz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "ludlows (via GitHub)" <gi...@apache.org>.
ludlows commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-6164961615
@kasakrisz @Aggarwal-Raghav do you have time to review the current version? many thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "deniskuzZ (via GitHub) via gitbox" <gi...@hive.apache.org>.
deniskuzZ commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4468507779
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java:
##########
@@ -131,7 +133,21 @@ WHEN NOT MATCHED THEN INSERT VALUES (source.a2, source.b2)
.sourceName(sourceName)
.sourceAlias(getSourceAlias(source, sourceName))
.onClauseAsText(onClauseAsText);
-
+
+ OnClauseAnalyzer oca = new OnClauseAnalyzer(onClause, targetTable, targetAlias,
+ conf, onClauseAsText);
+ oca.analyze();
+
+ boolean copyOnWriteMode = false;
+ HiveStorageHandler storageHandler = targetTable.getStorageHandler();
+ if (storageHandler != null) {
+ copyOnWriteMode = storageHandler.shouldOverwrite(targetTable, Context.Operation.MERGE);
+ }
+ // unresolved columns are not allowed in the on clause to avoid wrong results in Copy-On-Write mode
+ if (copyOnWriteMode && !oca.unresolvedColumns.isEmpty()) {
Review Comment:
This rejects any unqualified column, not just ambiguous ones. E.g. ON tgt.id = src_only_col works today on both COW and MOR, but would fail after this change. Only unqualified columns that match a target column are a problem for COW.
Also, please reuse the existing ambiguous-column ErrorMsg instead of a free-form message, so COW and MOR report the same error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "deniskuzZ (via GitHub) via gitbox" <gi...@hive.apache.org>.
deniskuzZ commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-6632744232
@ludlows the description says “no user-facing change”, but this does change behavior: COW MERGE statements with unqualified ON columns will now fail. Please update the description, and ideally narrow the check to genuinely ambiguous columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "deniskuzZ (via GitHub) via gitbox" <gi...@hive.apache.org>.
deniskuzZ commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4468614649
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java:
##########
@@ -131,7 +133,21 @@ WHEN NOT MATCHED THEN INSERT VALUES (source.a2, source.b2)
.sourceName(sourceName)
.sourceAlias(getSourceAlias(source, sourceName))
.onClauseAsText(onClauseAsText);
-
+
+ OnClauseAnalyzer oca = new OnClauseAnalyzer(onClause, targetTable, targetAlias,
Review Comment:
Moving oca.analyze() out of the TOK_INSERT branch makes it run for every MERGE, on all table types, not only COW. visit() throws IllegalArgumentException for nested DOT refs, and a struct field access in ON produces exactly that AST. So e.g. MERGE INTO tgt t USING src s ON t.info.id = s.id WHEN MATCHED THEN UPDATE ... would now fail even on ACID/MOR tables, where it previously compiled.
Please keep analyze() scoped to the insert branch. If the COW guard needs the unqualified columns, collect them separately, and handle DOT(DOT(...)) in visit(). A test with a struct column in the ON clause would be good too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "deniskuzZ (via GitHub) via gitbox" <gi...@hive.apache.org>.
deniskuzZ commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4468418279
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java:
##########
@@ -131,7 +133,21 @@ WHEN NOT MATCHED THEN INSERT VALUES (source.a2, source.b2)
.sourceName(sourceName)
.sourceAlias(getSourceAlias(source, sourceName))
.onClauseAsText(onClauseAsText);
-
+
+ OnClauseAnalyzer oca = new OnClauseAnalyzer(onClause, targetTable, targetAlias,
+ conf, onClauseAsText);
+ oca.analyze();
+
+ boolean copyOnWriteMode = false;
+ HiveStorageHandler storageHandler = targetTable.getStorageHandler();
+ if (storageHandler != null) {
+ copyOnWriteMode = storageHandler.shouldOverwrite(targetTable, Context.Operation.MERGE);
Review Comment:
+1 to @kasakrisz’s option 1.
The wrong result comes from the COW rewrite: CopyOnWriteMergeRewriter.handleSource projects target columns as `t__<col>`, and replaceColumnRefsWithTargetPrefix prefixes only qualified tgt. refs. So an unqualified a silently binds to src.a, `ON a = src.a` becomes `src.a = src.a`, and every target row matches.
**Could we move the guard into CopyOnWriteMergeRewriter**? For example, pass the unqualified refs that resolve to target columns from OnClauseAnalyzer (handleUnresolvedColumns already computes them) via MergeStatement, and throw the standard ambiguous-column error only for those. That gives parity with MOR, which is what HIVE-29783 expects, and it keeps ACID/MOR paths untouched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-5237245730
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6685) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6685&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6685&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6685&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6685&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6685&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6685)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "ludlows (via GitHub)" <gi...@apache.org>.
ludlows commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-5748769048
@kasakrisz @Aggarwal-Raghav thanks for the comments.
I will refine this PR in serveral days
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-6095808358
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6685) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6685&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6685&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6685&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6685&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6685&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6685)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-6097825261
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6685) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6685&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6685&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6685&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6685&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6685&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6685)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4026577683
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java:
##########
@@ -130,7 +130,15 @@ WHEN NOT MATCHED THEN INSERT VALUES (source.a2, source.b2)
.sourceName(sourceName)
.sourceAlias(getSourceAlias(source, sourceName))
.onClauseAsText(onClauseAsText);
-
+
+ OnClauseAnalyzer oca = new OnClauseAnalyzer(onClause, targetTable, targetAlias,
+ conf, onClauseAsText);
+ oca.analyze();
+ // unresolved columns are not allowed in the on clause to avoid wrong results
+ if (!oca.unresolvedColumns.isEmpty()) {
Review Comment:
I see two approaches for the fix:
1. Check for column ambiguity in COW only: Move the check to a COW-specific class like `CopyOnWriteMergeRewriter`.
2. Check generally (as proposed in this PR): Keep the check here in `MergeSemanticAnalyzer`, but ensure no breaking changes [1] are introduced to the supported SQL syntax. Please refine this further.
[1] https://github.com/apache/hive/pull/6685#discussion_r4007683241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4007683241
##########
ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java:
##########
@@ -1114,19 +1114,19 @@ public void testQuotedIdentifier() throws Exception {
"`d?*de e` decimal(5,2)," +
"vc varchar(128)) clustered by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true')");
runStatementOnDriver("create table " + src + "(gh int, j decimal(5,2), k varchar(128))");
- runStatementOnDriver("merge into " + target + " as `d/8` using " + src + " as `a/b` on i=gh " +
+ runStatementOnDriver("merge into " + target + " as `d/8` using " + src + " as `a/b` on `d/8`.i=`a/b`.gh " +
"\nwhen matched and i > 5 then delete " +
"\nwhen matched then update set vc='blah' " +
"\nwhen not matched then insert values(1,2.1,'baz')");
- runStatementOnDriver("merge into " + target + " as `d/8` using " + src + " as `a/b` on i=gh " +
+ runStatementOnDriver("merge into " + target + " as `d/8` using " + src + " as `a/b` on `d/8`.i=`a/b`.gh " +
Review Comment:
This seems to be a breaking change? For acid tables users will have to change the query. Maybe we can enforce the check only when its COW?
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java:
##########
@@ -130,7 +130,15 @@ WHEN NOT MATCHED THEN INSERT VALUES (source.a2, source.b2)
.sourceName(sourceName)
.sourceAlias(getSourceAlias(source, sourceName))
.onClauseAsText(onClauseAsText);
-
+
+ OnClauseAnalyzer oca = new OnClauseAnalyzer(onClause, targetTable, targetAlias,
+ conf, onClauseAsText);
+ oca.analyze();
+ // unresolved columns are not allowed in the on clause to avoid wrong results
+ if (!oca.unresolvedColumns.isEmpty()) {
Review Comment:
How about restricting the exception to COW?
```patch
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java b/ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java
index f5e23dd8a9..aa5ac64a84 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java
@@ -28,6 +28,8 @@
import org.apache.hadoop.hive.ql.lib.Node;
import org.apache.hadoop.hive.ql.metadata.HiveUtils;
import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.metadata.HiveStorageHandler;
import org.apache.hadoop.hive.ql.parse.rewrite.MergeStatement;
import org.apache.hadoop.hive.ql.parse.rewrite.RewriterFactory;
@@ -125,18 +127,25 @@ WHEN NOT MATCHED THEN INSERT VALUES (source.a2, source.b2)
String sourceName = getSimpleTableName(source);
ASTNode onClause = (ASTNode) tree.getChild(2);
String onClauseAsText = getMatchedText(onClause);
-
+
MergeStatement.MergeStatementBuilder mergeStatementBuilder = MergeStatement
.withTarget(targetTable, getFullTableNameForSQL(targetNameNode), targetAlias)
.sourceName(sourceName)
.sourceAlias(getSourceAlias(source, sourceName))
.onClauseAsText(onClauseAsText);
-
+
+ boolean copyOnWriteMode = false;
+ HiveStorageHandler storageHandler = targetTable.getStorageHandler();
+ if (storageHandler != null) {
+ copyOnWriteMode = storageHandler.shouldOverwrite(targetTable, Context.Operation.MERGE);
+ }
+
OnClauseAnalyzer oca = new OnClauseAnalyzer(onClause, targetTable, targetAlias,
conf, onClauseAsText);
oca.analyze();
- // unresolved columns are not allowed in the on clause to avoid wrong results
- if (!oca.unresolvedColumns.isEmpty()) {
+
+ // Only enforce unresolved column check in Copy-on-Write mode to avoid breaking standard ACID/MoR
+ if (copyOnWriteMode && !oca.unresolvedColumns.isEmpty()) {
throw new SemanticException("UnResolvedColumns exist: " + String.join(",", oca.unresolvedColumns) +
". We should assign a table name to each column in the ON clause like tbl.col.");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-5930684196
> https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-6685/6/tests/
>
> this pr failed to pass the following check
>
> ```sql
> alter table ice_orc COMPACT 'major' and wait where company_id=100 or dept_id in (1,2)
> ```
>
> @kasakrisz @Aggarwal-Raghav do you have any suggestions?
@ludlows
The test `iceberg_major_compaction_partition_evolution_w_id_spec_w_filter.q` passes locally with this patch. I don't see any code changes related to compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "ludlows (via GitHub)" <gi...@apache.org>.
ludlows commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-6512441241
@kasakrisz @deniskuzZ do you guys have time to take a look ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6685:
URL: https://github.com/apache/hive/pull/6685#issuecomment-5232063810
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6685) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6685&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6685&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6685&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6685&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6685&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6685)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "deniskuzZ (via GitHub) via gitbox" <gi...@hive.apache.org>.
deniskuzZ commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4468522561
##########
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergMerge.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.List;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.mr.TestHelper;
+import org.apache.iceberg.mr.hive.test.utils.HiveIcebergTestUtils;
+import org.apache.iceberg.relocated.com.google.common.base.Throwables;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+/**
+ * Tests Merge Related SQL features.
+ */
+public class TestHiveIcebergMerge extends HiveIcebergStorageHandlerWithEngineBase {
+
+ static final Schema SCHEMA = new Schema(
+ optional(1, "a", Types.IntegerType.get()),
+ optional(2, "b", Types.IntegerType.get(), "This is first name")
+ );
+
+ static final List<Record> TGT_RECORDS = TestHelper.RecordsBuilder.newInstance(SCHEMA)
+ .add(0, 1)
+ .add(9, 9)
+ .build();
+
+ @Test
+ public void testMergeIntoOnClauseColumnsNoAssignedTables() {
+ testTables.createTable(shell, "tgt", SCHEMA,
+ PartitionSpec.unpartitioned(), fileFormat, TGT_RECORDS, formatVersion);
+ shell.executeStatement("alter table default.tgt set tblproperties('write.merge.mode'='copy-on-write')");
+ shell.executeStatement("drop table if exists default.src");
+ shell.executeStatement("create table default.src (a int, b int) stored as orc");
+ shell.executeStatement("use default");
+ String sql = "MERGE INTO tgt using src on a= src.a when matched then update set b=10";
+ try {
+ shell.executeStatement(sql);
+ Assert.assertFalse(true); // place we should not touch
Review Comment:
assertFalse(true) throws an AssertionError, and the catch (Throwable) below swallows it. Please use Assert.assertThrows(...) and assert on the root cause.
Also, a parse-time check probably doesn’t need the full format/version matrix of HiveIcebergStorageHandlerWithEngineBase. A client-negative .q file might be cheaper.
Please also add coverage that an unqualified source-only column still works on COW and that MOR/ACID behavior is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "deniskuzZ (via GitHub) via gitbox" <gi...@hive.apache.org>.
deniskuzZ commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4468418279
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java:
##########
@@ -131,7 +133,21 @@ WHEN NOT MATCHED THEN INSERT VALUES (source.a2, source.b2)
.sourceName(sourceName)
.sourceAlias(getSourceAlias(source, sourceName))
.onClauseAsText(onClauseAsText);
-
+
+ OnClauseAnalyzer oca = new OnClauseAnalyzer(onClause, targetTable, targetAlias,
+ conf, onClauseAsText);
+ oca.analyze();
+
+ boolean copyOnWriteMode = false;
+ HiveStorageHandler storageHandler = targetTable.getStorageHandler();
+ if (storageHandler != null) {
+ copyOnWriteMode = storageHandler.shouldOverwrite(targetTable, Context.Operation.MERGE);
Review Comment:
+1 to @kasakrisz’s option 1.
The wrong result comes from the COW rewrite: CopyOnWriteMergeRewriter.handleSource projects target columns as `t__<col>`, and replaceColumnRefsWithTargetPrefix prefixes only qualified tgt. refs. So an unqualified a silently binds to src.a, `ON a = src.a` becomes `src.a = src.a`, and every target row matches.
**Could we move the guard into CopyOnWriteMergeRewriter**?
For example, pass the unqualified refs that resolve to target columns from OnClauseAnalyzer (handleUnresolvedColumns already computes them) via MergeStatement, and throw the standard ambiguous-column error only for those. That gives parity with MOR, which is what HIVE-29783 expects, and it keeps ACID/MOR paths untouched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "deniskuzZ (via GitHub) via gitbox" <gi...@hive.apache.org>.
deniskuzZ commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4468507779
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java:
##########
@@ -131,7 +133,21 @@ WHEN NOT MATCHED THEN INSERT VALUES (source.a2, source.b2)
.sourceName(sourceName)
.sourceAlias(getSourceAlias(source, sourceName))
.onClauseAsText(onClauseAsText);
-
+
+ OnClauseAnalyzer oca = new OnClauseAnalyzer(onClause, targetTable, targetAlias,
+ conf, onClauseAsText);
+ oca.analyze();
+
+ boolean copyOnWriteMode = false;
+ HiveStorageHandler storageHandler = targetTable.getStorageHandler();
+ if (storageHandler != null) {
+ copyOnWriteMode = storageHandler.shouldOverwrite(targetTable, Context.Operation.MERGE);
+ }
+ // unresolved columns are not allowed in the on clause to avoid wrong results in Copy-On-Write mode
+ if (copyOnWriteMode && !oca.unresolvedColumns.isEmpty()) {
Review Comment:
This rejects any unqualified column, not only ambiguous ones. E.g. ON tgt.id = src_only_col works on COW today, because the column exists only on the source side of the rewritten join, but it would fail after this change. Only unqualified columns that match a target column are a problem for COW.
Also, please use an ErrorMsg constant (e.g. AMBIGUOUS_COLUMN) instead of a free-form message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29783: Iceberg CoW MERGE INTO missing column ambiguity check in the `ON` clause [hive]
Posted by "deniskuzZ (via GitHub) via gitbox" <gi...@hive.apache.org>.
deniskuzZ commented on code in PR #6685:
URL: https://github.com/apache/hive/pull/6685#discussion_r4468614649
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java:
##########
@@ -131,7 +133,21 @@ WHEN NOT MATCHED THEN INSERT VALUES (source.a2, source.b2)
.sourceName(sourceName)
.sourceAlias(getSourceAlias(source, sourceName))
.onClauseAsText(onClauseAsText);
-
+
+ OnClauseAnalyzer oca = new OnClauseAnalyzer(onClause, targetTable, targetAlias,
Review Comment:
Moving oca.analyze() out of the TOK_INSERT branch makes it run for every MERGE, on all table types, not only COW. visit() throws IllegalArgumentException for nested DOT refs, and a struct field access in ON produces exactly that AST. So e.g. MERGE INTO tgt t USING src s ON t.info.id = s.id WHEN MATCHED THEN UPDATE ... would now fail even on ACID/MOR tables, where it previously compiled.
Please keep analyze() scoped to the insert branch. If the COW guard needs the unqualified columns, collect them separately, and handle DOT(DOT(...)) in visit(). A test with a struct column in the ON clause would be good too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org