You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2020/08/12 20:26:55 UTC
[impala] 01/02: IMPALA-9859: Full ACID Milestone 4: Part 2 Reading
modified tables (complex types)
This is an automated email from the ASF dual-hosted git repository.
stakiar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit da34d34a42ad1bb77d6911708f1363c53ac79018
Author: Zoltan Borok-Nagy <bo...@cloudera.com>
AuthorDate: Fri Jul 10 14:17:04 2020 +0200
IMPALA-9859: Full ACID Milestone 4: Part 2 Reading modified tables (complex types)
This implements scanning full ACID tables that contain complex types.
The same technique works that we use for primitive types. I.e. we add
a LEFT ANTI JOIN on top of the Hdfs scan node in order to subtract
the deleted rows from the inserted rows.
However, there were some types of queries where we couldn't do that.
These are the queries that scan the nested collection items directly.
E.g.: SELECT item FROM complextypestbl.int_array;
The above query only creates a single tuple descriptor that holds the
collection items. Since this tuple descriptor is not at the table-level,
we cannot add slot references to the hidden ACID column which are at the
top level of the table schema.
To resolve this I added a statement rewriter that rewrites the above
statement to the following:
SELECT item FROM complextypestbl $a$1, $a$1.int_array;
Now in this example we'll have two tuple descriptors, one for the
table-level, and one for the collection item. So we can add the ACID
slot refs to the table-level tuple descriptor. The rewrite is
implemented by the new AcidRewriter class.
Performance
I executed the following query with num_nodes=1 on a non-transactional
table (without the rewrite), and on an ACID table (with the rewrite):
select count(*) from customer_nested.c_orders.o_lineitems;
Without the rewrite:
Fetched 1 row(s) in 0.41s
+--------------+--------+-------+----------+----------+-------+------------+----------+---------------+---------------------------------------------------+
| Operator | #Hosts | #Inst | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail |
+--------------+--------+-------+----------+----------+-------+------------+----------+---------------+---------------------------------------------------+
| F00:ROOT | 1 | 1 | 13.61us | 13.61us | | | 0 B | 0 B | |
| 01:AGGREGATE | 1 | 1 | 3.68ms | 3.68ms | 1 | 1 | 16.00 KB | 10.00 MB | FINALIZE |
| 00:SCAN HDFS | 1 | 1 | 280.47ms | 280.47ms | 6.00M | 15.00M | 56.98 MB | 8.00 MB | tpch_nested_orc_def.customer.c_orders.o_lineitems |
+--------------+--------+-------+----------+----------+-------+------------+----------+---------------+---------------------------------------------------+
With the rewrite:
Fetched 1 row(s) in 0.42s
+---------------------------+--------+-------+----------+----------+---------+------------+----------+---------------+---------------------------------------+
| Operator | #Hosts | #Inst | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail |
+---------------------------+--------+-------+----------+----------+---------+------------+----------+---------------+---------------------------------------+
| F00:ROOT | 1 | 1 | 25.16us | 25.16us | | | 0 B | 0 B | |
| 05:AGGREGATE | 1 | 1 | 3.44ms | 3.44ms | 1 | 1 | 63.00 KB | 10.00 MB | FINALIZE |
| 01:SUBPLAN | 1 | 1 | 16.52ms | 16.52ms | 6.00M | 125.92M | 47.00 KB | 0 B | |
| |--04:NESTED LOOP JOIN | 1 | 1 | 188.47ms | 188.47ms | 0 | 10 | 24.00 KB | 12 B | CROSS JOIN |
| | |--02:SINGULAR ROW SRC | 1 | 1 | 0ns | 0ns | 0 | 1 | 0 B | 0 B | |
| | 03:UNNEST | 1 | 1 | 25.37ms | 25.37ms | 0 | 10 | 0 B | 0 B | $a$1.c_orders.o_lineitems o_lineitems |
| 00:SCAN HDFS | 1 | 1 | 96.26ms | 96.26ms | 100.00K | 12.59M | 38.19 MB | 72.00 MB | default.customer_nested $a$1 |
+---------------------------+--------+-------+----------+----------+---------+------------+----------+---------------+---------------------------------------+
So the overhead is very little.
Testing
* Added planner tests to PlannerTest/acid-scans.test
* E2E query tests to QueryTest/full-acid-complex-type-scans.test
* E2E tests for rowid-generation: QueryTest/full-acid-rowid.test
Change-Id: I8b2c6cd3d87c452c5b96a913b14c90ada78d4c6f
Reviewed-on: http://gerrit.cloudera.org:8080/16228
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
---
.../apache/impala/analysis/AnalysisContext.java | 13 +-
.../java/org/apache/impala/analysis/Analyzer.java | 14 +
.../org/apache/impala/analysis/FromClause.java | 17 +-
.../org/apache/impala/analysis/StmtRewriter.java | 81 ++++
.../java/org/apache/impala/analysis/TableRef.java | 11 +
.../apache/impala/planner/SingleNodePlanner.java | 15 +-
.../functional/functional_schema_template.sql | 15 +
.../datasets/functional/schema_constraints.csv | 1 +
.../queries/PlannerTest/acid-scans.test | 502 +++++++++++++++++++++
.../queries/QueryTest/acid-negative.test | 26 --
.../QueryTest/full-acid-complex-type-scans.test | 185 ++++++++
.../queries/QueryTest/full-acid-rowid.test | 80 ++++
tests/query_test/test_acid.py | 10 +-
13 files changed, 922 insertions(+), 48 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java b/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
index 71389b8..61d94c6 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
@@ -375,9 +375,14 @@ public class AnalysisContext {
public StatementBase getStmt() { return stmt_; }
public Analyzer getAnalyzer() { return analyzer_; }
public Set<TAccessEvent> getAccessEvents() { return analyzer_.getAccessEvents(); }
+ public boolean canRewriteStatement() {
+ return !isCreateViewStmt() && !isAlterViewStmt() && !isShowCreateTableStmt();
+ }
public boolean requiresSubqueryRewrite() {
- return analyzer_.containsSubquery() && !isCreateViewStmt() && !isAlterViewStmt()
- && !isShowCreateTableStmt();
+ return canRewriteStatement() && analyzer_.containsSubquery();
+ }
+ public boolean requiresAcidComplexScanRewrite() {
+ return canRewriteStatement() && analyzer_.hasTopLevelAcidCollectionTableRef();
}
public boolean requiresExprRewrite() {
return isQueryStmt() || isInsertStmt() || isCreateTableAsSelectStmt()
@@ -491,6 +496,10 @@ public class AnalysisContext {
new StmtRewriter().rewrite(analysisResult_);
reAnalyze = true;
}
+ if (analysisResult_.requiresAcidComplexScanRewrite()) {
+ new StmtRewriter.AcidRewriter().rewrite(analysisResult_);
+ reAnalyze = true;
+ }
if (!reAnalyze) return;
// The rewrites should have no user-visible effect. Remember the original result
diff --git a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
index 0bb2f24..796429d 100644
--- a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
+++ b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
@@ -215,6 +215,15 @@ public class Analyzer {
isSubquery_ = true;
globalState_.containsSubquery = true;
}
+
+ public void setHasTopLevelAcidCollectionTableRef() {
+ globalState_.hasTopLevelAcidCollectionTableRef = true;
+ }
+
+ public boolean hasTopLevelAcidCollectionTableRef() {
+ return globalState_.hasTopLevelAcidCollectionTableRef;
+ }
+
public boolean setHasPlanHints() { return globalState_.hasPlanHints = true; }
public boolean hasPlanHints() { return globalState_.hasPlanHints; }
public void setHasWithClause() { hasWithClause_ = true; }
@@ -350,6 +359,10 @@ public class Analyzer {
// which needs to be rewritten using joins.
public boolean setOperationNeedsRewrite = false;
+ // True when the query directly scans collection items, e.g.:
+ // select item from complextypestbl.int_array;
+ public boolean hasTopLevelAcidCollectionTableRef = false;
+
// all registered conjuncts (map from expr id to conjunct). We use a LinkedHashMap to
// preserve the order in which conjuncts are added.
public final Map<ExprId, Expr> conjuncts = new LinkedHashMap<>();
@@ -1619,6 +1632,7 @@ public class Analyzer {
public FeCatalog getCatalog() { return globalState_.stmtTableCache.catalog; }
public StmtTableCache getStmtTableCache() { return globalState_.stmtTableCache; }
public Set<String> getAliases() { return aliasMap_.keySet(); }
+ public void removeAlias(String alias) { aliasMap_.remove(alias); }
/**
* Returns list of candidate equi-join conjuncts to be evaluated by the join node
diff --git a/fe/src/main/java/org/apache/impala/analysis/FromClause.java b/fe/src/main/java/org/apache/impala/analysis/FromClause.java
index f81e6be..9feb5fe 100644
--- a/fe/src/main/java/org/apache/impala/analysis/FromClause.java
+++ b/fe/src/main/java/org/apache/impala/analysis/FromClause.java
@@ -24,6 +24,7 @@ import java.util.Iterator;
import java.util.List;
import org.apache.impala.common.AnalysisException;
+import org.apache.impala.util.AcidUtils;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
@@ -34,7 +35,6 @@ import com.google.common.collect.Lists;
* the class it implements the Iterable interface.
*/
public class FromClause extends StmtNode implements Iterable<TableRef> {
-
private final List<TableRef> tableRefs_;
private boolean analyzed_ = false;
@@ -77,6 +77,9 @@ public class FromClause extends StmtNode implements Iterable<TableRef> {
tblRef.setLeftTblRef(leftTblRef);
tblRef.analyze(analyzer);
leftTblRef = tblRef;
+ if (tblRef instanceof CollectionTableRef) {
+ checkTopLevelComplexAcidScan(analyzer, (CollectionTableRef)tblRef);
+ }
}
analyzed_ = true;
}
@@ -100,6 +103,17 @@ public class FromClause extends StmtNode implements Iterable<TableRef> {
}
}
+ private void checkTopLevelComplexAcidScan(Analyzer analyzer,
+ CollectionTableRef collRef) {
+ if (!AcidUtils.isFullAcidTable(
+ collRef.getTable().getMetaStoreTable().getParameters())) {
+ return;
+ }
+ if (collRef.getCollectionExpr() == null) {
+ analyzer.setHasTopLevelAcidCollectionTableRef();
+ }
+ }
+
@Override
public FromClause clone() {
List<TableRef> clone = new ArrayList<>();
@@ -188,4 +202,5 @@ public class FromClause extends StmtNode implements Iterable<TableRef> {
public TableRef get(int i) { return tableRefs_.get(i); }
public void set(int i, TableRef tableRef) { tableRefs_.set(i, tableRef); }
public void add(TableRef t) { tableRefs_.add(t); }
+ public void add(int i, TableRef t) { tableRefs_.add(i, t); }
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java b/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
index 2aa7f8f..19d9dc1 100644
--- a/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
+++ b/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
@@ -26,6 +26,7 @@ import org.apache.impala.analysis.SetOperationStmt.SetOperand;
import org.apache.impala.analysis.SetOperationStmt.SetOperator;
import org.apache.impala.common.AnalysisException;
import org.apache.impala.common.TableAliasGenerator;
+import org.apache.impala.util.AcidUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.common.base.Preconditions;
@@ -1744,4 +1745,84 @@ public class StmtRewriter {
LOG.trace("Rewritten HAVING Clause SQL: " + stmt.toSql(REWRITTEN));
}
}
+
+ /**
+ * Statement rewriter that rewrites queries against ACID tables. We need to do such
+ * rewrites when querying complex types in ACID tables, because ACID scanning needs
+ * to inject slot references to the hidden ACID columns, e.g. rowid.
+ * We can only inject those slot references if the scanner's base tuple is a table level
+ * tuple. This is not true for some complex type queries, e.g.:
+ * SELECT item FROM complextypestbl.int_array;
+ * So the above query needs to be rewritten to:
+ * SELECT item FROM complextypestbl $a$1, $a$1.int_array;
+ * With this rewrite the scanner's root tuple will be the table level tuple that can
+ * contain slot refs to the hidden columns.
+ */
+ static class AcidRewriter extends StmtRewriter {
+ @Override
+ protected void rewriteSelectStmtHook(SelectStmt stmt, Analyzer analyzer)
+ throws AnalysisException {
+ for (int i = 0; i < stmt.fromClause_.size(); ++i) {
+ TableRef tblRef = stmt.fromClause_.get(i);
+ if (tblRef instanceof CollectionTableRef &&
+ AcidUtils.isFullAcidTable(
+ tblRef.getTable().getMetaStoreTable().getParameters())) {
+ CollectionTableRef collRef = (CollectionTableRef)tblRef;
+ if (collRef.getCollectionExpr() == null) {
+ splitCollectionRef(analyzer, stmt, i);
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Rewritten ACID FROM Clause SQL: " + stmt.toSql(REWRITTEN));
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * We have a collection ref in the FROM clause like complextypestbl.int_array. But,
+ * for ACID scans we need to replace it with two table refs, like:
+ * complextypestbl $a$1, $a$1.int_array
+ * This method modifies the FROM clause of the SELECT statement 'stmt' in-place.
+ * @param stmt is the SELECT statement
+ * @param tblRefIdx is the index of the collection ref in the FROM clause that needs
+ * to be split.
+ */
+ private void splitCollectionRef(Analyzer analyzer, SelectStmt stmt, int tableRefIdx)
+ throws AnalysisException {
+ TableRef collTblRef = stmt.fromClause_.get(tableRefIdx);
+ Preconditions.checkState(collTblRef instanceof CollectionTableRef);
+ // Let's create a base table ref with a unique alias.
+ TableName tblName = collTblRef.getTable().getTableName();
+ List<String> rawTblPath = generatePathFrom(tblName);
+ String alias = stmt.getTableAliasGenerator().getNextAlias();
+ TableRef baseTblRef = TableRef.newTableRef(analyzer, rawTblPath, alias);
+ // Let's root the collection table ref from the base table ref.
+ List<String> newCollPath = new ArrayList<>(collTblRef.getPath());
+ if (newCollPath.get(0).equals(tblName.getDb())) {
+ // Let's remove db name from the path.
+ newCollPath.remove(0);
+ }
+ Preconditions.checkState(newCollPath.get(0).equals(tblName.getTbl()));
+ // Let's remove the table name from the path.
+ newCollPath.remove(0);
+ // So instead of <db name>.<table name>, let's start the new path with <alias>.
+ newCollPath.add(0, alias);
+ // Remove the alias of the old collection ref from the analyzer. We need to add
+ // the new collection ref with the same old alias.
+ analyzer.removeAlias(collTblRef.getUniqueAlias());
+ TableRef newCollTblRef =
+ TableRef.newTableRef(analyzer, newCollPath, collTblRef.getUniqueAlias());
+ // Substitute the old collection ref to 'newCollTblRef'.
+ stmt.fromClause_.set(tableRefIdx, newCollTblRef);
+ // Insert the base table ref in front of the collection ref.
+ stmt.fromClause_.add(tableRefIdx, baseTblRef);
+ }
+
+ private List<String> generatePathFrom(TableName tblName) {
+ List<String> rawTblPath = new ArrayList<>();
+ if (tblName.getDb() != null) rawTblPath.add(tblName.getDb());
+ rawTblPath.add(tblName.getTbl());
+ return rawTblPath;
+ }
+ }
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/TableRef.java b/fe/src/main/java/org/apache/impala/analysis/TableRef.java
index 96b4ebe..c476777 100644
--- a/fe/src/main/java/org/apache/impala/analysis/TableRef.java
+++ b/fe/src/main/java/org/apache/impala/analysis/TableRef.java
@@ -142,6 +142,17 @@ public class TableRef extends StmtNode {
// END: Members that need to be reset()
/////////////////////////////////////////
+ /**
+ * Returns a new, resolved, and analyzed table ref.
+ */
+ public static TableRef newTableRef(Analyzer analyzer, List<String> rawPath,
+ String alias) throws AnalysisException {
+ TableRef ret = new TableRef(rawPath, alias);
+ ret = analyzer.resolveTableRef(ret);
+ ret.analyze(analyzer);
+ return ret;
+ }
+
public TableRef(List<String> path, String alias) {
this(path, alias, Privilege.SELECT);
}
diff --git a/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java b/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
index b7bcf4c..b046c3e 100644
--- a/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
+++ b/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
@@ -1552,17 +1552,6 @@ public class SingleNodePlanner {
}
}
if (!areThereDeletedRows) return false;
- final String NOT_SUPPORTED_YET = "This query is not supported on full ACID tables " +
- "that have deleted rows and complex types. As a workaround you can run a " +
- "major compaction.";
- if (hdfsTblRef instanceof CollectionTableRef) {
- throw new AnalysisException(NOT_SUPPORTED_YET);
- }
- for (SlotDescriptor slotDesc : hdfsTblRef.getDesc().getSlots()) {
- if (slotDesc.getItemTupleDesc() != null) {
- throw new AnalysisException(NOT_SUPPORTED_YET);
- }
- }
addAcidSlots(analyzer, hdfsTblRef);
return true;
}
@@ -1630,10 +1619,8 @@ public class SingleNodePlanner {
}
// The followings just create separate scan nodes for insert deltas and delete deltas,
// plus adds a LEFT ANTI HASH JOIN above them.
- TableRef deleteDeltaRef = new TableRef(hdfsTblRef.getPath(),
+ TableRef deleteDeltaRef = TableRef.newTableRef(analyzer, hdfsTblRef.getPath(),
hdfsTblRef.getUniqueAlias() + "-delete-delta");
- deleteDeltaRef = analyzer.resolveTableRef(deleteDeltaRef);
- deleteDeltaRef.analyze(analyzer);
addAcidSlots(analyzer, deleteDeltaRef);
HdfsScanNode deltaScanNode = new HdfsScanNode(ctx_.getNextNodeId(),
hdfsTblRef.getDesc(), conjuncts, insertDeltaPartitions, hdfsTblRef,
diff --git a/testdata/datasets/functional/functional_schema_template.sql b/testdata/datasets/functional/functional_schema_template.sql
index fb7a0b4..5532c58 100644
--- a/testdata/datasets/functional/functional_schema_template.sql
+++ b/testdata/datasets/functional/functional_schema_template.sql
@@ -785,6 +785,21 @@ transactional=true
---- DATASET
functional
---- BASE_TABLE_NAME
+complextypestbl_deleted_rows
+---- COLUMNS
+id bigint
+int_array array<int>
+int_array_array array<array<int>>
+int_map map<string, int>
+int_map_array array<map<string, int>>
+nested_struct struct<a: int, b: array<int>, c: struct<d: array<array<struct<e: int, f: string>>>>, g: map<string, struct<h: struct<i: array<double>>>>>
+---- DEPENDENT_LOAD_ACID
+INSERT INTO TABLE {db_name}{db_suffix}.{table_name} SELECT * FROM {db_name}{db_suffix}.complextypestbl;
+DELETE FROM {db_name}{db_suffix}.{table_name} WHERE id % 2 = 0;
+====
+---- DATASET
+functional
+---- BASE_TABLE_NAME
complextypestbl_non_transactional
---- COLUMNS
id bigint
diff --git a/testdata/datasets/functional/schema_constraints.csv b/testdata/datasets/functional/schema_constraints.csv
index b8a8115..28c73e3 100644
--- a/testdata/datasets/functional/schema_constraints.csv
+++ b/testdata/datasets/functional/schema_constraints.csv
@@ -80,6 +80,7 @@ table_name:complextypes_multifileformat, constraint:restrict_to, table_format:te
table_name:complextypestbl, constraint:restrict_to, table_format:parquet/none/none
table_name:complextypestbl, constraint:restrict_to, table_format:orc/def/block
table_name:complextypestbl_minor_compacted, constraint:restrict_to, table_format:orc/def/block
+table_name:complextypestbl_deleted_rows, constraint:restrict_to, table_format:orc/def/block
table_name:complextypestbl_medium, constraint:restrict_to, table_format:parquet/none/none
table_name:complextypestbl_medium, constraint:restrict_to, table_format:orc/def/block
table_name:complextypestbl_non_transactional, constraint:restrict_to, table_format:orc/def/block
diff --git a/testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test b/testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test
index 171c894..ab982e1 100644
--- a/testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test
+++ b/testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test
@@ -827,3 +827,505 @@ PLAN-ROOT SINK
runtime filters: RF000 -> a.id
row-size=4B cardinality=6.12K
====
+# SELECT primitive type from table that has complex types and deleted rows.
+select id from functional_orc_def.complextypestbl_deleted_rows;
+---- PLAN
+PLAN-ROOT SINK
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| row-size=28B cardinality=2.57K
+|
+|--01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows functional_orc_def.complextypestbl_deleted_rows-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ row-size=28B cardinality=2.57K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+04:EXCHANGE [UNPARTITIONED]
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| row-size=28B cardinality=2.57K
+|
+|--03:EXCHANGE [BROADCAST]
+| |
+| 01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows functional_orc_def.complextypestbl_deleted_rows-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ row-size=28B cardinality=2.57K
+====
+# SELECT nested item with join syntax.
+select item from complextypestbl_deleted_rows t, t.int_array;
+---- PLAN
+PLAN-ROOT SINK
+|
+03:SUBPLAN
+| row-size=36B cardinality=25.68K
+|
+|--06:NESTED LOOP JOIN [CROSS JOIN]
+| | row-size=36B cardinality=10
+| |
+| |--04:SINGULAR ROW SRC
+| | row-size=32B cardinality=1
+| |
+| 05:UNNEST [t.int_array]
+| row-size=0B cardinality=10
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| row-size=32B cardinality=2.57K
+|
+|--01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows t-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows t]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ predicates: !empty(t.int_array)
+ row-size=32B cardinality=2.57K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+08:EXCHANGE [UNPARTITIONED]
+|
+03:SUBPLAN
+| row-size=36B cardinality=25.68K
+|
+|--06:NESTED LOOP JOIN [CROSS JOIN]
+| | row-size=36B cardinality=10
+| |
+| |--04:SINGULAR ROW SRC
+| | row-size=32B cardinality=1
+| |
+| 05:UNNEST [t.int_array]
+| row-size=0B cardinality=10
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| row-size=32B cardinality=2.57K
+|
+|--07:EXCHANGE [BROADCAST]
+| |
+| 01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows t-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows t]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ predicates: !empty(t.int_array)
+ row-size=32B cardinality=2.57K
+====
+# SELECT nested item directly. This gets rewritten to a plan that is similar to the
+# join syntax plan.
+select item from complextypestbl_deleted_rows.int_array;
+---- PLAN
+PLAN-ROOT SINK
+|
+03:SUBPLAN
+| row-size=36B cardinality=25.68K
+|
+|--06:NESTED LOOP JOIN [CROSS JOIN]
+| | row-size=36B cardinality=10
+| |
+| |--04:SINGULAR ROW SRC
+| | row-size=32B cardinality=1
+| |
+| 05:UNNEST [$a$1.int_array int_array]
+| row-size=0B cardinality=10
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| row-size=32B cardinality=2.57K
+|
+|--01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ predicates: !empty($a$1.int_array)
+ row-size=32B cardinality=2.57K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+08:EXCHANGE [UNPARTITIONED]
+|
+03:SUBPLAN
+| row-size=36B cardinality=25.68K
+|
+|--06:NESTED LOOP JOIN [CROSS JOIN]
+| | row-size=36B cardinality=10
+| |
+| |--04:SINGULAR ROW SRC
+| | row-size=32B cardinality=1
+| |
+| 05:UNNEST [$a$1.int_array int_array]
+| row-size=0B cardinality=10
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| row-size=32B cardinality=2.57K
+|
+|--07:EXCHANGE [BROADCAST]
+| |
+| 01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ predicates: !empty($a$1.int_array)
+ row-size=32B cardinality=2.57K
+====
+# SELECT nested items with a CROSS JOIN.
+select a1.item, a2.item
+from complextypestbl_deleted_rows.int_array a1, complextypestbl_deleted_rows.int_array a2;
+---- PLAN
+PLAN-ROOT SINK
+|
+14:NESTED LOOP JOIN [CROSS JOIN]
+| row-size=72B cardinality=659.46M
+|
+|--10:SUBPLAN
+| | row-size=36B cardinality=25.68K
+| |
+| |--13:NESTED LOOP JOIN [CROSS JOIN]
+| | | row-size=36B cardinality=10
+| | |
+| | |--11:SINGULAR ROW SRC
+| | | row-size=32B cardinality=1
+| | |
+| | 12:UNNEST [$a$2.int_array a2]
+| | row-size=0B cardinality=10
+| |
+| 09:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| | row-size=32B cardinality=2.57K
+| |
+| |--08:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$2-delete-delta]
+| | HDFS partitions=1/1 files=2 size=2.87KB
+| | row-size=20B cardinality=1.82K
+| |
+| 07:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$2]
+| HDFS partitions=1/1 files=2 size=4.04KB
+| predicates: !empty($a$2.int_array)
+| row-size=32B cardinality=2.57K
+|
+03:SUBPLAN
+| row-size=36B cardinality=25.68K
+|
+|--06:NESTED LOOP JOIN [CROSS JOIN]
+| | row-size=36B cardinality=10
+| |
+| |--04:SINGULAR ROW SRC
+| | row-size=32B cardinality=1
+| |
+| 05:UNNEST [$a$1.int_array a1]
+| row-size=0B cardinality=10
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| row-size=32B cardinality=2.57K
+|
+|--01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ predicates: !empty($a$1.int_array)
+ row-size=32B cardinality=2.57K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+18:EXCHANGE [UNPARTITIONED]
+|
+14:NESTED LOOP JOIN [CROSS JOIN, BROADCAST]
+| row-size=72B cardinality=659.46M
+|
+|--17:EXCHANGE [BROADCAST]
+| |
+| 10:SUBPLAN
+| | row-size=36B cardinality=25.68K
+| |
+| |--13:NESTED LOOP JOIN [CROSS JOIN]
+| | | row-size=36B cardinality=10
+| | |
+| | |--11:SINGULAR ROW SRC
+| | | row-size=32B cardinality=1
+| | |
+| | 12:UNNEST [$a$2.int_array a2]
+| | row-size=0B cardinality=10
+| |
+| 09:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| | row-size=32B cardinality=2.57K
+| |
+| |--16:EXCHANGE [BROADCAST]
+| | |
+| | 08:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$2-delete-delta]
+| | HDFS partitions=1/1 files=2 size=2.87KB
+| | row-size=20B cardinality=1.82K
+| |
+| 07:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$2]
+| HDFS partitions=1/1 files=2 size=4.04KB
+| predicates: !empty($a$2.int_array)
+| row-size=32B cardinality=2.57K
+|
+03:SUBPLAN
+| row-size=36B cardinality=25.68K
+|
+|--06:NESTED LOOP JOIN [CROSS JOIN]
+| | row-size=36B cardinality=10
+| |
+| |--04:SINGULAR ROW SRC
+| | row-size=32B cardinality=1
+| |
+| 05:UNNEST [$a$1.int_array a1]
+| row-size=0B cardinality=10
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| row-size=32B cardinality=2.57K
+|
+|--15:EXCHANGE [BROADCAST]
+| |
+| 01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ predicates: !empty($a$1.int_array)
+ row-size=32B cardinality=2.57K
+====
+# SELECT a more deeply nested item with a LIMIT.
+select e from complextypestbl_deleted_rows.nested_struct.c.d.item;
+---- PLAN
+PLAN-ROOT SINK
+|
+03:SUBPLAN
+| row-size=36B cardinality=25.68K
+|
+|--06:NESTED LOOP JOIN [CROSS JOIN]
+| | row-size=36B cardinality=10
+| |
+| |--04:SINGULAR ROW SRC
+| | row-size=32B cardinality=1
+| |
+| 05:UNNEST [$a$1.nested_struct.c.d.item item]
+| row-size=0B cardinality=10
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| row-size=32B cardinality=2.57K
+|
+|--01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ predicates: !empty($a$1.nested_struct.c.d.item)
+ row-size=32B cardinality=2.57K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+08:EXCHANGE [UNPARTITIONED]
+|
+03:SUBPLAN
+| row-size=36B cardinality=25.68K
+|
+|--06:NESTED LOOP JOIN [CROSS JOIN]
+| | row-size=36B cardinality=10
+| |
+| |--04:SINGULAR ROW SRC
+| | row-size=32B cardinality=1
+| |
+| 05:UNNEST [$a$1.nested_struct.c.d.item item]
+| row-size=0B cardinality=10
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| row-size=32B cardinality=2.57K
+|
+|--07:EXCHANGE [BROADCAST]
+| |
+| 01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ predicates: !empty($a$1.nested_struct.c.d.item)
+ row-size=32B cardinality=2.57K
+====
+# SELECT with nested SUBPLANs.
+select e from complextypestbl_deleted_rows.nested_struct.c.d a, a.item;
+---- PLAN
+PLAN-ROOT SINK
+|
+03:SUBPLAN
+| row-size=48B cardinality=256.80K
+|
+|--10:NESTED LOOP JOIN [CROSS JOIN]
+| | row-size=48B cardinality=100
+| |
+| |--04:SINGULAR ROW SRC
+| | row-size=32B cardinality=1
+| |
+| 06:SUBPLAN
+| | row-size=16B cardinality=100
+| |
+| |--09:NESTED LOOP JOIN [CROSS JOIN]
+| | | row-size=16B cardinality=10
+| | |
+| | |--07:SINGULAR ROW SRC
+| | | row-size=12B cardinality=1
+| | |
+| | 08:UNNEST [a.item]
+| | row-size=0B cardinality=10
+| |
+| 05:UNNEST [$a$1.nested_struct.c.d a]
+| row-size=0B cardinality=10
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| row-size=32B cardinality=2.57K
+|
+|--01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ predicates: !empty($a$1.nested_struct.c.d)
+ predicates on a: !empty(a.item)
+ row-size=32B cardinality=2.57K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+12:EXCHANGE [UNPARTITIONED]
+|
+03:SUBPLAN
+| row-size=48B cardinality=256.80K
+|
+|--10:NESTED LOOP JOIN [CROSS JOIN]
+| | row-size=48B cardinality=100
+| |
+| |--04:SINGULAR ROW SRC
+| | row-size=32B cardinality=1
+| |
+| 06:SUBPLAN
+| | row-size=16B cardinality=100
+| |
+| |--09:NESTED LOOP JOIN [CROSS JOIN]
+| | | row-size=16B cardinality=10
+| | |
+| | |--07:SINGULAR ROW SRC
+| | | row-size=12B cardinality=1
+| | |
+| | 08:UNNEST [a.item]
+| | row-size=0B cardinality=10
+| |
+| 05:UNNEST [$a$1.nested_struct.c.d a]
+| row-size=0B cardinality=10
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| row-size=32B cardinality=2.57K
+|
+|--11:EXCHANGE [BROADCAST]
+| |
+| 01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows $a$1]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ predicates: !empty($a$1.nested_struct.c.d)
+ predicates on a: !empty(a.item)
+ row-size=32B cardinality=2.57K
+====
+# Parent plan and analytic subplan.
+select a.id, v.key, v.rnum
+from functional_orc_def.complextypestbl_deleted_rows a,
+ (select key, row_number() over (order by key) rnum from a.int_map) v
+ where v.key != 'bad';
+---- PLAN
+PLAN-ROOT SINK
+|
+03:SUBPLAN
+| row-size=60B cardinality=2.57K
+|
+|--09:NESTED LOOP JOIN [CROSS JOIN]
+| | row-size=60B cardinality=1
+| |
+| |--04:SINGULAR ROW SRC
+| | row-size=40B cardinality=1
+| |
+| 08:SELECT
+| | predicates: key != 'bad'
+| | row-size=20B cardinality=1
+| |
+| 07:ANALYTIC
+| | functions: row_number()
+| | order by: key ASC
+| | window: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
+| | row-size=20B cardinality=10
+| |
+| 06:SORT
+| | order by: key ASC
+| | row-size=12B cardinality=10
+| |
+| 05:UNNEST [a.int_map]
+| row-size=0B cardinality=10
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| row-size=40B cardinality=2.57K
+|
+|--01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows a-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows a]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ row-size=40B cardinality=2.57K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+11:EXCHANGE [UNPARTITIONED]
+|
+03:SUBPLAN
+| row-size=60B cardinality=2.57K
+|
+|--09:NESTED LOOP JOIN [CROSS JOIN]
+| | row-size=60B cardinality=1
+| |
+| |--04:SINGULAR ROW SRC
+| | row-size=40B cardinality=1
+| |
+| 08:SELECT
+| | predicates: key != 'bad'
+| | row-size=20B cardinality=1
+| |
+| 07:ANALYTIC
+| | functions: row_number()
+| | order by: key ASC
+| | window: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
+| | row-size=20B cardinality=10
+| |
+| 06:SORT
+| | order by: key ASC
+| | row-size=12B cardinality=10
+| |
+| 05:UNNEST [a.int_map]
+| row-size=0B cardinality=10
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| row-size=40B cardinality=2.57K
+|
+|--10:EXCHANGE [BROADCAST]
+| |
+| 01:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows a-delete-delta]
+| HDFS partitions=1/1 files=2 size=2.87KB
+| row-size=20B cardinality=1.82K
+|
+00:SCAN HDFS [functional_orc_def.complextypestbl_deleted_rows a]
+ HDFS partitions=1/1 files=2 size=4.04KB
+ row-size=40B cardinality=2.57K
+====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/acid-negative.test b/testdata/workloads/functional-query/queries/QueryTest/acid-negative.test
index 59c16df..d26e72c 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/acid-negative.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/acid-negative.test
@@ -39,29 +39,3 @@ select * from test_promotion_fail;
---- CATCH
Found original file with unexpected name
====
----- QUERY
-create table complex_copy like functional_orc_def.complextypestbl;
-====
----- HIVE_QUERY
-use $DATABASE;
-insert into complex_copy select * from functional_orc_def.complextypestbl;
-delete from complex_copy where id % 2 = 0;
-====
----- QUERY
-select id from complex_copy;
----- RESULTS
-1
-3
-5
-7
-====
----- QUERY
-select id, item from complex_copy c, c.int_array;
----- CATCH
-This query is not supported on full ACID tables
-====
----- QUERY
-select item from complex_copy.int_array;
----- CATCH
-This query is not supported on full ACID tables
-====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/full-acid-complex-type-scans.test b/testdata/workloads/functional-query/queries/QueryTest/full-acid-complex-type-scans.test
new file mode 100644
index 0000000..949a7ae
--- /dev/null
+++ b/testdata/workloads/functional-query/queries/QueryTest/full-acid-complex-type-scans.test
@@ -0,0 +1,185 @@
+====
+---- QUERY
+select count(*) from complextypestbl_deleted_rows;
+---- RESULTS
+4
+---- TYPES
+BIGINT
+====
+---- QUERY
+select count(*) from complextypestbl_deleted_rows.int_array;
+---- RESULTS
+3
+---- TYPES
+BIGINT
+====
+---- QUERY
+select count(*) from complextypestbl_deleted_rows.int_array_array.item;
+---- RESULTS
+6
+---- TYPES
+BIGINT
+====
+---- QUERY
+select id from complextypestbl_deleted_rows;
+---- RESULTS
+1
+3
+5
+7
+---- TYPES
+BIGINT
+====
+---- QUERY
+select id, item
+from complextypestbl_deleted_rows t, t.int_array;
+---- RESULTS
+1,1
+1,2
+1,3
+---- TYPES
+BIGINT, INT
+====
+---- QUERY
+select item from complextypestbl_deleted_rows.int_array;
+---- RESULTS
+1
+2
+3
+---- TYPES
+INT
+====
+---- QUERY
+select a1.item, a2.item
+from complextypestbl_deleted_rows.int_array a1, complextypestbl_deleted_rows.int_array a2
+order by 1, 2;
+---- RESULTS
+1,1
+1,2
+1,3
+2,1
+2,2
+2,3
+3,1
+3,2
+3,3
+---- TYPES
+INT, INT
+====
+---- QUERY
+select id, e from complextypestbl_deleted_rows t, t.nested_struct.c.d a, a.item;
+---- RESULTS
+1,10
+1,-10
+1,11
+7,NULL
+---- TYPES
+BIGINT, INT
+====
+---- QUERY
+select e from complextypestbl_deleted_rows.nested_struct.c.d a, a.item;
+---- RESULTS
+10
+-10
+11
+NULL
+---- TYPES
+INT
+====
+---- QUERY
+select id, cnt from complextypestbl_deleted_rows t,
+(select count(item) cnt from t.int_array_array.item) v;
+---- RESULTS
+1,4
+3,0
+5,0
+7,2
+---- TYPES
+BIGINT, BIGINT
+====
+---- QUERY
+-- Materialize scalars, array with no slots, and array of arrays with no slots, with
+-- structs in schema
+select id, t.nested_struct.a, cnt1, cnt2 from complextypestbl t,
+ (select count(*) cnt1 from t.nested_struct.b) v1,
+ t.nested_struct.c.d, (select count(*) cnt2 from d.item arr) v2;
+---- RESULTS
+8,-1,1,1
+1,1,1,2
+1,1,1,1
+2,NULL,1,5
+2,NULL,1,2
+2,NULL,1,0
+2,NULL,1,0
+7,7,3,0
+7,7,3,1
+7,7,3,0
+---- TYPES
+BIGINT, INT, BIGINT, BIGINT
+====
+---- QUERY
+-- Materialize scalars, array with no slots, and array of arrays with no slots, with
+-- structs in schema, with deleted rows
+select id, t.nested_struct.a, cnt1, cnt2 from complextypestbl_deleted_rows t,
+ (select count(*) cnt1 from t.nested_struct.b) v1,
+ t.nested_struct.c.d, (select count(*) cnt2 from d.item arr) v2;
+---- RESULTS
+1,1,1,2
+1,1,1,1
+7,7,3,0
+7,7,3,1
+7,7,3,0
+---- TYPES
+BIGINT, INT, BIGINT, BIGINT
+====
+---- QUERY
+-- Query position slots.
+select id, a1.pos, a2.pos, a2.item
+ from complextypestbl t, t.int_array_array a1, a1.item a2;
+---- RESULTS
+8,0,0,-1
+8,0,1,-2
+1,0,0,1
+1,0,1,2
+1,1,0,3
+1,1,1,4
+2,0,0,NULL
+2,0,1,1
+2,0,2,2
+2,0,3,NULL
+2,1,0,3
+2,1,1,NULL
+2,1,2,4
+7,1,0,5
+7,1,1,6
+---- TYPES
+BIGINT, BIGINT, BIGINT, INT
+====
+---- QUERY
+-- Query position slots against table with deleted rows.
+select id, a1.pos, a2.pos, a2.item
+ from complextypestbl_deleted_rows t, t.int_array_array a1, a1.item a2;
+---- RESULTS
+1,0,0,1
+1,0,1,2
+1,1,0,3
+1,1,1,4
+7,1,0,5
+7,1,1,6
+---- TYPES
+BIGINT, BIGINT, BIGINT, INT
+====
+---- QUERY
+-- Parent plan and analytic subplan.
+select a.id, v.key, v.rnum
+from complextypestbl_deleted_rows a,
+ (select key, row_number() over (order by key) rnum from a.int_map) v
+ where v.key != 'bad';
+---- RESULTS
+1,'k1',1
+1,'k2',2
+7,'k1',1
+7,'k3',2
+---- TYPES
+BIGINT, STRING, BIGINT
+====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/full-acid-rowid.test b/testdata/workloads/functional-query/queries/QueryTest/full-acid-rowid.test
index 2f1570e..a0654fc 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/full-acid-rowid.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/full-acid-rowid.test
@@ -155,3 +155,83 @@ ROW__ID.ROWID, ROW__ID.CURRENTTRANSACTION, ROW__ID.OPERATION, ROW__ID.BUCKET, RO
---- TYPES
BIGINT, BIGINT, INT, INT, BIGINT, INT
====
+---- QUERY
+select row__id.originaltransaction, row__id.bucket, row__id.rowid, id
+from functional_orc_def.complextypestbl_deleted_rows;
+---- LABELS
+ROW__ID.ORIGINALTRANSACTION, ROW__ID.BUCKET, ROW__ID.ROWID, ID
+---- RESULTS
+1,536870912,0,1
+1,536870912,2,3
+1,536870912,4,5
+1,536870912,6,7
+---- TYPES
+BIGINT, INT, BIGINT, BIGINT
+====
+---- QUERY
+select row__id.originaltransaction, row__id.bucket, row__id.rowid, id, item
+from functional_orc_def.complextypestbl_deleted_rows t, t.int_array;
+---- LABELS
+ROW__ID.ORIGINALTRANSACTION, ROW__ID.BUCKET, ROW__ID.ROWID, ID, ITEM
+---- RESULTS
+1,536870912,0,1,1
+1,536870912,0,1,2
+1,536870912,0,1,3
+---- TYPES
+BIGINT, INT, BIGINT, BIGINT, INT
+====
+---- QUERY
+select row__id.originaltransaction, row__id.bucket, row__id.rowid, id, item
+from functional_orc_def.complextypestbl_deleted_rows t, t.int_array_array.item;
+---- LABELS
+ROW__ID.ORIGINALTRANSACTION, ROW__ID.BUCKET, ROW__ID.ROWID, ID, ITEM
+---- RESULTS
+1,536870912,0,1,1
+1,536870912,0,1,2
+1,536870912,0,1,3
+1,536870912,0,1,4
+1,536870912,6,7,5
+1,536870912,6,7,6
+---- TYPES
+BIGINT, INT, BIGINT, BIGINT, INT
+====
+---- QUERY
+select row__id.originaltransaction, row__id.bucket, row__id.rowid, id, e
+from functional_orc_def.complextypestbl_deleted_rows t, t.nested_struct.c.d.item;
+---- LABELS
+ROW__ID.ORIGINALTRANSACTION, ROW__ID.BUCKET, ROW__ID.ROWID, ID, E
+---- RESULTS
+1,536870912,0,1,10
+1,536870912,0,1,-10
+1,536870912,0,1,11
+1,536870912,6,7,NULL
+---- TYPES
+BIGINT, INT, BIGINT, BIGINT, INT
+====
+---- QUERY
+select row__id.originaltransaction, row__id.bucket, row__id.rowid, id, item
+from functional_orc_def.complextypestbl t, t.nested_struct.g.value.h.i;
+---- LABELS
+ROW__ID.ORIGINALTRANSACTION, ROW__ID.BUCKET, ROW__ID.ROWID, ID, ITEM
+---- RESULTS
+1,536870912,0,1,1.1
+1,536870912,1,2,2.2
+1,536870912,1,2,NULL
+1,536870912,4,5,2.2
+1,536870912,4,5,3.3
+---- TYPES
+BIGINT, INT, BIGINT, BIGINT, DOUBLE
+====
+====
+---- QUERY
+select row__id.originaltransaction, row__id.bucket, row__id.rowid, id, item
+from functional_orc_def.complextypestbl_deleted_rows t, t.nested_struct.g.value.h.i;
+---- LABELS
+ROW__ID.ORIGINALTRANSACTION, ROW__ID.BUCKET, ROW__ID.ROWID, ID, ITEM
+---- RESULTS
+1,536870912,0,1,1.1
+1,536870912,4,5,2.2
+1,536870912,4,5,3.3
+---- TYPES
+BIGINT, INT, BIGINT, BIGINT, DOUBLE
+====
diff --git a/tests/query_test/test_acid.py b/tests/query_test/test_acid.py
index 307db54..416e1b9 100644
--- a/tests/query_test/test_acid.py
+++ b/tests/query_test/test_acid.py
@@ -107,6 +107,11 @@ class TestAcid(ImpalaTestSuite):
def test_full_acid_scans(self, vector, unique_database):
self.run_test_case('QueryTest/full-acid-scans', vector, use_db=unique_database)
+ @SkipIfHive2.acid
+ def test_full_acid_complex_type_scans(self, vector, unique_database):
+ self.run_test_case('QueryTest/full-acid-complex-type-scans', vector,
+ use_db='functional_orc_def')
+
# When local CatalogV2 combines with hms_enent_polling enabled, it seems
# that Catalog loads tables by itself, the query statement cannot trigger
# loading tables. As the ValidWriteIdlists is part of table loading profile,
@@ -122,11 +127,6 @@ class TestAcid(ImpalaTestSuite):
self.run_test_case('QueryTest/acid-profile', vector, use_db=unique_database)
@SkipIfHive2.acid
- @SkipIfS3.hive
- @SkipIfABFS.hive
- @SkipIfADLS.hive
- @SkipIfIsilon.hive
- @SkipIfLocal.hive
def test_full_acid_rowid(self, vector, unique_database):
self.run_test_case('QueryTest/full-acid-rowid', vector, use_db=unique_database)