You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2026/02/09 10:59:13 UTC
(impala) branch master updated (09bed366f -> 1753c3aa7)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from 09bed366f IMPALA-14507: Register column-level privilege requests for INSERT
new 825d42307 IMPALA-14541: Remove deprecated CalciteJniFrontend class
new ac5e1fa49 IMPALA-14693: Document Iceberg REST Catalog support
new 49196804f IMPALA-14623: Optimize memory usage for Iceberg file path hashes
new 1753c3aa7 IMPALA-14683: remove AES_128_GCM test case from encryption_exprs_errors
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/catalog/catalog-server.cc | 1 +
be/src/catalog/catalog.cc | 1 +
be/src/rpc/hs2-http-test.cc | 1 +
be/src/runtime/descriptors.cc | 1 +
be/src/service/frontend.cc | 1 +
be/src/util/container-util.h | 1 +
.../util/{compression-util.h => thash128-util.h} | 21 +-
common/thrift/CatalogObjects.thrift | 17 +-
docs/impala.ditamap | 1 +
docs/topics/impala_iceberg.xml | 27 ++
docs/topics/impala_iceberg_rest_catalog.xml | 279 +++++++++++++
fe/pom.xml | 6 +
.../org/apache/impala/catalog/FeIcebergTable.java | 3 +-
.../impala/catalog/IcebergContentFileStore.java | 36 +-
.../apache/impala/planner/IcebergScanPlanner.java | 3 +-
.../main/java/org/apache/impala/util/Hash128.java | 83 ++++
.../java/org/apache/impala/util/IcebergUtil.java | 13 +-
.../impala/catalog/local/LocalCatalogTest.java | 3 +-
.../org/apache/impala/common/FrontendTestBase.java | 11 +-
.../java/org/apache/impala/util/Hash128Test.java | 185 +++++++++
.../org/apache/impala/util/IcebergUtilTest.java | 4 +-
.../impala/calcite/service/CalciteJniFrontend.java | 301 --------------
.../calcite/service/CalciteMetadataHandler.java | 116 +-----
.../impala/calcite/service/CalciteOptimizer.java | 9 -
.../calcite/service/CalcitePhysPlanCreator.java | 8 -
.../impala/calcite/service/CalciteQueryParser.java | 4 -
.../calcite/service/CalciteRelNodeConverter.java | 14 -
.../impala/calcite/service/CalciteValidator.java | 101 -----
.../impala/calcite/service/ExecRequestCreator.java | 434 ---------------------
.../{ => calcite}/planner/CalcitePlannerTest.java | 3 +-
.../{ => calcite}/planner/TestCalciteStats.java | 37 +-
.../planner/TestReduceExprShuttle.java | 98 +++--
.../queries/QueryTest/encryption_exprs_errors.test | 6 -
33 files changed, 718 insertions(+), 1111 deletions(-)
copy be/src/util/{compression-util.h => thash128-util.h} (57%)
create mode 100644 docs/topics/impala_iceberg_rest_catalog.xml
create mode 100644 fe/src/main/java/org/apache/impala/util/Hash128.java
create mode 100644 fe/src/test/java/org/apache/impala/util/Hash128Test.java
delete mode 100644 java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java
delete mode 100644 java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java
delete mode 100644 java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java
rename java/calcite-planner/src/test/java/org/apache/impala/{ => calcite}/planner/CalcitePlannerTest.java (95%)
rename java/calcite-planner/src/test/java/org/apache/impala/{ => calcite}/planner/TestCalciteStats.java (95%)
rename java/calcite-planner/src/test/java/org/apache/impala/{ => calcite}/planner/TestReduceExprShuttle.java (85%)
(impala) 04/04: IMPALA-14683: remove AES_128_GCM test case from encryption_exprs_errors
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 1753c3aa7c4829882fd9d32518da9b7a667f5cf1
Author: Peter Rozsa <pr...@cloudera.com>
AuthorDate: Fri Feb 6 14:39:59 2026 +0100
IMPALA-14683: remove AES_128_GCM test case from encryption_exprs_errors
This change removes AES_128_GCM aes_decrypt test case as the AES_256_GCM
checks the same code path and therefore it gives no additional value.
This test could have failed on systems where the openssl version already
deprecated the AES_128_GCM mode.
Change-Id: If6cf709d6e16a8d0e70037d4545c9de12cacf251
Reviewed-on: http://gerrit.cloudera.org:8080/23947
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Peter Rozsa <pr...@cloudera.com>
---
.../functional-query/queries/QueryTest/encryption_exprs_errors.test | 6 ------
1 file changed, 6 deletions(-)
diff --git a/testdata/workloads/functional-query/queries/QueryTest/encryption_exprs_errors.test b/testdata/workloads/functional-query/queries/QueryTest/encryption_exprs_errors.test
index 74d1c0817..e774fc4cb 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/encryption_exprs_errors.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/encryption_exprs_errors.test
@@ -54,12 +54,6 @@ select aes_decrypt(base64decode('F/DLkSwEikFOlqzXVCysy1JX7Q=='),'123456789012345
UDF ERROR: Invalid AES 'mode': AES_256_GC
====
---- QUERY
-select aes_decrypt("a",'1234567890123456','AES_128_GCM','1234567890123456');
----- RESULTS
----- CATCH
-UDF ERROR: AES GCM input too short to contain a tag
-====
----- QUERY
select aes_decrypt("a",'12345678901234567890123456789012','AES_256_GCM','1234567890123456');
---- RESULTS
---- CATCH
(impala) 01/04: IMPALA-14541: Remove deprecated CalciteJniFrontend class
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 825d423079726072c07021aff14b6e8ed77625e8
Author: Steve Carlin <sc...@cloudera.com>
AuthorDate: Mon Feb 2 10:28:08 2026 -0800
IMPALA-14541: Remove deprecated CalciteJniFrontend class
The CalciteJniFrontend class is no longer used. The main
interface for the Calcite code is now CalciteCompilerFactory
which is instantiated by the Frontend class.
Various other method and classes only used by the CalciteJniFrontend
method have also been deleted.
The test framework has been modified to use the CalciteCompilerFactory
interface.
One minor change also in this commit: Some tests were moved to the
calcite/planner directory to be more consistent.
Change-Id: I3797acf0f23da4f94098e2f8f5f7aa378c9ae91a
Reviewed-on: http://gerrit.cloudera.org:8080/23929
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Steve Carlin <sc...@cloudera.com>
---
.../org/apache/impala/common/FrontendTestBase.java | 11 +-
.../impala/calcite/service/CalciteJniFrontend.java | 301 --------------
.../calcite/service/CalciteMetadataHandler.java | 116 +-----
.../impala/calcite/service/CalciteOptimizer.java | 9 -
.../calcite/service/CalcitePhysPlanCreator.java | 8 -
.../impala/calcite/service/CalciteQueryParser.java | 4 -
.../calcite/service/CalciteRelNodeConverter.java | 14 -
.../impala/calcite/service/CalciteValidator.java | 101 -----
.../impala/calcite/service/ExecRequestCreator.java | 434 ---------------------
.../{ => calcite}/planner/CalcitePlannerTest.java | 3 +-
.../{ => calcite}/planner/TestCalciteStats.java | 37 +-
.../planner/TestReduceExprShuttle.java | 98 +++--
12 files changed, 71 insertions(+), 1065 deletions(-)
diff --git a/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java b/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
index 6a80c7068..0fc3a7c75 100644
--- a/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
+++ b/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
@@ -345,11 +345,20 @@ public class FrontendTestBase extends AbstractFrontendTest {
return parseAndAnalyze(stmt, ctx, frontend_);
}
+ protected AnalysisResult parseAndAnalyze(String stmt, AnalysisContext ctx,
+ CompilerFactory compilerFactory) throws ImpalaException {
+ return parseAndAnalyze(stmt, ctx, frontend_, compilerFactory);
+ }
+
protected AnalysisResult parseAndAnalyze(String stmt, AnalysisContext ctx, Frontend fe)
throws ImpalaException {
+ return parseAndAnalyze(stmt, ctx, fe, new CompilerFactoryImpl());
+ }
+
+ protected AnalysisResult parseAndAnalyze(String stmt, AnalysisContext ctx, Frontend fe,
+ CompilerFactory compilerFactory) throws ImpalaException {
try (FrontendProfile.Scope scope = FrontendProfile.createNewWithScope()) {
ctx.getQueryCtx().getClient_request().setStmt(stmt);
- CompilerFactory compilerFactory = new CompilerFactoryImpl();
ParsedStatement parsedStmt =
compilerFactory.createParsedStatement(ctx.getQueryCtx());
User user = new User(TSessionStateUtil.getEffectiveUser(ctx.getQueryCtx().session));
diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java
deleted file mode 100644
index 65bfe0b95..000000000
--- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java
+++ /dev/null
@@ -1,301 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.impala.calcite.service;
-
-import org.apache.impala.util.EventSequence;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.metadata.DefaultRelMetadataProvider;
-import org.apache.calcite.rel.metadata.JaninoRelMetadataProvider;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
-import org.apache.calcite.sql.SqlExplain;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.impala.analysis.Parser;
-import org.apache.impala.analysis.SelectStmt;
-import org.apache.impala.analysis.StmtMetadataLoader.StmtTableCache;
-import org.apache.impala.calcite.functions.FunctionResolver;
-import org.apache.impala.calcite.operators.ImpalaOperatorTable;
-import org.apache.impala.calcite.rel.node.NodeWithExprs;
-import org.apache.impala.calcite.rel.node.ImpalaPlanRel;
-import org.apache.impala.catalog.BuiltinsDb;
-import org.apache.impala.common.ImpalaException;
-import org.apache.impala.common.JniUtil;
-import org.apache.impala.common.ParseException;
-import org.apache.impala.common.UnsupportedFeatureException;
-import org.apache.impala.service.Frontend;
-import org.apache.impala.service.FrontendProfile;
-import org.apache.impala.service.JniFrontend;
-import org.apache.impala.thrift.TClientRequest;
-import org.apache.impala.thrift.TExecRequest;
-import org.apache.impala.thrift.TQueryCtx;
-import org.apache.impala.thrift.TQueryOptions;
-import org.apache.thrift.TException;
-import org.apache.commons.lang3.StringUtils;
-import org.apache.commons.lang3.exception.ExceptionUtils;
-
-import java.util.List;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * CalciteJniFrontend. This is a frontend that uses the Calcite code
- * to walk through all the steps of compiling the query (e.g. parsing, validating,
- * etc... to generate a TExecRequest that can be used by the execution engine.
- */
-public class CalciteJniFrontend extends JniFrontend {
-
- protected static final Logger LOG =
- LoggerFactory.getLogger(CalciteJniFrontend.class.getName());
-
- private static Pattern LEFT_SEMI = Pattern.compile(".*\\bleft\\ssemi\\b.*",
- Pattern.CASE_INSENSITIVE);
-
- private static Pattern RIGHT_SEMI = Pattern.compile(".*\\bright\\ssemi\\b.*",
- Pattern.CASE_INSENSITIVE);
-
- private static Pattern LEFT_ANTI = Pattern.compile(".*\\bleft\\santi\\b.*",
- Pattern.CASE_INSENSITIVE);
-
- private static Pattern RIGHT_ANTI = Pattern.compile(".*\\bright\\santi\\b.*",
- Pattern.CASE_INSENSITIVE);
-
- private static Pattern INPUT_FILE_NAME = Pattern.compile(".*\\binput__file__name\\b.*",
- Pattern.CASE_INSENSITIVE);
-
- private static Pattern FILE_POSITION = Pattern.compile(".*\\bfile__position\\b.*",
- Pattern.CASE_INSENSITIVE);
-
- private static Pattern TABLE_NOT_FOUND =
- Pattern.compile(".*\\bTable '(.*)' not found\\b.*", Pattern.CASE_INSENSITIVE);
-
- private static Pattern COLUMN_NOT_FOUND =
- Pattern.compile(".*\\bColumn '(.*)' not found\\b.*", Pattern.CASE_INSENSITIVE);
-
- public CalciteJniFrontend(byte[] thriftBackendConfig, boolean isBackendTest)
- throws ImpalaException, TException {
- super(thriftBackendConfig, isBackendTest);
- loadCalciteImpalaFunctions();
- }
-
- /**
- * Jni wrapper for Frontend.createExecRequest(). Accepts a serialized
- * TQueryContext; returns a serialized TQueryExecRequest.
- */
- @Override
- public byte[] createExecRequest(byte[] thriftQueryContext)
- throws ImpalaException {
- // Needed for Calcite's JaninoRelMetadataProvider
- Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader());
-
- QueryContext queryCtx = new QueryContext(thriftQueryContext, getFrontend());
-
- CalciteMetadataHandler mdHandler = null;
-
- try {
- CalciteCompilerFactory.checkOptionSupportedInCalcite(queryCtx.getTQueryCtx());
- } catch (UnsupportedFeatureException e) {
- return runThroughOriginalPlanner(thriftQueryContext, queryCtx);
- }
-
- try (FrontendProfile.Scope scope = FrontendProfile.createNewWithScope()) {
- LOG.info("Using Calcite Planner for the following query: " + queryCtx.getStmt());
-
- // Parse the query
- RelMetadataQuery.THREAD_PROVIDERS.set(
- JaninoRelMetadataProvider.of(DefaultRelMetadataProvider.INSTANCE));
- CalciteQueryParser queryParser = new CalciteQueryParser(queryCtx);
- SqlNode parsedSqlNode = queryParser.parse();
- markEvent(queryParser, parsedSqlNode, queryCtx, "Parsed query");
-
- // Make sure the metadata cache has all the info for the query.
- mdHandler = new CalciteMetadataHandler(parsedSqlNode, queryCtx);
- markEvent(mdHandler, null, queryCtx, "Loaded tables");
-
- boolean isExplain = false;
- if (parsedSqlNode instanceof SqlExplain) {
- isExplain = true;
- parsedSqlNode = ((SqlExplain) parsedSqlNode).getExplicandum();
- }
- // Validate the parsed query
- CalciteValidator validator = new CalciteValidator(mdHandler, queryCtx);
- SqlNode validatedNode = validator.validate(parsedSqlNode);
- markEvent(mdHandler, validatedNode, queryCtx, "Validated query");
-
- // Convert the query to RelNodes which can be optimized
- CalciteRelNodeConverter relNodeConverter = new CalciteRelNodeConverter(validator);
- RelNode logicalPlan = relNodeConverter.convert(validatedNode);
- markEvent(mdHandler, logicalPlan, queryCtx, "Created initial logical plan");
-
- // Optimize the query
- CalciteOptimizer optimizer = new CalciteOptimizer(validator,
- mdHandler.getAnalyzer(), queryCtx.getTimeline(), queryCtx.getTQueryCtx());
- ImpalaPlanRel optimizedPlan = optimizer.optimize(logicalPlan);
- markEvent(mdHandler, optimizedPlan, queryCtx, "Optimized logical plan");
-
- // Create Physical Impala PlanNodes
- CalcitePhysPlanCreator physPlanCreator =
- new CalcitePhysPlanCreator(mdHandler, queryCtx);
- NodeWithExprs rootNode = physPlanCreator.create(optimizedPlan);
- markEvent(mdHandler, rootNode, queryCtx, "Created physical plan");
-
- // Create exec request for the server
- ExecRequestCreator execRequestCreator =
- new ExecRequestCreator(physPlanCreator, queryCtx, mdHandler, isExplain);
- TExecRequest execRequest = execRequestCreator.create(rootNode);
- markEvent(mdHandler, execRequest, queryCtx, "Created exec request");
-
- byte[] serializedRequest = JniUtil.serializeToThrift(execRequest);
- queryCtx.getTimeline().markEvent("Serialized request");
-
- return serializedRequest;
- } catch (ParseException e) {
- throwUnsupportedIfKnownException(e);
- // do a quick parse just to make sure it's not a select stmt. If it is
- // a select statement, we fail the query since all select statements
- // should be run through the Calcite Planner.
- if (Parser.parse(queryCtx.getStmt()) instanceof SelectStmt) {
- throw e;
- }
- LOG.info("Calcite planner failed to parse query: " + queryCtx.getStmt());
- LOG.info("Going to use original Impala planner.");
- return runThroughOriginalPlanner(thriftQueryContext, queryCtx);
- } catch (ImpalaException e) {
- if (mdHandler != null) {
- throwUnsupportedIfKnownException(e, mdHandler.getStmtTableCache());
- }
- throw e;
- } catch (Exception e) {
- throw e;
- }
- }
-
- /**
- * Fallback planner method
- */
- public byte[] runThroughOriginalPlanner(byte[] thriftQueryContext,
- QueryContext queryCtx) throws ImpalaException {
- LOG.info("Using Impala Planner for the following query: " + queryCtx.getStmt());
- return super.createExecRequest(thriftQueryContext);
- }
-
- private void markEvent(CompilerStep compilerStep, Object stepResult,
- QueryContext queryCtx, String stepMessage) {
- LOG.info(stepMessage);
- queryCtx.getTimeline().markEvent(stepMessage);
- compilerStep.logDebug(stepResult);
- }
-
- private static void loadCalciteImpalaFunctions() {
- ImpalaOperatorTable.create(BuiltinsDb.getInstance());
- }
-
- private static void throwUnsupportedIfKnownException(Exception e)
- throws ImpalaException {
- String s = e.toString().replace("\n"," ");;
- if (LEFT_ANTI.matcher(s).matches() || RIGHT_ANTI.matcher(s).matches()) {
- throw new UnsupportedFeatureException("Anti joins not supported.");
- }
- if (LEFT_SEMI.matcher(s).matches() || RIGHT_SEMI.matcher(s).matches()) {
- throw new UnsupportedFeatureException("Semi joins not supported.");
- }
- if (INPUT_FILE_NAME.matcher(s).matches() || FILE_POSITION.matcher(s).matches()) {
- throw new UnsupportedFeatureException("Virtual columns not supported.");
- }
- }
-
- public static void throwUnsupportedIfKnownException(ImpalaException e,
- StmtTableCache stmtTableCache) throws ImpalaException {
- throwUnsupportedIfKnownException(e);
- String s = e.toString().replace("\n"," ");;
- Matcher m = TABLE_NOT_FOUND.matcher(s);
- if (m.matches()) {
- if (CalciteMetadataHandler.anyTableContainsColumn(stmtTableCache, m.group(1))) {
- throw new UnsupportedFeatureException(
- "Complex column " + m.group(1) + " not supported.");
- }
- }
-
- m = COLUMN_NOT_FOUND.matcher(s);
- if (m.matches()) {
- if (CalciteMetadataHandler.anyTableContainsColumn(stmtTableCache, m.group(1))) {
- throw new UnsupportedFeatureException(
- "Complex column " + m.group(1) + " not supported.");
- }
- }
- }
-
- public static class QueryContext {
- private final TQueryCtx queryCtx_;
- private final String stmt_;
- private final String currentDb_;
- private final Frontend frontend_;
- private final EventSequence timeline_;
-
- public QueryContext(byte[] thriftQueryContext,
- Frontend frontend) throws ImpalaException {
- this.queryCtx_ = new TQueryCtx();
- JniUtil.deserializeThrift(queryCtx_, thriftQueryContext);
-
- // hack to match the code in Frontend.java:
- // If unset, set MT_DOP to 0 to simplify the rest of the code.
- if (queryCtx_.getClient_request() != null &&
- queryCtx_.getClient_request().getQuery_options() != null) {
- if (!queryCtx_.getClient_request().getQuery_options().isSetMt_dop()) {
- queryCtx_.getClient_request().getQuery_options().setMt_dop(0);
- }
- }
-
- this.frontend_ = frontend;
- this.stmt_ = queryCtx_.getClient_request().getStmt();
- this.currentDb_ = queryCtx_.getSession().getDatabase();
- this.timeline_ = new EventSequence("Frontend Timeline (Calcite Planner)");
- }
-
- public QueryContext(TQueryOptions options, Frontend frontend,
- String stmt) throws ImpalaException {
- this.queryCtx_ = new TQueryCtx();
- this.queryCtx_.setClient_request(new TClientRequest("FeTests", options));
- this.frontend_ = frontend;
- this.stmt_ = stmt;
- this.currentDb_ = "default";
- this.timeline_ = new EventSequence("Frontend Timeline (Calcite Planner)");
- }
-
- public TQueryCtx getTQueryCtx() {
- return queryCtx_;
- }
-
- public Frontend getFrontend() {
- return frontend_;
- }
-
- public String getStmt() {
- return stmt_;
- }
-
- public String getCurrentDb() {
- return currentDb_;
- }
-
- public EventSequence getTimeline() {
- return timeline_;
- }
- }
-}
diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java
index 67e4e5674..63f6b0c93 100644
--- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java
+++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java
@@ -71,47 +71,11 @@ import org.slf4j.LoggerFactory;
* from catalogd into the coordinator and populating the Calcite schema with
* these tables.
*/
-public class CalciteMetadataHandler implements CompilerStep {
+public class CalciteMetadataHandler {
protected static final Logger LOG =
LoggerFactory.getLogger(CalciteMetadataHandler.class.getName());
- // StmtTableCache needed by Impala's Analyzer class at planning time.
- private final StmtMetadataLoader.StmtTableCache stmtTableCache_;
-
- // CalciteCatalogReader is a context class that holds global information that
- // may be needed by the CalciteTable object
- private final CalciteCatalogReader reader_;
-
- private final Analyzer analyzer_;
-
- public CalciteMetadataHandler(SqlNode parsedNode,
- CalciteJniFrontend.QueryContext queryCtx) throws ImpalaException {
-
- StmtMetadataLoader stmtMetadataLoader = new StmtMetadataLoader(
- queryCtx.getFrontend(), queryCtx.getCurrentDb(), queryCtx.getTimeline());
-
- // retrieve all the tablenames in the query, will be in tableVisitor.tableNames
- TableVisitor tableVisitor = new TableVisitor(queryCtx.getCurrentDb());
- parsedNode.accept(tableVisitor);
-
- // load the relevant tables in the query from catalogd
- this.stmtTableCache_ = stmtMetadataLoader.loadTables(tableVisitor.tableNames_);
-
- this.reader_ = createCalciteCatalogReader(stmtTableCache_,
- queryCtx.getTQueryCtx(), queryCtx.getCurrentDb());
-
- this.analyzer_ = createAnalyzer(stmtTableCache_, queryCtx);
-
- // populate calcite schema. This step needs to be done after the loader because the
- // schema needs to contain the columns in the table for validation, which cannot
- // be done when it's an IncompleteTable
- List<String> errorTables = populateCalciteSchema(reader_,
- queryCtx.getFrontend().getCatalog(), stmtTableCache_, analyzer_);
-
- tableVisitor.checkForComplexTable(stmtTableCache_, errorTables, queryCtx);
- }
-
/**
* Creates CalciteCatalogReader object which will contain information about the schema.
* Since the individual Table objects have reference to the Schema, this also serves
@@ -170,26 +134,6 @@ public class CalciteMetadataHandler implements CompilerStep {
return notFoundTables;
}
- public StmtMetadataLoader.StmtTableCache getStmtTableCache() {
- return stmtTableCache_;
- }
-
- public CalciteCatalogReader getCalciteCatalogReader() {
- return reader_;
- }
-
- public Analyzer getAnalyzer() {
- return analyzer_;
- }
-
- private Analyzer createAnalyzer(StmtMetadataLoader.StmtTableCache stmtTableCache,
- CalciteJniFrontend.QueryContext queryCtx) throws ImpalaException {
- // XXX: using NoopAuthorizationFactory, but this part of the code will
- // eventually either be deprecated or only used in the test environment.
- return new SimplifiedAnalyzer(stmtTableCache, queryCtx.getTQueryCtx(),
- new NoopAuthorizationFactory(), null);
- }
-
/**
* TableVisitor walks through the AST and places all the tables into
* tableNames
@@ -290,55 +234,6 @@ public class CalciteMetadataHandler implements CompilerStep {
}
return false;
}
-
- /**
- * Check if the error table is actually a table with a complex column. There is Impala
- * syntax where a complex column uses the same syntax in the FROM clause as a table.
- * This method is passed in all the tables that are not found and checks to see if
- * the table turned out to be a complex column rather than an actual table. If so,
- * this throws an unsupported feature exception (for the time being). If it's not
- * a table with a complex column, a table not found error will eventually be thrown
- * in a different place.
- */
- public void checkForComplexTable(StmtMetadataLoader.StmtTableCache stmtTableCache,
- List<String> errorTables, CalciteJniFrontend.QueryContext queryCtx)
- throws ImpalaException {
- List<String> allErrorTables = new ArrayList<>();
- allErrorTables.addAll(errorTables_);
- allErrorTables.addAll(errorTables);
- for (String errorTable : allErrorTables) {
- List<String> parts = Splitter.on('.').splitToList(errorTable);
- // if there are 3 parts, then it has to be db.table.column and must be a
- // complex column.
- if (parts.size() > 2) {
- throw new UnsupportedFeatureException("Complex column " +
- errorTable + " not supported.");
- }
- // if there are 2 parts, then it is either a missing db.table or a
- // table.column. We look to see if the column can be found in any
- // of the tables, in which case, it is a complex column being referenced.
- if (parts.size() == 2) {
- // first check the already existing cache for the error table.
- if (anyTableContainsColumn(stmtTableCache, parts.get(1))) {
- throw new UnsupportedFeatureException("Complex column " +
- errorTable + " not supported.");
- }
- // it's possible that the table wasn't loaded yet because this method is
- // only called when there is an error finding a table. Try loading the table
- // from catalogd just in case, and check to see if it's a complex column.
- TableName potentialComplexTable = new TableName(
- currentDb_.toLowerCase(), parts.get(0).toLowerCase());
- StmtMetadataLoader errorLoader = new StmtMetadataLoader(
- queryCtx.getFrontend(), queryCtx.getCurrentDb(), queryCtx.getTimeline());
- StmtMetadataLoader.StmtTableCache errorCache =
- errorLoader.loadTables(Sets.newHashSet(potentialComplexTable));
- if (anyTableContainsColumn(errorCache, parts.get(1))) {
- throw new UnsupportedFeatureException("Complex column " +
- errorTable + " not supported.");
- }
- }
- }
- }
}
public static boolean anyTableContainsColumn(
@@ -351,13 +246,4 @@ public class CalciteMetadataHandler implements CompilerStep {
}
return false;
}
-
- @Override
- public void logDebug(Object resultObject) {
- if (!LOG.isDebugEnabled()) return;
- String allTables = stmtTableCache_.tables.values().stream()
- .map(feTable -> feTable.getName().toString())
- .collect(Collectors.joining( ", " ));
- LOG.debug("Loaded tables: {}", allTables);
- }
}
diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
index fd6305254..a1306850b 100644
--- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
+++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
@@ -106,15 +106,6 @@ public class CalciteOptimizer implements CompilerStep {
this.queryCtx_ = analyzer_.getQueryCtx();
}
- public CalciteOptimizer(CalciteValidator validator, Analyzer analyzer,
- EventSequence timeline, TQueryCtx queryCtx) {
- this.reader_ = validator.getCatalogReader();
- this.validator_ = validator.getSqlValidator();
- this.timeline_ = timeline;
- this.queryCtx_ = queryCtx;
- this.analyzer_ = analyzer;
- }
-
public ImpalaPlanRel optimize(RelNode logPlan) throws ImpalaException {
RelBuilder relBuilder = ImpalaCoreRules.LOGICAL_BUILDER_NO_SIMPLIFY.create(
logPlan.getCluster(), reader_);
diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java
index a1950345b..4d2d7ad17 100644
--- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java
+++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java
@@ -48,14 +48,6 @@ public class CalcitePhysPlanCreator implements CompilerStep {
plannerContext_ = ctx;
}
- public CalcitePhysPlanCreator(CalciteMetadataHandler mdHandler,
- CalciteJniFrontend.QueryContext queryCtx) throws ImpalaException {
- this.analyzer_ = mdHandler.getAnalyzer();
- this.plannerContext_ =
- new PlannerContext(analyzer_, queryCtx.getTQueryCtx(), queryCtx.getTimeline());
-
- }
-
/**
* returns the root plan node along with its output expressions.
*/
diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java
index a72b7c9c3..ce2911646 100644
--- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java
+++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java
@@ -42,10 +42,6 @@ public class CalciteQueryParser implements CompilerStep {
private final String sqlStmt_;
- public CalciteQueryParser(CalciteJniFrontend.QueryContext queryCtx) {
- this.sqlStmt_ = queryCtx.getStmt();
- }
-
public CalciteQueryParser(String stmt) {
this.sqlStmt_ = stmt;
}
diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java
index e82e3531f..71c4233d9 100644
--- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java
+++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java
@@ -97,20 +97,6 @@ public class CalciteRelNodeConverter implements CompilerStep {
cluster_.setMetadataProvider(ImpalaRelMetadataProvider.DEFAULT);
}
- public CalciteRelNodeConverter(CalciteValidator validator) {
- this.typeFactory_ = validator.getTypeFactory();
- this.reader_ = validator.getCatalogReader();
- this.sqlValidator_ = validator.getSqlValidator();
- this.planner_ = new VolcanoPlanner();
- planner_.addRelTraitDef(ConventionTraitDef.INSTANCE);
- planner_.setExecutor(new RemoveUnraggedCharCastRexExecutor());
- cluster_ =
- RelOptCluster.create(planner_, new RexBuilder(typeFactory_));
- viewExpander_ = createViewExpander(validator.getCatalogReader()
- .getRootSchema().plus());
- cluster_.setMetadataProvider(ImpalaRelMetadataProvider.DEFAULT);
- }
-
private static RelOptTable.ViewExpander createViewExpander(SchemaPlus schemaPlus) {
SqlParser.Config parserConfig =
SqlParser.configBuilder().setCaseSensitive(false).build()
diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java
deleted file mode 100644
index 11d2d6d6a..000000000
--- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java
+++ /dev/null
@@ -1,101 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.impala.calcite.service;
-
-import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
-import org.apache.calcite.prepare.CalciteCatalogReader;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.runtime.CalciteContextException;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
-import org.apache.calcite.sql.validate.SqlValidator;
-import org.apache.calcite.sql.validate.SqlValidatorUtil;
-import org.apache.impala.calcite.operators.ImpalaOperatorTable;
-import org.apache.impala.calcite.type.ImpalaTypeCoercionFactory;
-import org.apache.impala.calcite.type.ImpalaTypeSystemImpl;
-import org.apache.impala.calcite.validate.ImpalaConformance;
-import org.apache.impala.common.AnalysisException;
-
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * SqlValidator. Responsible for validating the parsed SQL AST.
- */
-public class CalciteValidator implements CompilerStep {
- protected static final Logger LOG =
- LoggerFactory.getLogger(CalciteValidator.class.getName());
-
- private final CalciteMetadataHandler mdHandler;
- private final CalciteJniFrontend.QueryContext queryCtx;
- private final RelDataTypeFactory typeFactory;
- private final CalciteCatalogReader catalogReader;
- private final SqlValidator sqlValidator;
-
- public CalciteValidator(CalciteMetadataHandler mdHandler,
- CalciteJniFrontend.QueryContext queryCtx) {
- this.mdHandler = mdHandler;
- this.queryCtx = queryCtx;
- this.typeFactory = new JavaTypeFactoryImpl(new ImpalaTypeSystemImpl());
- this.catalogReader = mdHandler.getCalciteCatalogReader();
-
- this.sqlValidator = SqlValidatorUtil.newValidator(
- ImpalaOperatorTable.getInstance(),
- catalogReader, typeFactory,
- SqlValidator.Config.DEFAULT
- // Impala requires identifier expansion (tpcds test queries fail
- // without this)
- .withIdentifierExpansion(true)
- .withConformance(ImpalaConformance.INSTANCE)
- .withTypeCoercionEnabled(true)
- .withTypeCoercionFactory(new ImpalaTypeCoercionFactory())
- );
- }
-
- public SqlNode validate(SqlNode parsedNode) throws AnalysisException {
- try {
- // Validate the initial AST
- SqlNode node = sqlValidator.validate(parsedNode);
- return node;
- } catch (CalciteContextException e) {
- throw new AnalysisException(e.getMessage(), e.getCause());
- }
- }
-
- public RelDataTypeFactory getTypeFactory() {
- return typeFactory;
- }
-
- public SqlValidator getSqlValidator() {
- return sqlValidator;
- }
-
- public CalciteCatalogReader getCatalogReader() {
- return catalogReader;
- }
-
- @Override
- public void logDebug(Object resultObject) {
- if (!(resultObject instanceof SqlNode)) {
- LOG.debug("Finished validator step, but unknown result: {}", resultObject);
- return;
- }
- LOG.debug("Validated node: {}", resultObject);
- }
-}
diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java
deleted file mode 100644
index 66efff738..000000000
--- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java
+++ /dev/null
@@ -1,434 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.impala.calcite.service;
-
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-
-import org.apache.impala.analysis.Analyzer;
-import org.apache.impala.analysis.Expr;
-import org.apache.impala.analysis.JoinOperator;
-import org.apache.impala.calcite.rel.node.NodeWithExprs;
-import org.apache.impala.catalog.FeFsTable;
-import org.apache.impala.catalog.FeTable;
-import org.apache.impala.catalog.HdfsTable;
-import org.apache.impala.common.ImpalaException;
-import org.apache.impala.planner.DataPartition;
-import org.apache.impala.planner.DistributedPlanner;
-import org.apache.impala.planner.JoinNode;
-import org.apache.impala.planner.NestedLoopJoinNode;
-import org.apache.impala.planner.ParallelPlanner;
-import org.apache.impala.planner.PlanFragment;
-import org.apache.impala.planner.PlanNode;
-import org.apache.impala.planner.PlanRootSink;
-import org.apache.impala.planner.Planner;
-import org.apache.impala.planner.PlannerContext;
-import org.apache.impala.planner.RuntimeFilterGenerator;
-import org.apache.impala.planner.SingleNodePlanner;
-import org.apache.impala.planner.SingularRowSrcNode;
-import org.apache.impala.planner.SubplanNode;
-import org.apache.impala.service.Frontend;
-import org.apache.impala.service.FrontendProfile;
-import org.apache.impala.thrift.TColumn;
-import org.apache.impala.thrift.TExecRequest;
-import org.apache.impala.thrift.TExplainLevel;
-import org.apache.impala.thrift.TNetworkAddress;
-import org.apache.impala.thrift.TPlanExecInfo;
-import org.apache.impala.thrift.TPlanFragment;
-import org.apache.impala.thrift.TQueryCtx;
-import org.apache.impala.thrift.TQueryExecRequest;
-import org.apache.impala.thrift.TResultSetMetadata;
-import org.apache.impala.thrift.TRuntimeFilterMode;
-import org.apache.impala.thrift.TRuntimeProfileNode;
-import org.apache.impala.thrift.TStmtType;
-import org.apache.impala.util.EventSequence;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Set;
-
-/**
- * ExecRequestCreator. Responsible for taking a PlanNode and the output Expr list
- * from the top level PlanNode and convert it into a TExecRequest thrift object
- * which is needed by the backend executor code. The input PlanNode tree is
- * an optimized logical tree based on the Calcite rules. This class is also
- * responsible for creating physical node optimizations which are located in
- * the SingleNodePlanner and DistributedPlanner.
- *
- * TODO: This class is very similar to the Frontend.createExecRequest method and
- * contains duplicate code. Accordingly, This class and that method should be
- * refactored to prevent this duplication.
- **/
-public class ExecRequestCreator implements CompilerStep {
- protected static final Logger LOG =
- LoggerFactory.getLogger(ExecRequestCreator.class.getName());
-
- private final CalcitePhysPlanCreator physPlanCreator;
- private final CalciteJniFrontend.QueryContext queryCtx;
- private final CalciteMetadataHandler mdHandler;
- private final boolean isExplain;
-
- public ExecRequestCreator(CalcitePhysPlanCreator physPlanCreator,
- CalciteJniFrontend.QueryContext queryCtx, CalciteMetadataHandler mdHandler,
- boolean isExplain) {
- this.physPlanCreator = physPlanCreator;
- this.queryCtx = queryCtx;
- this.mdHandler = mdHandler;
- this.isExplain = isExplain;
- }
-
- /**
- * create() is the main public method responsible for taking the NodeWithExprs
- * object containing the PlanNode and output Expr list and returning the TExecRequest.
- */
- public TExecRequest create(NodeWithExprs nodeWithExprs) throws ImpalaException {
- TExecRequest request = createExecRequest(nodeWithExprs.planNode_,
- queryCtx.getTQueryCtx(), physPlanCreator.getPlannerContext(),
- physPlanCreator.getAnalyzer(), nodeWithExprs.outputExprs_,
- nodeWithExprs.fieldNames_,
- mdHandler.getStmtTableCache().tables.values());
-
- return request;
- }
-
- /**
- * Create an exec request for Impala to execute based on the supplied plan. This
- * method is similar to the Frontend.createExecRequest method and needs to be
- * refactored.
- */
- private TExecRequest createExecRequest(PlanNode planNodeRoot, TQueryCtx queryCtx,
- PlannerContext plannerContext, Analyzer analyzer, List<Expr> outputExprs,
- List<String> fieldNames,
- Collection<FeTable> tables) throws ImpalaException {
- List<PlanFragment> fragments =
- createPlans(planNodeRoot, analyzer, plannerContext, outputExprs);
- PlanFragment planFragmentRoot = fragments.get(0);
-
- TQueryExecRequest queryExecRequest = new TQueryExecRequest();
- TExecRequest result = createExecRequest(queryCtx, planFragmentRoot,
- queryExecRequest, fieldNames);
- queryExecRequest.setHost_list(getHostLocations(tables));
- queryExecRequest.setCores_required(-1);
-
- // compute resource requirements of the final plan
- Planner.computeResourceReqs(fragments, queryCtx, queryExecRequest,
- plannerContext, true /*isQuery*/);
-
- // create the plan's exec-info and assign fragment idx
- int idx = 0;
- for (PlanFragment planRoot : fragments) {
- TPlanExecInfo tPlanExecInfo = Frontend.createPlanExecInfo(planRoot, queryCtx);
- queryExecRequest.addToPlan_exec_info(tPlanExecInfo);
- for (TPlanFragment fragment : tPlanExecInfo.fragments) {
- fragment.setIdx(idx++);
- }
- }
-
- // create EXPLAIN output after setting everything else
- queryExecRequest.setQuery_ctx(queryCtx); // needed by getExplainString()
-
- List<PlanFragment> allFragments = planFragmentRoot.getNodesPreOrder();
- // to mimic the original planner behavior, use EXTENDED mode explain except for
- // EXPLAIN statements.
- TExplainLevel explainLevel =
- isExplain ? plannerContext.getQueryOptions().getExplain_level() :
- TExplainLevel.EXTENDED;
- if (isExplain) {
- result.setStmt_type(TStmtType.EXPLAIN);
- }
- String explainString = getExplainString(allFragments, explainLevel, plannerContext);
- queryExecRequest.setQuery_plan(explainString);
-
- queryCtx.setDesc_tbl_serialized(
- plannerContext.getRootAnalyzer().getDescTbl().toSerializedThrift());
-
- plannerContext.getTimeline().markEvent("Execution request created");
- EventSequence eventSequence = plannerContext.getTimeline();
- result.setTimeline(eventSequence.toThrift());
-
- TRuntimeProfileNode calciteProfile =
- this.queryCtx.getFrontend().createTRuntimeProfileNode(Frontend.PLANNER_PROFILE);
- this.queryCtx.getFrontend().addPlannerToProfile("CalcitePlanner", null);
- result.setProfile(FrontendProfile.getCurrent().emitAsThrift());
- result.setProfile_children(FrontendProfile.getCurrent().emitChildrenAsThrift());
- if (isExplain) {
- this.queryCtx.getFrontend().createExplainRequest(explainString, result);
- }
- return result;
- }
-
- List<PlanFragment> createPlans(PlanNode planNodeRoot, Analyzer analyzer,
- PlannerContext ctx, List<Expr> outputExprs) throws ImpalaException {
- // Create the values transfer graph in the Analyzer. Note that Calcite plans
- // don't register equijoin predicates in the Analyzer's GlobalState since
- // Calcite should have already done the predicate inferencing analysis.
- // Hence, the GlobalState's registeredValueTransfers will be empty. It is
- // still necessary to instantiate the graph because otherwise
- // RuntimeFilterGenerator tries to de-reference it and encounters NPE.
- analyzer.computeValueTransferGraph();
- Planner.checkForSmallQueryOptimization(planNodeRoot, ctx);
-
- // Although the Calcite plan creates the relative order among different
- // joins, currently it does not swap left and right inputs if the right
- // input has higher estimated cardinality. Do this through Impala's method
- // since we are using Impala's cardinality estimates in the physical planning.
- invertJoins(planNodeRoot, ctx.isSingleNodeExec(), ctx.getRootAnalyzer());
- SingleNodePlanner.validatePlan(ctx, planNodeRoot);
-
- List<PlanFragment> fragments =
- createPlanFragments(planNodeRoot, ctx, analyzer, outputExprs);
- PlanFragment planFragmentRoot = fragments.get(0);
- List<PlanFragment> rootFragments;
- if (Planner.useParallelPlan(ctx)) {
- ParallelPlanner parallelPlanner = new ParallelPlanner(ctx);
- // The rootFragmentList contains the 'root' fragments of each of
- // the parallel plans
- rootFragments = parallelPlanner.createPlans(planFragmentRoot);
- ctx.getTimeline().markEvent("Parallel plans created");
- } else {
- rootFragments = new ArrayList(Arrays.asList(planFragmentRoot));
- }
- return rootFragments;
- }
-
- /**
- * Create one or more plan fragments corresponding to the supplied single node physical
- * plan. This function calls Impala's DistributedPlanner to create the plan fragments
- * and does some post-processing. It is loosely based on Impala's Planner.createPlan()
- * function.
- */
- private List<PlanFragment> createPlanFragments(PlanNode planNodeRoot,
- PlannerContext ctx, Analyzer analyzer,
- List<Expr> outputExprs) throws ImpalaException {
-
- DistributedPlanner distributedPlanner = new DistributedPlanner(ctx);
- List<PlanFragment> fragments;
-
- if (ctx.isSingleNodeExec()) {
- // create one fragment containing the entire single-node plan tree
- fragments = Lists.newArrayList(new PlanFragment(
- ctx.getNextFragmentId(), planNodeRoot, DataPartition.UNPARTITIONED));
- } else {
- fragments = new ArrayList<>();
- // Create distributed plan. For insert/CTAS without limit,
- // isPartitioned should be true.
- // TODO: only query statements are currently supported
- // final boolean isPartitioned = stmtType_ ==
- // TStmtType.DML && !planNodeRoot.hasLimit();
- boolean isPartitioned = false;
- distributedPlanner.createPlanFragments(planNodeRoot, isPartitioned, fragments);
- }
-
- PlanFragment rootFragment = fragments.get(fragments.size() - 1);
- // Create runtime filters.
- if (ctx.getQueryOptions().getRuntime_filter_mode() != TRuntimeFilterMode.OFF) {
- RuntimeFilterGenerator.generateRuntimeFilters(ctx, rootFragment.getPlanRoot());
- ctx.getTimeline().markEvent("Runtime filters computed");
- }
-
- rootFragment.verifyTree();
-
- List<Expr> resultExprs = outputExprs;
- rootFragment.setSink(PlanRootSink.create(ctx, resultExprs, true));
-
- Planner.checkForDisableCodegen(rootFragment.getPlanRoot(), ctx);
- // finalize exchanges: this ensures that for hash partitioned joins, the partitioning
- // keys on both sides of the join have compatible data types
- for (PlanFragment fragment: fragments) {
- fragment.finalizeExchanges(analyzer);
- }
-
- Collections.reverse(fragments);
- ctx.getTimeline().markEvent("Distributed plan created");
- return fragments;
- }
-
- private TExecRequest createExecRequest(TQueryCtx queryCtx,
- PlanFragment planFragmentRoot, TQueryExecRequest queryExecRequest,
- List<String> fieldNames) {
- TExecRequest result = new TExecRequest();
- // NOTE: the below 4 are mandatory fields
- result.setQuery_options(queryCtx.getClient_request().getQuery_options());
-
- // TODO: Need to populate these 3 fields
- result.setAccess_events(new ArrayList<>());
- result.setAnalysis_warnings(new ArrayList<>());
- result.setUser_has_profile_access(true);
-
- result.setQuery_exec_request(queryExecRequest);
-
- // TODO: only query currently supported
- // result.setStmt_type(stmtType_);
- // result.getQuery_exec_request().setStmt_type(stmtType_);
- result.setStmt_type(TStmtType.QUERY);
- result.getQuery_exec_request().setStmt_type(TStmtType.QUERY);
-
- // fill in the metadata using the root fragment's PlanRootSink
- Preconditions.checkState(planFragmentRoot.hasSink());
- List<Expr> outputExprs = new ArrayList<>();
-
- planFragmentRoot.getSink().collectExprs(outputExprs);
- result.setResult_set_metadata(createQueryResultSetMetadata(outputExprs, fieldNames));
-
- return result;
- }
-
- // TODO: Refactor and share Impala's getExplainString()
- private String getExplainString(List<PlanFragment> fragments,
- TExplainLevel explainLevel, PlannerContext ctx) {
- if (explainLevel.ordinal() < TExplainLevel.VERBOSE.ordinal()) {
- // Print the non-fragmented parallel plan.
- return fragments.get(0).getExplainString(ctx.getQueryOptions(), explainLevel);
- }
-
- StringBuffer sb = new StringBuffer();
- // Print the fragmented parallel plan.
- for (int i = 0; i < fragments.size(); ++i) {
- PlanFragment fragment = fragments.get(i);
- sb.append(fragment.getExplainString(ctx.getQueryOptions(), explainLevel));
- if (i < fragments.size() - 1) {
- sb.append("\n");
- }
- }
- return sb.toString();
- }
-
- private TResultSetMetadata createQueryResultSetMetadata(List<Expr> outputExprs,
- List<String> fieldNames) {
- TResultSetMetadata metadata = new TResultSetMetadata();
- int colCnt = outputExprs.size();
- for (int i = 0; i < colCnt; ++i) {
- TColumn colDesc = new TColumn(fieldNames.get(i).toLowerCase(),
- outputExprs.get(i).getType().toThrift());
- metadata.addToColumns(colDesc);
- }
- return metadata;
- }
-
- private List<TNetworkAddress> getHostLocations(Collection<FeTable> tables) {
- Set<TNetworkAddress> hostLocations = new HashSet<>();
- for (FeTable table : tables) {
- if (table instanceof FeFsTable) {
- hostLocations.addAll(((FeFsTable) table).getHostIndex().getList());
- }
- }
- return new ArrayList<>(hostLocations);
- }
-
- /**
- * Traverses the plan tree rooted at 'root' and inverts joins in the following
- * situations:
- * 1. If the left-hand side is a SingularRowSrcNode then we invert the join because
- * then the build side is guaranteed to have only a single row.
- * 2. There is no backend support for distributed non-equi right outer/semi joins,
- * so we invert them (any distributed left semi/outer join is ok).
- * 3. If we estimate that the inverted join is cheaper (see isInvertedJoinCheaper()).
- * Do not invert if relevant stats are missing.
- * The first two inversion rules are independent of the presence/absence of stats.
- * Left Null Aware Anti Joins are never inverted due to lack of backend support.
- * Joins that originate from query blocks with a straight join hint are not inverted.
- * The 'isLocalPlan' parameter indicates whether the plan tree rooted at 'root'
- * will be executed locally within one machine, i.e., without any data exchanges.
- * Return true if any join in the plan rooted at 'root' was inverted.
- *
- * TODO: This should be replaced once we conclude the changes contained in this method
- * are safe to be pushed to Planner.invertJoins, i.e., they do not cause any
- * performance regressions with Impala FE. A couple of differences in this version
- * of invertJoins include:
- * - The computeStats is done here. The Calcite planner will eventually have
- * the computeStats built in during optimization time, but the join stats here
- * are specific for Impala. So we need an explicit computeStats call only for
- * Impala here.
- * - We only need to do the computeStats when the plan changes via inversion, so
- * the method returns a boolean whenever the inversion happens.
- * - A Jira has been filed for this: IMPALA-12958. Probably the best fix for this
- * is to put the inversion inside a Calcite rule.
- *
- **/
- private static boolean invertJoins(PlanNode root, boolean isLocalPlan,
- Analyzer analyzer) {
- boolean inverted = false;
- if (root instanceof SubplanNode) {
- inverted |= invertJoins(root.getChild(0), isLocalPlan, analyzer);
- inverted |= invertJoins(root.getChild(1), true, analyzer);
- } else {
- for (PlanNode child: root.getChildren()) {
- inverted |= invertJoins(child, isLocalPlan, analyzer);
- }
- }
-
- if (root instanceof JoinNode) {
- JoinNode joinNode = (JoinNode) root;
- JoinOperator joinOp = joinNode.getJoinOp();
-
- if (!joinNode.isInvertible(isLocalPlan)) {
- if (inverted) {
- // Re-compute tuple ids since their order must correspond to the order
- // of children.
- root.computeTupleIds();
- // Re-compute stats since PK-FK inference and cardinality may have changed after
- // inversion.
- root.computeStats(analyzer);
- }
- return inverted;
- }
-
- if (joinNode.getChild(0) instanceof SingularRowSrcNode) {
- // Always place a singular row src on the build side because it
- // only produces a single row.
- joinNode.invertJoin();
- inverted = true;
- } else if (!isLocalPlan && joinNode instanceof NestedLoopJoinNode &&
- (joinOp.isRightSemiJoin() || joinOp.isRightOuterJoin())) {
- // The current join is a distributed non-equi right outer or semi join
- // which has no backend support. Invert the join to make it executable.
- joinNode.invertJoin();
- inverted = true;
- } else if (Planner.isInvertedJoinCheaper(joinNode, isLocalPlan)) {
- joinNode.invertJoin();
- inverted = true;
- }
- // Re-compute the numNodes and numInstances based on the new input order
- joinNode.recomputeNodes();
- }
-
- if (inverted) {
- // Re-compute tuple ids because the backend assumes that their order corresponds to
- // the order of children.
- root.computeTupleIds();
- // Re-compute stats since PK-FK inference and cardinality may have changed after
- // inversion.
- root.computeStats(analyzer);
- }
- return inverted;
- }
-
- @Override
- public void logDebug(Object resultObject) {
- if (!(resultObject instanceof TExecRequest)) {
- LOG.debug("Finished exec request step, but unknown result: {}", resultObject);
- }
- LOG.debug("Exec request: {}", resultObject);
- }
-}
diff --git a/java/calcite-planner/src/test/java/org/apache/impala/planner/CalcitePlannerTest.java b/java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/CalcitePlannerTest.java
similarity index 95%
rename from java/calcite-planner/src/test/java/org/apache/impala/planner/CalcitePlannerTest.java
rename to java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/CalcitePlannerTest.java
index 34d0fefdd..3604d1e9e 100644
--- a/java/calcite-planner/src/test/java/org/apache/impala/planner/CalcitePlannerTest.java
+++ b/java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/CalcitePlannerTest.java
@@ -15,10 +15,11 @@
// specific language governing permissions and limitations
// under the License.
-package org.apache.impala.planner;
+package org.apache.impala.calcite.planner;
import java.nio.file.Paths;
+import org.apache.impala.planner.PlannerTestBase;
import org.apache.impala.thrift.TQueryOptions;
import org.junit.Test;
diff --git a/java/calcite-planner/src/test/java/org/apache/impala/planner/TestCalciteStats.java b/java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TestCalciteStats.java
similarity index 95%
rename from java/calcite-planner/src/test/java/org/apache/impala/planner/TestCalciteStats.java
rename to java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TestCalciteStats.java
index c113b85d6..ff90cea4b 100644
--- a/java/calcite-planner/src/test/java/org/apache/impala/planner/TestCalciteStats.java
+++ b/java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TestCalciteStats.java
@@ -15,13 +15,8 @@
// specific language governing permissions and limitations
// under the License.
-package org.apache.impala.planner;
+package org.apache.impala.calcite.planner;
-
-import org.apache.impala.common.ImpalaException;
-import org.apache.impala.common.RuntimeEnv;
-import org.apache.impala.thrift.TQueryOptions;
-import org.apache.impala.thrift.TSessionState;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
@@ -54,14 +49,17 @@ import org.apache.impala.calcite.rel.util.PrunedPartitionHelper;
import org.apache.impala.calcite.schema.ImpalaRelMetadataProvider;
import org.apache.impala.calcite.schema.CalciteTable;
import org.apache.impala.calcite.schema.FilterSelectivityEstimator;
-import org.apache.impala.calcite.service.CalciteJniFrontend.QueryContext;
-import org.apache.impala.calcite.service.CalciteMetadataHandler;
-import org.apache.impala.calcite.service.CalciteQueryParser;
+import org.apache.impala.calcite.service.CalciteAnalysisResult;
+import org.apache.impala.calcite.service.CalciteCompilerFactory;
import org.apache.impala.calcite.service.CalciteRelNodeConverter;
-import org.apache.impala.calcite.service.CalciteValidator;
import org.apache.impala.calcite.type.ImpalaTypeSystemImpl;
import org.apache.impala.catalog.BuiltinsDb;
import org.apache.impala.catalog.FeFsPartition;
+import org.apache.impala.common.ImpalaException;
+import org.apache.impala.common.RuntimeEnv;
+import org.apache.impala.planner.PlannerTestBase;
+import org.apache.impala.thrift.TQueryOptions;
+import org.apache.impala.thrift.TSessionState;
import com.google.common.base.Preconditions;
import static org.junit.Assert.assertEquals;
@@ -106,20 +104,11 @@ public class TestCalciteStats extends PlannerTestBase {
}
private RelNode getRelNodeForQuery(String query) throws ImpalaException {
- QueryContext queryCtx = new QueryContext(options, frontend_, query);
- TSessionState session = new TSessionState();
- session.setConnected_user("dummy");
- queryCtx.getTQueryCtx().setSession(session);
- CalciteQueryParser queryParser = new CalciteQueryParser(queryCtx);
- SqlNode parsedSqlNode = queryParser.parse();
-
- // Make sure the metadata cache has all the info for the query.
- CalciteMetadataHandler mdHandler =
- new CalciteMetadataHandler(parsedSqlNode, queryCtx);
- CalciteValidator validator = new CalciteValidator(mdHandler, queryCtx);
- SqlNode validatedNode = validator.validate(parsedSqlNode);
- CalciteRelNodeConverter relNodeConverter = new CalciteRelNodeConverter(validator);
- return relNodeConverter.convert(validatedNode);
+ CalciteAnalysisResult analysisResult = (CalciteAnalysisResult) parseAndAnalyze(query,
+ feFixture_.createAnalysisCtx(), new CalciteCompilerFactory());
+ CalciteRelNodeConverter relNodeConverter =
+ new CalciteRelNodeConverter(analysisResult);
+ return relNodeConverter.convert(analysisResult.getValidatedNode());
}
private RelMetadataQuery getMQ() {
diff --git a/java/calcite-planner/src/test/java/org/apache/impala/planner/TestReduceExprShuttle.java b/java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TestReduceExprShuttle.java
similarity index 85%
rename from java/calcite-planner/src/test/java/org/apache/impala/planner/TestReduceExprShuttle.java
rename to java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TestReduceExprShuttle.java
index 554740553..6a2d5ba88 100644
--- a/java/calcite-planner/src/test/java/org/apache/impala/planner/TestReduceExprShuttle.java
+++ b/java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TestReduceExprShuttle.java
@@ -15,13 +15,8 @@
// specific language governing permissions and limitations
// under the License.
-package org.apache.impala.planner;
+package org.apache.impala.calcite.planner;
-
-import org.apache.impala.common.ImpalaException;
-import org.apache.impala.common.RuntimeEnv;
-import org.apache.impala.thrift.TQueryOptions;
-import org.apache.impala.thrift.TSessionState;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
@@ -48,15 +43,18 @@ import org.apache.impala.analysis.Analyzer;
import org.apache.impala.analysis.Expr;
import org.apache.impala.calcite.operators.ImpalaOperatorTable;
import org.apache.impala.calcite.rules.ImpalaRexExecutor;
-import org.apache.impala.calcite.service.CalciteJniFrontend.QueryContext;
-import org.apache.impala.calcite.service.CalciteMetadataHandler;
-import org.apache.impala.calcite.service.CalciteQueryParser;
+import org.apache.impala.calcite.service.CalciteAnalysisResult;
+import org.apache.impala.calcite.service.CalciteCompilerFactory;
import org.apache.impala.calcite.service.CalciteRelNodeConverter;
-import org.apache.impala.calcite.service.CalciteValidator;
import org.apache.impala.calcite.type.ImpalaTypeSystemImpl;
import org.apache.impala.catalog.BuiltinsDb;
-import org.apache.impala.thrift.TColumnValue;
+import org.apache.impala.common.ImpalaException;
+import org.apache.impala.common.RuntimeEnv;
+import org.apache.impala.planner.PlannerTestBase;
import org.apache.impala.thrift.TQueryCtx;
+import org.apache.impala.thrift.TQueryOptions;
+import org.apache.impala.thrift.TSessionState;
+import org.apache.impala.thrift.TColumnValue;
import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableMap;
@@ -84,7 +82,7 @@ public class TestReduceExprShuttle extends PlannerTestBase {
@Test
public void testFoldAddTinyInt() {
try {
- ReduceShuttleObjects queryObj = new ReduceShuttleObjects("SELECT 1 + 1");
+ ReduceShuttleObjects queryObj = createReduceShuttleObjects("SELECT 1 + 1");
TColumnValue reducedValue = new TColumnValue();
reducedValue.setShort_val((short)2);
@@ -107,7 +105,7 @@ public class TestReduceExprShuttle extends PlannerTestBase {
try {
String expr = "CAST(1 AS SMALLINT) + CAST(2 AS SMALLINT)";
ReduceShuttleObjects queryObj =
- new ReduceShuttleObjects("SELECT " + expr);
+ createReduceShuttleObjects("SELECT " + expr);
TColumnValue reducedCast1 = new TColumnValue();
reducedCast1.setShort_val((short)1);
TColumnValue reducedCast2 = new TColumnValue();
@@ -139,8 +137,7 @@ public class TestReduceExprShuttle extends PlannerTestBase {
public void testFoldAddInt() {
try {
String expr = "CAST(1 AS INT) + CAST(2 AS INT)";
- ReduceShuttleObjects queryObj =
- new ReduceShuttleObjects("SELECT " + expr);
+ ReduceShuttleObjects queryObj = createReduceShuttleObjects("SELECT " + expr);
TColumnValue reducedCast1 = new TColumnValue();
reducedCast1.setInt_val(1);
TColumnValue reducedCast2 = new TColumnValue();
@@ -171,8 +168,7 @@ public class TestReduceExprShuttle extends PlannerTestBase {
public void testFoldAddDecimal() {
try {
String expr = "1.1 + 2.2";
- ReduceShuttleObjects queryObj =
- new ReduceShuttleObjects("SELECT " + expr);
+ ReduceShuttleObjects queryObj = createReduceShuttleObjects("SELECT " + expr);
TColumnValue reducedCast1 = new TColumnValue();
reducedCast1.setString_val("3.3");
@@ -197,8 +193,7 @@ public class TestReduceExprShuttle extends PlannerTestBase {
public void testFoldConcatString() {
try {
String expr = "CONCAT(cast('a' as string), cast('b' as string))";
- ReduceShuttleObjects queryObj =
- new ReduceShuttleObjects("SELECT " + expr);
+ ReduceShuttleObjects queryObj = createReduceShuttleObjects("SELECT " + expr);
TColumnValue reducedCast1 = new TColumnValue();
reducedCast1.setBinary_val("a".getBytes());
TColumnValue reducedCast2 = new TColumnValue();
@@ -230,8 +225,7 @@ public class TestReduceExprShuttle extends PlannerTestBase {
public void testBoolean() {
try {
String expr = "istrue(false)";
- ReduceShuttleObjects queryObj =
- new ReduceShuttleObjects("SELECT " + expr);
+ ReduceShuttleObjects queryObj = createReduceShuttleObjects("SELECT " + expr);
TColumnValue reducedValue = new TColumnValue();
reducedValue.setBool_val(false);
@@ -252,7 +246,7 @@ public class TestReduceExprShuttle extends PlannerTestBase {
@Test
public void testPartialExpr() {
try {
- ReduceShuttleObjects queryObj = new ReduceShuttleObjects(
+ ReduceShuttleObjects queryObj = createReduceShuttleObjects(
"SELECT 1 + 1 + tinyint_col from functional.alltypestiny");
TColumnValue reducedValue = new TColumnValue();
reducedValue.setShort_val((short)2);
@@ -275,8 +269,7 @@ public class TestReduceExprShuttle extends PlannerTestBase {
public void testNonDeterministic() {
try {
String expr = "rand()";
- ReduceShuttleObjects queryObj =
- new ReduceShuttleObjects("SELECT " + expr);
+ ReduceShuttleObjects queryObj = createReduceShuttleObjects("SELECT " + expr);
TestReducerTmp testReducer = new TestReducerTmp();
List<RexNode> reducedExprs = new ArrayList<>();
RexExecutor executor = new ImpalaRexExecutor(
@@ -295,8 +288,7 @@ public class TestReduceExprShuttle extends PlannerTestBase {
public void testFoldMultipleFields() {
try {
String expr = "1 + 2, 3 + 4";
- ReduceShuttleObjects queryObj =
- new ReduceShuttleObjects("SELECT " + expr);
+ ReduceShuttleObjects queryObj = createReduceShuttleObjects("SELECT " + expr);
TColumnValue reducedAdd1 = new TColumnValue();
reducedAdd1.setShort_val((short)3);
TColumnValue reducedAdd2 = new TColumnValue();
@@ -326,8 +318,7 @@ public class TestReduceExprShuttle extends PlannerTestBase {
try {
String expr = "add_months(cast('2012-07-01 00:00:00' as timestamp), " +
"cast(2 as integer))";
- ReduceShuttleObjects queryObj =
- new ReduceShuttleObjects("SELECT " + expr);
+ ReduceShuttleObjects queryObj = createReduceShuttleObjects("SELECT " + expr);
TColumnValue reducedTime1 = new TColumnValue();
reducedTime1.setString_val("2012-07-01 00:00:00");
TColumnValue reducedTime2 = new TColumnValue();
@@ -356,36 +347,37 @@ public class TestReduceExprShuttle extends PlannerTestBase {
}
+ private ReduceShuttleObjects createReduceShuttleObjects(String query)
+ throws ImpalaException {
+ CalciteAnalysisResult analysisResult =
+ (CalciteAnalysisResult) parseAndAnalyze(query,
+ feFixture_.createAnalysisCtx(), new CalciteCompilerFactory());
+ Analyzer analyzer = analysisResult.getAnalyzer();
+ TQueryCtx queryCtx = analyzer.getQueryCtx();
+ CalciteRelNodeConverter relNodeConverter =
+ new CalciteRelNodeConverter(analysisResult);
+ RelNode rootNode = relNodeConverter.convert(analysisResult.getValidatedNode());
+
+ Preconditions.checkState(rootNode instanceof Project);
+ Project project = (Project) rootNode;
+
+ RelDataTypeFactory typeFactory =
+ new JavaTypeFactoryImpl(new ImpalaTypeSystemImpl());
+ RexBuilder rexBuilder = new RexBuilder(typeFactory);
+ return new ReduceShuttleObjects(analyzer, queryCtx, project, rexBuilder);
+ }
private static class ReduceShuttleObjects {
public final Analyzer analyzer_;
public final TQueryCtx queryCtx_;
- public final RexBuilder rexBuilder_;
public final Project project_;
+ public final RexBuilder rexBuilder_;
- public ReduceShuttleObjects(String query) throws ImpalaException {
- QueryContext queryCtx = new QueryContext(options, frontend_, query);
- queryCtx_ = queryCtx.getTQueryCtx();
-
- TSessionState session = new TSessionState();
- session.setConnected_user("dummy");
- queryCtx.getTQueryCtx().setSession(session);
- CalciteQueryParser queryParser = new CalciteQueryParser(queryCtx);
- SqlNode parsedSqlNode = queryParser.parse();
-
- CalciteMetadataHandler mdHandler =
- new CalciteMetadataHandler(parsedSqlNode, queryCtx);
- analyzer_ = mdHandler.getAnalyzer();
-
- CalciteValidator validator = new CalciteValidator(mdHandler, queryCtx);
- SqlNode validatedNode = validator.validate(parsedSqlNode);
- CalciteRelNodeConverter relNodeConverter = new CalciteRelNodeConverter(validator);
- RelNode rootNode = relNodeConverter.convert(validatedNode);
- Preconditions.checkState(rootNode instanceof Project);
- project_ = (Project) rootNode;
-
- RelDataTypeFactory typeFactory =
- new JavaTypeFactoryImpl(new ImpalaTypeSystemImpl());
- rexBuilder_ = new RexBuilder(typeFactory);
+ public ReduceShuttleObjects(Analyzer analyzer, TQueryCtx queryCtx, Project project,
+ RexBuilder rexBuilder) {
+ analyzer_ = analyzer;
+ queryCtx_ = queryCtx;
+ project_ = project;
+ rexBuilder_ = rexBuilder;;
}
}
(impala) 03/04: IMPALA-14623: Optimize memory usage for Iceberg file path hashes
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 49196804f8e74db346001d951b44777d6e0b2bfe
Author: Arnab Karmakar <ar...@gmail.com>
AuthorDate: Fri Feb 6 02:13:49 2026 -0800
IMPALA-14623: Optimize memory usage for Iceberg file path hashes
Use THash128 Thrift struct (16 bytes) instead of String (64 bytes)
for storing 128-bit XXH128 hashes of Iceberg file paths,
achieving 4x memory reduction.
Key Changes:
- Added THash128 Thrift struct with two i64 fields (high/low)
- Updated TIcebergContentFileStore to use THash128 as map keys
- Created Hash128 Java class with Thrift serialization support
- Migrated from Murmur3 to XXH128 for better performance
- Added C++ comparison operators for THash128
Testing:
- Added comprehensive JUnit tests for Hash128 class
Change-Id: Ie0de793de2434dae3b60c3aa4f87dba203eee3c1
Reviewed-on: http://gerrit.cloudera.org:8080/23946
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>
---
be/src/catalog/catalog-server.cc | 1 +
be/src/catalog/catalog.cc | 1 +
be/src/rpc/hs2-http-test.cc | 1 +
be/src/runtime/descriptors.cc | 1 +
be/src/service/frontend.cc | 1 +
be/src/util/container-util.h | 1 +
be/src/util/thash128-util.h | 38 +++++
common/thrift/CatalogObjects.thrift | 17 +-
fe/pom.xml | 6 +
.../org/apache/impala/catalog/FeIcebergTable.java | 3 +-
.../impala/catalog/IcebergContentFileStore.java | 36 ++--
.../apache/impala/planner/IcebergScanPlanner.java | 3 +-
.../main/java/org/apache/impala/util/Hash128.java | 83 +++++++++
.../java/org/apache/impala/util/IcebergUtil.java | 13 +-
.../impala/catalog/local/LocalCatalogTest.java | 3 +-
.../java/org/apache/impala/util/Hash128Test.java | 185 +++++++++++++++++++++
.../org/apache/impala/util/IcebergUtilTest.java | 4 +-
17 files changed, 364 insertions(+), 33 deletions(-)
diff --git a/be/src/catalog/catalog-server.cc b/be/src/catalog/catalog-server.cc
index 88a0eb571..077ec22cb 100644
--- a/be/src/catalog/catalog-server.cc
+++ b/be/src/catalog/catalog-server.cc
@@ -28,6 +28,7 @@
#include "gen-cpp/CatalogService_types.h"
#include "statestore/statestore-subscriber.h"
#include "util/collection-metrics.h"
+#include "util/container-util.h"
#include "util/debug-util.h"
#include "util/event-metrics.h"
#include "util/json-util.h"
diff --git a/be/src/catalog/catalog.cc b/be/src/catalog/catalog.cc
index 08d8fad6a..4f88d6767 100644
--- a/be/src/catalog/catalog.cc
+++ b/be/src/catalog/catalog.cc
@@ -24,6 +24,7 @@
#include "common/thread-debug-info.h"
#include "rpc/jni-thrift-util.h"
#include "util/backend-gflag-util.h"
+#include "util/container-util.h"
#include "common/names.h"
diff --git a/be/src/rpc/hs2-http-test.cc b/be/src/rpc/hs2-http-test.cc
index b3db0b0ba..35ffbef30 100644
--- a/be/src/rpc/hs2-http-test.cc
+++ b/be/src/rpc/hs2-http-test.cc
@@ -21,6 +21,7 @@
#include "gen-cpp/ImpalaHiveServer2Service.h"
#include "rpc/authentication.h"
+#include "util/container-util.h"
#include "util/kudu-status-util.h"
#include "util/network-util.h"
#include "util/os-util.h"
diff --git a/be/src/runtime/descriptors.cc b/be/src/runtime/descriptors.cc
index d7de726d1..e018f7a9d 100644
--- a/be/src/runtime/descriptors.cc
+++ b/be/src/runtime/descriptors.cc
@@ -35,6 +35,7 @@
#include "gen-cpp/PlanNodes_types.h"
#include "rpc/thrift-util.h"
#include "runtime/runtime-state.h"
+#include "util/container-util.h"
#include "util/hash-util.h"
#include "common/names.h"
diff --git a/be/src/service/frontend.cc b/be/src/service/frontend.cc
index 6255f8fc8..94aa8f140 100644
--- a/be/src/service/frontend.cc
+++ b/be/src/service/frontend.cc
@@ -24,6 +24,7 @@
#include "common/logging.h"
#include "rpc/jni-thrift-util.h"
#include "util/backend-gflag-util.h"
+#include "util/container-util.h"
#include "util/jni-util.h"
#include "util/test-info.h"
#include "util/time.h"
diff --git a/be/src/util/container-util.h b/be/src/util/container-util.h
index 01f586b0f..f472ba48e 100644
--- a/be/src/util/container-util.h
+++ b/be/src/util/container-util.h
@@ -32,6 +32,7 @@
#include "gen-cpp/Types_types.h"
#include "gen-cpp/StatestoreService_types.h"
#include "gen-cpp/common.pb.h"
+#include "util/thash128-util.h"
/// Comparators for types that we commonly use in containers.
namespace impala {
diff --git a/be/src/util/thash128-util.h b/be/src/util/thash128-util.h
new file mode 100644
index 000000000..e92af5898
--- /dev/null
+++ b/be/src/util/thash128-util.h
@@ -0,0 +1,38 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+// either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "gen-cpp/CatalogObjects_types.h"
+
+namespace impala {
+
+/// Comparison operators for THash128 (used as key in std::map)
+inline bool operator==(const THash128& lhs, const THash128& rhs) {
+ return lhs.high == rhs.high && lhs.low == rhs.low;
+}
+
+inline bool operator!=(const THash128& lhs, const THash128& rhs) {
+ return !(lhs == rhs);
+}
+
+/// Required for using THash128 as std::map key
+inline bool operator<(const THash128& lhs, const THash128& rhs) {
+ return std::tie(lhs.high, lhs.low) < std::tie(rhs.high, rhs.low);
+}
+
+} // namespace impala
diff --git a/common/thrift/CatalogObjects.thrift b/common/thrift/CatalogObjects.thrift
index 7b0fd2f88..60009e523 100644
--- a/common/thrift/CatalogObjects.thrift
+++ b/common/thrift/CatalogObjects.thrift
@@ -657,12 +657,19 @@ struct TIcebergPartitionStats {
3: required i64 file_size_in_bytes;
}
-// Contains maps from 128-bit Murmur3 hash of file path to its file descriptor
+// Represents a 128-bit hash value (XXH128 hash of file path)
+// Stored as two 64-bit longs for memory efficiency
+struct THash128 {
+ 1: required i64 high
+ 2: required i64 low
+}
+
+// Contains maps from 128-bit XXH128 hash of file path to its file descriptor
struct TIcebergContentFileStore {
- 1: optional map<string, THdfsFileDesc> path_hash_to_data_file_without_deletes
- 2: optional map<string, THdfsFileDesc> path_hash_to_data_file_with_deletes
- 3: optional map<string, THdfsFileDesc> path_hash_to_position_delete_file
- 4: optional map<string, THdfsFileDesc> path_hash_to_equality_delete_file
+ 1: optional map<THash128, THdfsFileDesc> path_hash_to_data_file_without_deletes
+ 2: optional map<THash128, THdfsFileDesc> path_hash_to_data_file_with_deletes
+ 3: optional map<THash128, THdfsFileDesc> path_hash_to_position_delete_file
+ 4: optional map<THash128, THdfsFileDesc> path_hash_to_equality_delete_file
5: optional bool has_avro
6: optional bool has_orc
7: optional bool has_parquet
diff --git a/fe/pom.xml b/fe/pom.xml
index b280ca87a..25aa141fb 100644
--- a/fe/pom.xml
+++ b/fe/pom.xml
@@ -585,6 +585,12 @@ under the License.
</exclusion>
</exclusions>
</dependency>
+
+ <dependency>
+ <groupId>net.openhft</groupId>
+ <artifactId>zero-allocation-hashing</artifactId>
+ <version>0.16</version>
+ </dependency>
</dependencies>
<reporting>
diff --git a/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java b/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
index 637613f74..dc2a2b988 100644
--- a/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
@@ -84,6 +84,7 @@ import org.apache.impala.thrift.TIcebergTable;
import org.apache.impala.thrift.TNetworkAddress;
import org.apache.impala.thrift.TResultSet;
import org.apache.impala.thrift.TResultSetMetadata;
+import org.apache.impala.util.Hash128;
import org.apache.impala.util.HdfsCachingUtil;
import org.apache.impala.util.IcebergSchemaConverter;
import org.apache.impala.util.IcebergUtil;
@@ -496,7 +497,7 @@ public interface FeIcebergTable extends FeFsTable {
rowBuilder.add(filePath);
// Try to get actual file size from content file store
- String pathHash = IcebergUtil.getFilePathHash(filePath);
+ Hash128 pathHash = IcebergUtil.getFilePathHash(filePath);
FileDescriptor fd = contentFileStore.getDataFileDescriptor(pathHash);
if (fd == null) {
fd = contentFileStore.getDeleteFileDescriptor(pathHash);
diff --git a/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java b/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
index 9baf60ece..95434cea2 100644
--- a/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
+++ b/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
@@ -44,10 +44,12 @@ import org.apache.impala.fb.FbFileDesc;
import org.apache.impala.fb.FbFileMetadata;
import org.apache.impala.fb.FbIcebergDataFileFormat;
import org.apache.impala.fb.FbIcebergMetadata;
+import org.apache.impala.thrift.THash128;
import org.apache.impala.thrift.THdfsFileDesc;
import org.apache.impala.thrift.TIcebergContentFileStore;
import org.apache.impala.thrift.TIcebergPartition;
import org.apache.impala.thrift.TNetworkAddress;
+import org.apache.impala.util.Hash128;
import org.apache.impala.util.IcebergUtil;
import org.apache.impala.util.ListMap;
@@ -97,12 +99,12 @@ public class IcebergContentFileStore {
// Auxiliary class for holding file descriptors in both a map and a list.
private static class MapListContainer {
// Key is the ContentFile path hash, value is FileDescriptor transformed from DataFile
- private final Map<String, EncodedFileDescriptor> fileDescMap_ = new HashMap<>();
+ private final Map<Hash128, EncodedFileDescriptor> fileDescMap_ = new HashMap<>();
private final List<EncodedFileDescriptor> fileDescList_ = new ArrayList<>();
// Adds a file to the map. If this is a new entry, then add it to the list as well.
// Return true if 'desc' was a new entry.
- public boolean add(String pathHash, EncodedFileDescriptor desc) {
+ public boolean add(Hash128 pathHash, EncodedFileDescriptor desc) {
if (fileDescMap_.put(pathHash, desc) == null) {
fileDescList_.add(desc);
return true;
@@ -110,7 +112,7 @@ public class IcebergContentFileStore {
return false;
}
- public IcebergFileDescriptor get(String pathHash) {
+ public IcebergFileDescriptor get(Hash128 pathHash) {
if (!fileDescMap_.containsKey(pathHash)) return null;
return decode(fileDescMap_.get(pathHash));
}
@@ -124,27 +126,27 @@ public class IcebergContentFileStore {
}
// It's enough to only convert the map part to thrift.
- Map<String, THdfsFileDesc> toThrift() {
- Map<String, THdfsFileDesc> ret = new HashMap<>();
- for (Map.Entry<String, EncodedFileDescriptor> entry : fileDescMap_.entrySet()) {
+ Map<THash128, THdfsFileDesc> toThrift() {
+ Map<THash128, THdfsFileDesc> ret = new HashMap<>();
+ for (Map.Entry<Hash128, EncodedFileDescriptor> entry : fileDescMap_.entrySet()) {
ret.put(
- entry.getKey(),
+ entry.getKey().toThrift(),
decode(entry.getValue()).toThrift());
}
return ret;
}
- static MapListContainer fromThrift(Map<String, THdfsFileDesc> thriftMap,
+ static MapListContainer fromThrift(Map<THash128, THdfsFileDesc> thriftMap,
List<TNetworkAddress> networkAddresses, ListMap<TNetworkAddress> hostIndex) {
MapListContainer ret = new MapListContainer();
- for (Map.Entry<String, THdfsFileDesc> entry : thriftMap.entrySet()) {
+ for (Map.Entry<THash128, THdfsFileDesc> entry : thriftMap.entrySet()) {
IcebergFileDescriptor fd = IcebergFileDescriptor.fromThrift(entry.getValue());
Preconditions.checkNotNull(fd);
if (networkAddresses != null) {
Preconditions.checkNotNull(hostIndex);
fd = fd.cloneWithNewHostIndex(networkAddresses, hostIndex);
}
- ret.add(entry.getKey(), encode(fd));
+ ret.add(Hash128.fromThrift(entry.getKey()), encode(fd));
}
return ret;
}
@@ -161,7 +163,7 @@ public class IcebergContentFileStore {
private Map<TIcebergPartition, Integer> partitions_;
// Caches file descriptors loaded during time-travel queries.
- private final ConcurrentMap<String, EncodedFileDescriptor> oldFileDescMap_ =
+ private final ConcurrentMap<Hash128, EncodedFileDescriptor> oldFileDescMap_ =
new ConcurrentHashMap<>();
// Caches the partitions of file descriptors loaded during time-travel queries.
private final ConcurrentMap<TIcebergPartition, Integer> oldPartitionMap_ =
@@ -206,7 +208,7 @@ public class IcebergContentFileStore {
private void storeFile(ContentFile<?> contentFile,
Map<String, IcebergFileDescriptor> fileDescMap, MapListContainer container) {
- Pair<String, EncodedFileDescriptor> pathHashAndFd =
+ Pair<Hash128, EncodedFileDescriptor> pathHashAndFd =
getPathHashAndFd(contentFile, fileDescMap);
if (pathHashAndFd.second != null) {
container.add(pathHashAndFd.first, pathHashAndFd.second);
@@ -217,7 +219,7 @@ public class IcebergContentFileStore {
// This is only invoked during time travel, when we are querying a snapshot that has
// data files which have been removed since.
- public void addOldFileDescriptor(String pathHash, IcebergFileDescriptor desc) {
+ public void addOldFileDescriptor(Hash128 pathHash, IcebergFileDescriptor desc) {
oldFileDescMap_.put(pathHash, encode(desc));
}
@@ -227,19 +229,19 @@ public class IcebergContentFileStore {
oldPartitionMap_.put(partition, id);
}
- public IcebergFileDescriptor getDataFileDescriptor(String pathHash) {
+ public IcebergFileDescriptor getDataFileDescriptor(Hash128 pathHash) {
IcebergFileDescriptor desc = dataFilesWithoutDeletes_.get(pathHash);
if (desc != null) return desc;
return dataFilesWithDeletes_.get(pathHash);
}
- public IcebergFileDescriptor getDeleteFileDescriptor(String pathHash) {
+ public IcebergFileDescriptor getDeleteFileDescriptor(Hash128 pathHash) {
IcebergFileDescriptor ret = positionDeleteFiles_.get(pathHash);
if (ret != null) return ret;
return equalityDeleteFiles_.get(pathHash);
}
- public IcebergFileDescriptor getOldFileDescriptor(String pathHash) {
+ public IcebergFileDescriptor getOldFileDescriptor(Hash128 pathHash) {
if (!oldFileDescMap_.containsKey(pathHash)) return null;
return decode(oldFileDescMap_.get(pathHash));
}
@@ -332,7 +334,7 @@ public class IcebergContentFileStore {
}
}
- private Pair<String, EncodedFileDescriptor> getPathHashAndFd(
+ private Pair<Hash128, EncodedFileDescriptor> getPathHashAndFd(
ContentFile<?> contentFile, Map<String, IcebergFileDescriptor> fileDescMap) {
return new Pair<>(
IcebergUtil.getFilePathHash(contentFile),
diff --git a/fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java b/fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
index 76b16f166..2d0bbd87f 100644
--- a/fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
+++ b/fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
@@ -88,6 +88,7 @@ import org.apache.impala.thrift.TIcebergPartition;
import org.apache.impala.thrift.TIcebergPartitionTransformType;
import org.apache.impala.thrift.TQueryOptions;
import org.apache.impala.thrift.TVirtualColumnType;
+import org.apache.impala.util.Hash128;
import org.apache.impala.util.IcebergUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -801,7 +802,7 @@ public class IcebergScanPlanner {
private Pair<IcebergFileDescriptor, Boolean> getFileDescriptor(ContentFile<?> cf,
IcebergContentFileStore fileStore)
throws ImpalaRuntimeException {
- String pathHash = IcebergUtil.getFilePathHash(cf);
+ Hash128 pathHash = IcebergUtil.getFilePathHash(cf);
IcebergFileDescriptor iceFileDesc = cf.content() == FileContent.DATA ?
fileStore.getDataFileDescriptor(pathHash) :
diff --git a/fe/src/main/java/org/apache/impala/util/Hash128.java b/fe/src/main/java/org/apache/impala/util/Hash128.java
new file mode 100644
index 000000000..718281f00
--- /dev/null
+++ b/fe/src/main/java/org/apache/impala/util/Hash128.java
@@ -0,0 +1,83 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.impala.util;
+
+import com.google.common.base.Preconditions;
+import java.util.Objects;
+import org.apache.impala.thrift.THash128;
+
+/**
+ * Represents a 128-bit hash value using two longs (high and low 64 bits).
+ *
+ * Usage:
+ * - Create: Hash128 hash = IcebergUtil.getFilePathHash(filePath);
+ * - As map key: map.put(hash, value);
+ * - Thrift: THash128 t = hash.toThrift(); Hash128 h = Hash128.fromThrift(t);
+ * - Logging: LOG.debug("Hash: {}", hash.toString());
+ */
+public class Hash128 {
+ private final long high_;
+ private final long low_;
+
+ public Hash128(long high, long low) {
+ high_ = high;
+ low_ = low;
+ }
+
+ public long getHigh() {
+ return high_;
+ }
+
+ public long getLow() {
+ return low_;
+ }
+
+ /**
+ * Converts this hash to a Thrift THash128 struct for serialization.
+ */
+ public THash128 toThrift() {
+ return new THash128(high_, low_);
+ }
+
+ /**
+ * Creates a Hash128 from a Thrift THash128 struct.
+ */
+ public static Hash128 fromThrift(THash128 thrift) {
+ Preconditions.checkNotNull(thrift);
+ return new Hash128(thrift.getHigh(), thrift.getLow());
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (o == null || getClass() != o.getClass()) return false;
+ Hash128 hash128 = (Hash128) o;
+ return high_ == hash128.high_ && low_ == hash128.low_;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(high_, low_);
+ }
+
+ @Override
+ public String toString() {
+ // Return hexadecimal representation for debugging
+ return String.format("%016x%016x", high_, low_);
+ }
+}
diff --git a/fe/src/main/java/org/apache/impala/util/IcebergUtil.java b/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
index 8c8736cbd..38f10a20d 100644
--- a/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
+++ b/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
@@ -734,16 +734,17 @@ public class IcebergUtil {
}
/**
- * Use ContentFile path to generate 128-bit Murmur3 hash as map key, cached in memory
+ * Use ContentFile path to generate 128-bit XXH128 hash as map key, cached in memory
*/
- public static String getFilePathHash(ContentFile contentFile) {
+ public static Hash128 getFilePathHash(ContentFile contentFile) {
return getFilePathHash(contentFile.path().toString());
}
- public static String getFilePathHash(String path) {
- Hasher hasher = Hashing.murmur3_128().newHasher();
- hasher.putUnencodedChars(path);
- return hasher.hash().toString();
+ public static Hash128 getFilePathHash(String path) {
+ net.openhft.hashing.LongTupleHashFunction hasher =
+ net.openhft.hashing.LongTupleHashFunction.xx128();
+ long[] result = hasher.hashChars(path);
+ return new Hash128(result[0], result[1]);
}
/**
diff --git a/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java b/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
index deb8bd647..a213336c3 100644
--- a/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
+++ b/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
@@ -52,6 +52,7 @@ import org.apache.impala.thrift.TMetadataOpcode;
import org.apache.impala.thrift.TNetworkAddress;
import org.apache.impala.thrift.TPartialTableInfo;
import org.apache.impala.thrift.TResultSet;
+import org.apache.impala.util.Hash128;
import org.apache.impala.util.IcebergUtil;
import org.apache.impala.util.ListMap;
import org.apache.impala.util.MetaStoreUtil;
@@ -292,7 +293,7 @@ public class LocalCatalogTest {
// For this test table the manifest files contain data paths without FS-scheme, so
// they are loaded to the file content store without them.
path = path.substring(path.indexOf("/test-warehouse"));
- String pathHash = IcebergUtil.getFilePathHash(path);
+ Hash128 pathHash = IcebergUtil.getFilePathHash(path);
FileDescriptor catalogFd = catalogFileStore.getDataFileDescriptor(pathHash);
assertEquals(localFd.getNumFileBlocks(), 1);
FbFileBlock localBlock = localFd.getFbFileBlock(0);
diff --git a/fe/src/test/java/org/apache/impala/util/Hash128Test.java b/fe/src/test/java/org/apache/impala/util/Hash128Test.java
new file mode 100644
index 000000000..a70a3f6ad
--- /dev/null
+++ b/fe/src/test/java/org/apache/impala/util/Hash128Test.java
@@ -0,0 +1,185 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.impala.util;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertNull;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.impala.thrift.THash128;
+import org.junit.Test;
+
+/**
+ * Unit tests for the Hash128 class which stores 128-bit hash values
+ * as two longs for memory efficiency.
+ */
+public class Hash128Test {
+
+ @Test
+ public void testBasicConstruction() {
+ Hash128 hash = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654321L);
+ assertEquals(0x1234567890ABCDEFL, hash.getHigh());
+ assertEquals(0xFEDCBA0987654321L, hash.getLow());
+ }
+
+ @Test
+ public void testEquality() {
+ Hash128 hash1 = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654321L);
+ Hash128 hash2 = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654321L);
+ Hash128 hash3 = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654320L);
+ Hash128 hash4 = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654321L);
+
+ // Test equals
+ assertEquals(hash1, hash2);
+ assertEquals(hash1, hash4);
+ assertNotEquals(hash1, hash3);
+ assertNotEquals(hash1, null);
+ assertNotEquals(hash1, "not a hash");
+
+ // Test hashCode consistency
+ assertEquals(hash1.hashCode(), hash2.hashCode());
+ assertEquals(hash1.hashCode(), hash4.hashCode());
+ }
+
+ @Test
+ public void testHashMapUsage() {
+ // Hash128 should work correctly as a HashMap key
+ Map<Hash128, String> map = new HashMap<>();
+ Hash128 key1 = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654321L);
+ Hash128 key2 = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654321L);
+ Hash128 key3 = new Hash128(0xAAAAAAAAAAAAAAAAL, 0xBBBBBBBBBBBBBBBBL);
+
+ map.put(key1, "value1");
+ map.put(key3, "value3");
+
+ // key2 should retrieve the same value as key1 since they're equal
+ assertEquals("value1", map.get(key2));
+ assertEquals("value3", map.get(key3));
+ assertNull(map.get(new Hash128(0x0L, 0x0L)));
+ }
+
+ @Test
+ public void testToString() {
+ Hash128 hash = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654321L);
+ String str = hash.toString();
+
+ // Should be 32 hex characters (16 for each long)
+ assertEquals(32, str.length());
+ assertEquals("1234567890abcdeffedcba0987654321", str);
+
+ // Test with zero values
+ Hash128 zeroHash = new Hash128(0x0L, 0x0L);
+ assertEquals("00000000000000000000000000000000", zeroHash.toString());
+
+ // Test with max values
+ Hash128 maxHash = new Hash128(0xFFFFFFFFFFFFFFFFL, 0xFFFFFFFFFFFFFFFFL);
+ assertEquals("ffffffffffffffffffffffffffffffff", maxHash.toString());
+ }
+
+ @Test
+ public void testThriftRoundTrip() {
+ Hash128 original = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654321L);
+
+ // Convert to Thrift and back
+ THash128 thrift = original.toThrift();
+ Hash128 restored = Hash128.fromThrift(thrift);
+
+ assertEquals(original, restored);
+ assertEquals(original.getHigh(), restored.getHigh());
+ assertEquals(original.getLow(), restored.getLow());
+ }
+
+ @Test
+ public void testThriftStructFields() {
+ Hash128 hash = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654321L);
+ THash128 thrift = hash.toThrift();
+
+ // Verify Thrift struct has correct values
+ assertEquals(0x1234567890ABCDEFL, thrift.getHigh());
+ assertEquals(0xFEDCBA0987654321L, thrift.getLow());
+ }
+
+ @Test
+ public void testThriftWithEdgeCases() {
+ // Test with zero
+ Hash128 zero = new Hash128(0x0L, 0x0L);
+ THash128 thriftZero = zero.toThrift();
+ Hash128 restoredZero = Hash128.fromThrift(thriftZero);
+ assertEquals(zero, restoredZero);
+
+ // Test with all ones
+ Hash128 ones = new Hash128(0xFFFFFFFFFFFFFFFFL, 0xFFFFFFFFFFFFFFFFL);
+ THash128 thriftOnes = ones.toThrift();
+ Hash128 restoredOnes = Hash128.fromThrift(thriftOnes);
+ assertEquals(ones, restoredOnes);
+
+ // Test with high bit patterns
+ Hash128 pattern = new Hash128(0x8000000000000000L, 0x8000000000000001L);
+ THash128 thriftPattern = pattern.toThrift();
+ Hash128 restoredPattern = Hash128.fromThrift(thriftPattern);
+ assertEquals(pattern, restoredPattern);
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testNullThriftInput() {
+ Hash128.fromThrift(null);
+ }
+
+ @Test
+ public void testConsistentHashing() {
+ // Multiple Hash128 objects with same values should have consistent hashCode
+ Hash128 hash1 = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654321L);
+ Hash128 hash2 = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654321L);
+
+ // Create and recreate through serialization
+ Hash128 hash3 = Hash128.fromThrift(hash1.toThrift());
+
+ assertEquals(hash1.hashCode(), hash2.hashCode());
+ assertEquals(hash1.hashCode(), hash3.hashCode());
+ }
+
+ @Test
+ public void testDifferentHashesDifferentHashCodes() {
+ // Different Hash128 objects should (likely) have different hashCodes
+ Hash128 hash1 = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654321L);
+ Hash128 hash2 = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654320L);
+ Hash128 hash3 = new Hash128(0x1234567890ABCDEEL, 0xFEDCBA0987654321L);
+
+ // Note: hashCode collision is possible but unlikely
+ assertNotEquals(hash1.hashCode(), hash2.hashCode());
+ assertNotEquals(hash1.hashCode(), hash3.hashCode());
+ }
+
+ @Test
+ public void testThriftAsMapKey() {
+ // Verify that Hash128 can be used as a map key through Thrift round-trip
+ Map<Hash128, String> map = new HashMap<>();
+ Hash128 key1 = new Hash128(0x1234567890ABCDEFL, 0xFEDCBA0987654321L);
+ map.put(key1, "value1");
+
+ // Serialize and deserialize
+ THash128 thriftKey = key1.toThrift();
+ Hash128 key2 = Hash128.fromThrift(thriftKey);
+
+ // Should retrieve the same value
+ assertEquals("value1", map.get(key2));
+ }
+}
diff --git a/fe/src/test/java/org/apache/impala/util/IcebergUtilTest.java b/fe/src/test/java/org/apache/impala/util/IcebergUtilTest.java
index 15e8c9879..d00a127ae 100644
--- a/fe/src/test/java/org/apache/impala/util/IcebergUtilTest.java
+++ b/fe/src/test/java/org/apache/impala/util/IcebergUtilTest.java
@@ -259,9 +259,9 @@ public class IcebergUtilTest {
*/
@Test
public void testGetDataFilePathHash() {
- String hash = getFilePathHash(FILE_A);
+ Hash128 hash = getFilePathHash(FILE_A);
assertNotNull(hash);
- String hash2 = getFilePathHash(FILE_A);
+ Hash128 hash2 = getFilePathHash(FILE_A);
assertEquals(hash, hash2);
}
(impala) 02/04: IMPALA-14693: Document Iceberg REST Catalog support
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit ac5e1fa4986d3a024a2ab5f2ebb382f7efb0cd69
Author: Peter Rozsa <pr...@cloudera.com>
AuthorDate: Fri Jan 23 14:23:31 2026 +0100
IMPALA-14693: Document Iceberg REST Catalog support
This change adds a new docs page about Iceberg REST Catalog support that
presents configuration options, deployment modes, and example usecases.
Change-Id: I1a125bfcd5c4058f6cbcd81b8d56c0571a06981d
Generated-by: Github Copilot (Claude Sonnet 4.5)
Reviewed-on: http://gerrit.cloudera.org:8080/23897
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
docs/impala.ditamap | 1 +
docs/topics/impala_iceberg.xml | 27 +++
docs/topics/impala_iceberg_rest_catalog.xml | 279 ++++++++++++++++++++++++++++
3 files changed, 307 insertions(+)
diff --git a/docs/impala.ditamap b/docs/impala.ditamap
index 88854d59a..a5e229b85 100644
--- a/docs/impala.ditamap
+++ b/docs/impala.ditamap
@@ -350,6 +350,7 @@ under the License.
<topicref href="topics/impala_kudu.xml"/>
<topicref href="topics/impala_hbase.xml"/>
<topicref rev="4.1.0" href="topics/impala_iceberg.xml"/>
+ <topicref rev="5.0.0" href="topics/impala_iceberg_rest_catalog.xml"/>
<topicref rev="5.0.0" href="topics/impala_paimon.xml"/>
<topicref href="topics/impala_s3.xml"/>
<topicref rev="2.9.0" href="topics/impala_adls.xml"/>
diff --git a/docs/topics/impala_iceberg.xml b/docs/topics/impala_iceberg.xml
index 70e6b446d..cac82b10d 100644
--- a/docs/topics/impala_iceberg.xml
+++ b/docs/topics/impala_iceberg.xml
@@ -1213,4 +1213,31 @@ iceberg.io.manifest.cache.max-content-length=8388608;
</p>
</conbody>
</concept>
+
+ <concept id="iceberg_rest_catalog">
+ <title>Using Impala with Iceberg REST Catalog</title>
+ <titlealts audience="PDF"><navtitle>REST Catalog</navtitle></titlealts>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Concepts"/>
+ <data name="Category" value="Iceberg"/>
+ <data name="Category" value="REST Catalog"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+ <p>
+ <indexterm audience="hidden">Iceberg REST Catalog</indexterm>
+ Impala supports connecting directly to Iceberg REST Catalog servers.
+ For complete documentation on using Impala with Iceberg REST Catalog, including configuration,
+ deployment modes, see
+ <xref href="impala_iceberg_rest_catalog.xml"/>.
+ </p>
+
+ <p>
+ For more information on the Apache Iceberg REST Catalog specification, see
+ <xref href="https://iceberg.apache.org/rest-catalog-spec/" scope="external" format="html">Iceberg REST Catalog</xref>.
+ </p>
+ </conbody>
+ </concept>
</concept>
diff --git a/docs/topics/impala_iceberg_rest_catalog.xml b/docs/topics/impala_iceberg_rest_catalog.xml
new file mode 100644
index 000000000..69786bb91
--- /dev/null
+++ b/docs/topics/impala_iceberg_rest_catalog.xml
@@ -0,0 +1,279 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="impala_iceberg_rest_catalog">
+ <title>Using Impala with Iceberg REST Catalog</title>
+ <titlealts audience="PDF"><navtitle>Iceberg REST Catalog</navtitle></titlealts>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Concepts"/>
+ <data name="Category" value="Iceberg"/>
+ <data name="Category" value="REST Catalog"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+ <p>
+ <indexterm audience="hidden">Iceberg REST Catalog</indexterm>
+ Impala supports Apache Iceberg REST Catalog, which enables
+ Impala to connect directly to Iceberg REST Catalog servers. This feature allows you to:
+ </p>
+ <ul>
+ <li>Run Impala without Hive Metastore (HMS) or catalogd dependency</li>
+ <li>Connect to multiple REST catalog endpoints simultaneously</li>
+ <li>Use hybrid deployments with both HMS and REST catalogs</li>
+ <li>Integrate with open-source REST catalog implementations</li>
+ </ul>
+
+ <p>
+ For more information on the Apache Iceberg REST Catalog specification, see
+ <xref href="https://iceberg.apache.org/rest-catalog-spec/" scope="external" format="html">Iceberg REST Catalog</xref>.
+ </p>
+
+ <p outputclass="toc inpage"/>
+ </conbody>
+
+ <concept id="rest_catalog_overview">
+ <title>REST Catalog Overview</title>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Concepts"/>
+ </metadata>
+ </prolog>
+ <conbody>
+ <p>
+ The Iceberg REST Catalog is a standardized HTTP API for managing Iceberg table metadata.
+ Unlike traditional Hive Metastore (HMS) deployments, REST Catalog provides a lightweight,
+ cloud-native approach to catalog management that works seamlessly across different compute engines.
+ </p>
+ </conbody>
+ </concept>
+
+ <concept id="rest_catalog_configuration">
+ <title>Configuration</title>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Concepts"/>
+ </metadata>
+ </prolog>
+ <conbody>
+ <p>
+ REST Catalog configuration in Impala uses Java properties files that specify connection
+ details and authentication settings. Configuration files must be placed in a directory
+ specified by the <codeph>--catalog_config_dir</codeph> flag.
+ </p>
+
+ <section id="config_file_format">
+ <title>Configuration File Format</title>
+ <p>
+ Configuration files use the standard Java Properties format, with key=value pairs. Each file
+ represents a separate REST Catalog.
+ </p>
+ </section>
+
+ <section id="required_properties">
+ <title>Required Properties</title>
+ <p>
+ The following properties are required in every REST Catalog configuration file:
+ </p>
+ <ul>
+ <li>
+ <codeph>connector.name</codeph>: Must be set to <codeph>iceberg</codeph>
+ </li>
+ <li>
+ <codeph>iceberg.catalog.type</codeph>: Must be set to <codeph>rest</codeph>
+ </li>
+ <li>
+ <codeph>uri</codeph> or <codeph>iceberg.rest-catalog.uri</codeph>: REST Catalog endpoint URL
+ </li>
+ </ul>
+ <p>
+ <b>Example:</b>
+ </p>
+ <codeblock>connector.name=iceberg
+iceberg.catalog.type=rest
+uri=http://rest-catalog-server:9084</codeblock>
+ </section>
+
+ <section id="optional_properties">
+ <title>Optional Properties</title>
+ <p>
+ <b>Warehouse Location:</b>
+ </p>
+ <codeblock>warehouse=hdfs://namenode:8020/warehouse/iceberg
+# Or:
+iceberg.rest-catalog.warehouse=hdfs://namenode:8020/warehouse/iceberg</codeblock>
+ <p>
+ <b>Catalog Prefix:</b>
+ </p>
+ <codeblock>prefix=my_catalog
+# Or:
+iceberg.rest-catalog.prefix=my_catalog</codeblock>
+ <p>
+ <b>OAuth Configuration:</b>
+ </p>
+ <codeblock>iceberg.rest-catalog.security=OAUTH2
+iceberg.rest-catalog.oauth2.credential=client_secret
+iceberg.rest-catalog.oauth2.server-uri=http://localhost:7070/realms/lakekeeper-realm/protocol/openid-connect/token</codeblock>
+ </section>
+
+ <section id="environment_variable_substitution">
+ <title>Environment Variable Substitution</title>
+ <p>
+ Impala supports environment variable substitution in .properties files to securely
+ manage credentials and environment-specific settings. This feature is particularly
+ useful for storing secrets outside of configuration files.
+ </p>
+ <p>
+ <b>Syntax:</b> <codeph>${ENV:VARIABLE_NAME}</codeph>
+ </p>
+ <p>
+ <b>Example:</b>
+ </p>
+ <codeblock>iceberg.rest-catalog.oauth2.credential=${ENV:OAUTH_CLIENT_SECRET}
+connection-password=${ENV:DB_PASSWORD}</codeblock>
+ <p>
+ <b>Requirements:</b>
+ </p>
+ <ul>
+ <li>Variable names must match the pattern: <codeph>[a-zA-Z][a-zA-Z0-9_-]*</codeph></li>
+ <li>Environment variables must be set before starting Impala</li>
+ <li>Undefined variables will cause Impala startup to fail</li>
+ </ul>
+ </section>
+ </conbody>
+ </concept>
+
+ <concept id="rest_catalog_deployment_modes">
+ <title>Deployment Modes</title>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Concepts"/>
+ </metadata>
+ </prolog>
+ <conbody>
+ <p>
+ Impala supports three deployment modes for REST Catalog, each suited to different use cases and architectural requirements.
+ </p>
+
+ <section id="standalone_rest_catalog">
+ <title>Mode 1: Standalone REST Catalog (No HMS/Catalogd)</title>
+ <p>
+ This mode runs Impala without HMS or catalogd, connecting directly to one or more REST Catalog servers.
+ It provides the simplest architecture for pure Iceberg deployments.
+ </p>
+ <p>
+ <b>Example Use Case:</b> New deployments that only need Iceberg tables and want minimal operational complexity.
+ </p>
+ <p>
+ <b>Configuration:</b>
+ </p>
+ <p>
+ <i>Impalad Flags:</i>
+ </p>
+ <codeblock>--use_local_catalog=true
+--catalogd_deployed=false
+--catalog_config_dir=/path/to/config/directory</codeblock>
+ <p>
+ <i>Configuration File</i> (<codeph>/path/to/config/directory/rest.properties</codeph>):
+ </p>
+ <codeblock>connector.name=iceberg
+iceberg.catalog.type=rest
+iceberg.rest-catalog.uri=http://rest-catalog-server:9084</codeblock>
+ </section>
+
+ <section id="multi_catalog_mode">
+ <title>Mode 2: Multi-Catalog (REST Catalog(s) + HMS)</title>
+ <p>
+ This hybrid mode connects to both REST Catalogs and HMS simultaneously, allowing access
+ to both Iceberg tables in the REST Catalog and traditional HMS tables.
+ </p>
+ <p>
+ <b>Example Use Case:</b> Migration scenarios or environments that need both new Iceberg tables and legacy HMS tables.
+ </p>
+ <p>
+ <b>Configuration:</b>
+ </p>
+ <p>
+ <i>Impalad Flags:</i>
+ </p>
+ <codeblock>--use_local_catalog=true
+--catalog_config_dir=/path/to/config/directory</codeblock>
+ <p>
+ <i>Catalogd Flags:</i>
+ </p>
+ <codeblock>--catalog_topic_mode=minimal</codeblock>
+ <p>
+ <i>Configuration File</i> (<codeph>/path/to/config/directory/rest.properties</codeph>):
+ </p>
+ <codeblock>connector.name=iceberg
+iceberg.catalog.type=rest
+iceberg.rest-catalog.uri=http://rest-catalog-server:9084</codeblock>
+ <p>
+ <b>Handling Ambiguous Table Names:</b>
+ </p>
+ <p>
+ If a table with the same name exists in multiple catalogs, queries will fail with an
+ ambiguity error. The error message will indicate which catalogs contain the table.
+ </p>
+ </section>
+ </conbody>
+ </concept>
+
+ <concept id="unsupported_features">
+ <title>Limitations and Unsupported Features</title>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Concepts"/>
+ </metadata>
+ </prolog>
+ <conbody>
+ <section id="limitations">
+ <p>
+ The following Iceberg REST Catalog features are not currently supported:
+ </p>
+ <ul>
+ <li>
+ <b>Vended Credentials:</b> The <codeph>vended-credentials-enabled</codeph> property
+ must be set to <codeph>false</codeph>. Impala does not support delegated credential
+ management from the REST Catalog server.
+ </li>
+ <li>
+ <b>User Sessions:</b> Only <codeph>NONE</codeph> session type is supported. Per-user
+ session isolation is not available.
+ </li>
+ <li>
+ <b>Nested Namespaces:</b> The <codeph>nested-namespace-enabled</codeph> property must
+ be set to <codeph>false</codeph>. Only flat namespace hierarchies are supported.
+ </li>
+ <li>
+ <b>Write Operations:</b> Write operations (INSERT, UPDATE, DELETE, MERGE) are not supported.
+ </li>
+ <li>
+ <b>DDL Operations:</b> DDL operations (CREATE, ALTER, DROP,...) are not supported.
+ </li>
+ <li>
+ <b>Views:</b> Iceberg views are not supported through the REST Catalog.
+ </li>
+ </ul>
+ </section>
+ </conbody>
+ </concept>
+</concept>