You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu> on 2021/07/30 03:57:25 UTC
Change in asterixdb[master]: WIP: option to exclude unknowns in secondary indexes
From Ali Alsuliman <al...@gmail.com>:
Ali Alsuliman has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/12583 )
Change subject: WIP: option to exclude unknowns in secondary indexes
......................................................................
WIP: option to exclude unknowns in secondary indexes
Change-Id: I5a5111f9c8c4f1ae1c311609a64d7ebd413ca18f
---
M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java
M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ConcurrentInsertTest.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-open-index/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-enforced/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateIndexStatement.java
M asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Index.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslator.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryBTreeOperationsHelper.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java
M asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslatorTest.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorNodePushable.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorNodePushable.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/metadata/IMetadataProvider.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/IndexInsertDeleteUpsertOperator.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/IndexInsertDeleteUpsertPOperator.java
M hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/SetAlgebricksPhysicalOperatorsRule.java
36 files changed, 263 insertions(+), 125 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/83/12583/1
diff --git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java
index a6ae0af..e3d5100 100644
--- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java
+++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java
@@ -313,6 +313,7 @@
// Set our key variables and expressions for non-array indexes. Our secondary keys for array indexes will
// always be an empty list.
List<LogicalVariable> secondaryKeyVars = new ArrayList<>();
+ List<LogicalVariable> beforeOpSecondaryKeyVars = new ArrayList<>();
List<Mutable<ILogicalExpression>> secondaryExpressions = new ArrayList<>();
List<Mutable<ILogicalExpression>> beforeOpSecondaryExpressions = new ArrayList<>();
ILogicalOperator replicateOutput;
@@ -324,23 +325,30 @@
secondaryKeyVars.add(skVar);
VariableReferenceExpression skVarRef = new VariableReferenceExpression(skVar);
skVarRef.setSourceLocation(sourceLoc);
- secondaryExpressions.add(new MutableObject<ILogicalExpression>(skVarRef));
+ secondaryExpressions.add(new MutableObject<>(skVarRef));
if (primaryIndexModificationOp.getOperation() == Kind.UPSERT) {
- VariableReferenceExpression varRef =
- new VariableReferenceExpression(fieldVarsForBeforeOperation.get(indexFieldId));
+ LogicalVariable beforeKeyVar = fieldVarsForBeforeOperation.get(indexFieldId);
+ beforeOpSecondaryKeyVars.add(beforeKeyVar);
+ VariableReferenceExpression varRef = new VariableReferenceExpression(beforeKeyVar);
varRef.setSourceLocation(sourceLoc);
- beforeOpSecondaryExpressions.add(new MutableObject<ILogicalExpression>(varRef));
+ beforeOpSecondaryExpressions.add(new MutableObject<>(varRef));
}
}
}
IndexInsertDeleteUpsertOperator indexUpdate;
+ boolean skipUnknowns = index.isSkipUnknowns();
if (index.getIndexType() != IndexType.RTREE) {
// Create an expression per key
- Mutable<ILogicalExpression> filterExpression = (primaryIndexModificationOp.getOperation() == Kind.UPSERT
- || index.getIndexType() == IndexType.BTREE) ? null
- : createFilterExpression(secondaryKeyVars, context.getOutputTypeEnvironment(currentTop),
- index.getIndexDetails().isOverridingKeyFieldTypes());
+ Mutable<ILogicalExpression> filterExpression =
+ createFilterExpression(secondaryKeyVars, context.getOutputTypeEnvironment(currentTop),
+ index.getIndexDetails().isOverridingKeyFieldTypes(), skipUnknowns);
+ Mutable<ILogicalExpression> beforeOpFilterExpression = null;
+ if (primaryIndexModificationOp.getOperation() == Kind.UPSERT) {
+ beforeOpFilterExpression = createFilterExpression(beforeOpSecondaryKeyVars,
+ context.getOutputTypeEnvironment(currentTop),
+ index.getIndexDetails().isOverridingKeyFieldTypes(), skipUnknowns);
+ }
DataSourceIndex dataSourceIndex = new DataSourceIndex(index, dataverseName, datasetName, mp);
// Introduce the TokenizeOperator only when doing bulk-load,
@@ -406,8 +414,8 @@
indexUpdate = new IndexInsertDeleteUpsertOperator(dataSourceIndex,
OperatorManipulationUtil
.cloneExpressions(primaryIndexModificationOp.getPrimaryKeyExpressions()),
- tokenizeKeyExprs, filterExpression, primaryIndexModificationOp.getOperation(),
- primaryIndexModificationOp.isBulkload(),
+ tokenizeKeyExprs, filterExpression, beforeOpFilterExpression,
+ primaryIndexModificationOp.getOperation(), primaryIndexModificationOp.isBulkload(),
primaryIndexModificationOp.getAdditionalNonFilteringExpressions() == null ? 0
: primaryIndexModificationOp.getAdditionalNonFilteringExpressions().size());
indexUpdate.setSourceLocation(sourceLoc);
@@ -419,8 +427,8 @@
indexUpdate = new IndexInsertDeleteUpsertOperator(dataSourceIndex,
OperatorManipulationUtil
.cloneExpressions(primaryIndexModificationOp.getPrimaryKeyExpressions()),
- secondaryExpressions, filterExpression, primaryIndexModificationOp.getOperation(),
- primaryIndexModificationOp.isBulkload(),
+ secondaryExpressions, filterExpression, beforeOpFilterExpression,
+ primaryIndexModificationOp.getOperation(), primaryIndexModificationOp.isBulkload(),
primaryIndexModificationOp.getAdditionalNonFilteringExpressions() == null ? 0
: primaryIndexModificationOp.getAdditionalNonFilteringExpressions().size());
indexUpdate.setSourceLocation(sourceLoc);
@@ -455,7 +463,7 @@
filterExpression = (primaryIndexModificationOp.getOperation() == Kind.UPSERT) ? null
: createFilterExpression(unnestSIDXBranch.lastFieldVars,
context.getOutputTypeEnvironment(unnestSIDXBranch.currentTop),
- index.getIndexDetails().isOverridingKeyFieldTypes());
+ index.getIndexDetails().isOverridingKeyFieldTypes(), skipUnknowns);
if (filterExpression != null) {
unnestSIDXBranch.applyFilteringExpression(filterExpression);
}
@@ -500,7 +508,7 @@
// Update the filter expression to include these new keys.
filterExpression = createFilterExpression(unnestSIDXBranch.lastFieldVars,
context.getOutputTypeEnvironment(unnestSIDXBranch.currentTop),
- index.getIndexDetails().isOverridingKeyFieldTypes());
+ index.getIndexDetails().isOverridingKeyFieldTypes(), skipUnknowns);
indexUpdate.setFilterExpression(filterExpression);
if (replicateOp != null) {
@@ -560,8 +568,11 @@
assignCoordinates.getInputs().add(new MutableObject<ILogicalOperator>(currentTop));
context.computeAndSetTypeEnvironmentForOperator(assignCoordinates);
replicateOutput = assignCoordinates;
- Mutable<ILogicalExpression> filterExpression = null;
+ boolean forceFilter = keyPairType.second;
+ Mutable<ILogicalExpression> filterExpression = createFilterExpression(keyVarList,
+ context.getOutputTypeEnvironment(assignCoordinates), forceFilter, skipUnknowns);
AssignOperator originalAssignCoordinates = null;
+ Mutable<ILogicalExpression> beforeOpFilterExpression = null;
// We do something similar for beforeOp key if the operation is an upsert
if (primaryIndexModificationOp.getOperation() == Kind.UPSERT) {
List<LogicalVariable> originalKeyVarList = new ArrayList<>();
@@ -592,19 +603,15 @@
originalAssignCoordinates.setSourceLocation(sourceLoc);
originalAssignCoordinates.getInputs().add(new MutableObject<ILogicalOperator>(assignCoordinates));
context.computeAndSetTypeEnvironmentForOperator(originalAssignCoordinates);
- } else {
- // We must enforce the filter if the originating spatial type is
- // nullable.
- boolean forceFilter = keyPairType.second;
- filterExpression = createFilterExpression(keyVarList,
- context.getOutputTypeEnvironment(assignCoordinates), forceFilter);
+ beforeOpFilterExpression = createFilterExpression(originalKeyVarList,
+ context.getOutputTypeEnvironment(originalAssignCoordinates), forceFilter, skipUnknowns);
}
DataSourceIndex dataSourceIndex = new DataSourceIndex(index, dataverseName, datasetName, mp);
indexUpdate = new IndexInsertDeleteUpsertOperator(dataSourceIndex,
OperatorManipulationUtil
.cloneExpressions(primaryIndexModificationOp.getPrimaryKeyExpressions()),
- secondaryExpressions, filterExpression, primaryIndexModificationOp.getOperation(),
- primaryIndexModificationOp.isBulkload(),
+ secondaryExpressions, filterExpression, beforeOpFilterExpression,
+ primaryIndexModificationOp.getOperation(), primaryIndexModificationOp.isBulkload(),
primaryIndexModificationOp.getAdditionalNonFilteringExpressions() == null ? 0
: primaryIndexModificationOp.getAdditionalNonFilteringExpressions().size());
indexUpdate.setSourceLocation(sourceLoc);
@@ -943,7 +950,10 @@
}
private Mutable<ILogicalExpression> createFilterExpression(List<LogicalVariable> secondaryKeyVars,
- IVariableTypeEnvironment typeEnv, boolean forceFilter) throws AlgebricksException {
+ IVariableTypeEnvironment typeEnv, boolean forceFilter, boolean skipUnknowns) throws AlgebricksException {
+ if (!skipUnknowns) {
+ return null;
+ }
List<Mutable<ILogicalExpression>> filterExpressions = new ArrayList<>();
// Add 'is not null' to all nullable secondary index keys as a filtering
// condition.
@@ -971,10 +981,10 @@
Mutable<ILogicalExpression> filterExpression;
if (filterExpressions.size() > 1) {
// Create a conjunctive condition.
- ScalarFunctionCallExpression andExpr = new ScalarFunctionCallExpression(
- FunctionUtil.getFunctionInfo(BuiltinFunctions.AND), filterExpressions);
- andExpr.setSourceLocation(sourceLoc);
- filterExpression = new MutableObject<>(andExpr);
+ ScalarFunctionCallExpression orExpr = new ScalarFunctionCallExpression(
+ FunctionUtil.getFunctionInfo(BuiltinFunctions.OR), filterExpressions);
+ orExpr.setSourceLocation(sourceLoc);
+ filterExpression = new MutableObject<>(orExpr);
} else {
filterExpression = filterExpressions.get(0);
}
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
index 267bd76..9d5c5f5 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
@@ -1291,7 +1291,7 @@
}
Index newIndex = new Index(dataverseName, datasetName, indexName, indexType, indexDetails,
- stmtCreateIndex.isEnforced(), false, MetadataUtil.PENDING_ADD_OP);
+ stmtCreateIndex.isEnforced(), false, MetadataUtil.PENDING_ADD_OP, stmtCreateIndex.isSkipUnknowns());
bActiveTxn = false; // doCreateIndexImpl() takes over the current transaction
doCreateIndexImpl(hcc, metadataProvider, ds, newIndex, jobFlags, sourceLoc);
@@ -1489,7 +1489,7 @@
IndexingConstants.getFilesIndexName(index.getDatasetName()), IndexType.BTREE,
new Index.ValueIndexDetails(ExternalIndexingOperations.FILE_INDEX_FIELD_NAMES, null,
ExternalIndexingOperations.FILE_INDEX_FIELD_TYPES, false),
- false, false, MetadataUtil.PENDING_ADD_OP);
+ false, false, MetadataUtil.PENDING_ADD_OP, index.isSkipUnknowns());
MetadataManager.INSTANCE.addIndex(metadataProvider.getMetadataTxnContext(), filesIndex);
// Add files to the external files index
for (ExternalFile file : externalFilesSnapshot) {
@@ -2137,7 +2137,8 @@
MetadataManager.INSTANCE.dropIndex(mdTxnCtx, dataverseName, datasetName, indexName);
MetadataManager.INSTANCE.addIndex(mdTxnCtx,
new Index(dataverseName, datasetName, indexName, index.getIndexType(), index.getIndexDetails(),
- index.isEnforced(), index.isPrimaryIndex(), MetadataUtil.PENDING_DROP_OP));
+ index.isEnforced(), index.isPrimaryIndex(), MetadataUtil.PENDING_DROP_OP,
+ index.isSkipUnknowns()));
// #. commit the existing transaction before calling runJob.
MetadataManager.INSTANCE.commitTransaction(mdTxnCtx);
@@ -2189,7 +2190,7 @@
new Index(dataverseName, datasetName, externalIndex.getIndexName(),
externalIndex.getIndexType(), externalIndex.getIndexDetails(),
externalIndex.isEnforced(), externalIndex.isPrimaryIndex(),
- MetadataUtil.PENDING_DROP_OP));
+ MetadataUtil.PENDING_DROP_OP, externalIndex.isSkipUnknowns()));
}
}
}
@@ -2198,7 +2199,8 @@
MetadataManager.INSTANCE.dropIndex(mdTxnCtx, dataverseName, datasetName, indexName);
MetadataManager.INSTANCE.addIndex(mdTxnCtx,
new Index(dataverseName, datasetName, indexName, index.getIndexType(), index.getIndexDetails(),
- index.isEnforced(), index.isPrimaryIndex(), MetadataUtil.PENDING_DROP_OP));
+ index.isEnforced(), index.isPrimaryIndex(), MetadataUtil.PENDING_DROP_OP,
+ index.isSkipUnknowns()));
// #. commit the existing transaction before calling runJob.
MetadataManager.INSTANCE.commitTransaction(mdTxnCtx);
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
index c668fb6..9e8edd3 100644
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
@@ -721,7 +721,7 @@
index = new Index(dataset.getDataverseName(), dataset.getDatasetName(), dataset.getDatasetName(),
IndexType.BTREE,
new Index.ValueIndexDetails(keyFieldNames, primaryKeyIndicators, keyFieldTypes, false), false, true,
- MetadataUtil.PENDING_NO_OP);
+ MetadataUtil.PENDING_NO_OP, false);
List<String> nodes = Collections.singletonList(ExecutionTestUtil.integrationUtil.ncs[0].getId());
CcApplicationContext appCtx =
(CcApplicationContext) ExecutionTestUtil.integrationUtil.cc.getApplicationContext();
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java
index 1913233..caab0b8 100644
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java
@@ -171,7 +171,7 @@
partitioningKeys, null, null, null, false, null, null),
null, DatasetType.INTERNAL, DATASET_ID, 0);
secondaryIndex = new Index(dvName, DATASET_NAME, INDEX_NAME, INDEX_TYPE, INDEX_FIELD_NAMES,
- INDEX_FIELD_INDICATORS, INDEX_FIELD_TYPES, false, false, false, 0);
+ INDEX_FIELD_INDICATORS, INDEX_FIELD_TYPES, false, false, false, 0, false);
taskCtx = null;
primaryIndexDataflowHelper = null;
secondaryIndexDataflowHelper = null;
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ConcurrentInsertTest.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ConcurrentInsertTest.java
index 76dd035..841536e 100644
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ConcurrentInsertTest.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ConcurrentInsertTest.java
@@ -128,7 +128,8 @@
Index secondaryIndexEntity = new Index(StorageTestUtils.DATASET.getDataverseName(),
StorageTestUtils.DATASET.getDatasetName(), "TestIndex", IndexType.BTREE,
Arrays.asList(Arrays.asList(StorageTestUtils.RECORD_TYPE.getFieldNames()[1])),
- Arrays.asList(Index.RECORD_INDICATOR), Arrays.asList(BuiltinType.AINT64), false, false, false, 0);
+ Arrays.asList(Index.RECORD_INDICATOR), Arrays.asList(BuiltinType.AINT64), false, false, false, 0,
+ false);
SecondaryIndexInfo secondaryIndexInfo =
nc.createSecondaryIndex(primaryIndexInfo, secondaryIndexEntity, StorageTestUtils.STORAGE_MANAGER, 0);
@@ -142,7 +143,7 @@
Index primaryKeyIndexEntity = new Index(StorageTestUtils.DATASET.getDataverseName(),
StorageTestUtils.DATASET.getDatasetName(), "PrimaryKeyIndex", IndexType.BTREE, Arrays.asList(),
- Arrays.asList(), Arrays.asList(), false, false, false, 0);
+ Arrays.asList(), Arrays.asList(), false, false, false, 0, false);
SecondaryIndexInfo primaryKeyIndexInfo =
nc.createSecondaryIndex(primaryIndexInfo, primaryKeyIndexEntity, StorageTestUtils.STORAGE_MANAGER, 0);
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java
index 2aad416..f408758 100644
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java
@@ -187,7 +187,7 @@
dataset = StorageTestUtils.DATASET;
secondaryIndexEntity = new Index(dataset.getDataverseName(), dataset.getDatasetName(), SECONDARY_INDEX_NAME,
SECONDARY_INDEX_TYPE, SECONDARY_INDEX_FIELD_NAMES, SECONDARY_INDEX_FIELD_INDICATORS,
- SECONDARY_INDEX_FIELD_TYPES, false, false, false, 0);
+ SECONDARY_INDEX_FIELD_TYPES, false, false, false, 0, false);
primaryIndexInfos = new PrimaryIndexInfo[NUM_PARTITIONS];
secondaryIndexInfo = new SecondaryIndexInfo[NUM_PARTITIONS];
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java
index ef3a7cb..42ddd95 100644
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java
@@ -161,7 +161,7 @@
partitioningKeys, null, null, null, false, null, null),
null, DatasetType.INTERNAL, DATASET_ID, 0);
secondaryIndex = new Index(dvName, DATASET_NAME, INDEX_NAME, INDEX_TYPE, INDEX_FIELD_NAMES,
- INDEX_FIELD_INDICATORS, INDEX_FIELD_TYPES, false, false, false, 0);
+ INDEX_FIELD_INDICATORS, INDEX_FIELD_TYPES, false, false, false, 0, false);
taskCtxs = new IHyracksTaskContext[NUM_PARTITIONS];
primaryIndexDataflowHelpers = new IIndexDataflowHelper[NUM_PARTITIONS];
secondaryIndexDataflowHelpers = new IIndexDataflowHelper[NUM_PARTITIONS];
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
index 62dfbe2..606c2e7 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
@@ -26,5 +26,5 @@
use test;
-create index ngram_index on DBLP (title) type ngram (3);
+create index ngram_index on DBLP (title) type ngram (3) exclude unknown key;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
index 662b250..5e6021c 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
@@ -20,5 +20,5 @@
use test;
-create index rtree_index_point on MyData (point) type rtree;
+create index rtree_index_point on MyData (point) type rtree exclude unknown key;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
index c57941c..6876008 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
@@ -26,5 +26,5 @@
use test;
-create index ngram_index on DBLP (nested.title) type ngram (3);
+create index ngram_index on DBLP (nested.title) type ngram (3) exclude unknown key;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
index 5683fe0..4eae583 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
@@ -20,5 +20,5 @@
use test;
-create index rtree_index_point on MyData (nested.point) type rtree;
+create index rtree_index_point on MyData (nested.point) type rtree exclude unknown key;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-open-index/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-open-index/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
index 01e97f6..5603f55 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-open-index/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-open-index/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
@@ -59,5 +59,5 @@
create dataset TweetMessagesTmp(TweetMessageNestedType) primary key tweetid;
-create index msgNgramIx on TweetMessages (nested.`message-text`:string?) type ngram (3) enforced;
+create index msgNgramIx on TweetMessages (nested.`message-text`:string?) type ngram (3) enforced exclude unknown key;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-enforced/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-enforced/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
index 4ff0a38..ac7843d 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-enforced/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-enforced/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
@@ -54,5 +54,5 @@
create dataset TweetMessagesTmp(TweetMessageType) primary key tweetid;
-create index msgNgramIx on TweetMessages (`message-text`:string?) type ngram (3) enforced;
+create index msgNgramIx on TweetMessages (`message-text`:string?) type ngram (3) enforced exclude unknown key;
diff --git a/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateIndexStatement.java b/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateIndexStatement.java
index 468006c..d7d66a6 100644
--- a/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateIndexStatement.java
+++ b/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateIndexStatement.java
@@ -48,10 +48,11 @@
private final int gramLength;
// Specific to FullText indexes.
private final String fullTextConfigName;
+ private final boolean skipUnknowns;
public CreateIndexStatement(DataverseName dataverseName, Identifier datasetName, Identifier indexName,
IndexType indexType, List<IndexedElement> indexedElements, boolean enforced, int gramLength,
- String fullTextConfigName, boolean ifNotExists) {
+ String fullTextConfigName, boolean ifNotExists, boolean skipUnknowns) {
this.dataverseName = dataverseName;
this.datasetName = Objects.requireNonNull(datasetName);
this.indexName = Objects.requireNonNull(indexName);
@@ -61,6 +62,7 @@
this.gramLength = gramLength;
this.ifNotExists = ifNotExists;
this.fullTextConfigName = fullTextConfigName;
+ this.skipUnknowns = skipUnknowns;
}
public String getFullTextConfigName() {
@@ -91,6 +93,10 @@
return enforced;
}
+ public boolean isSkipUnknowns() {
+ return skipUnknowns;
+ }
+
public int getGramLength() {
return gramLength;
}
diff --git a/asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj b/asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
index de9ea7a..fb24366 100644
--- a/asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
+++ b/asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
@@ -227,6 +227,7 @@
private static final String CURRENT = "CURRENT";
private static final String DEFAULT = "DEFAULT";
private static final String EXCLUDE = "EXCLUDE";
+ private static final String INCLUDE = "INCLUDE";
private static final String FIRST = "FIRST";
private static final String FOLLOWING = "FOLLOWING";
private static final String GROUPING = "GROUPING";
@@ -1131,6 +1132,7 @@
boolean hasUnnest = false;
String fullTextConfigName = null;
Token startElementToken = null;
+ boolean excludeUnknown = false;
}
{
(
@@ -1149,6 +1151,17 @@
)*
<RIGHTPAREN>
( <TYPE> indexParams = IndexType() )? ( <ENFORCED> { enforced = true; } )?
+ ( <IDENTIFIER>
+ {
+ if (isToken(EXCLUDE)) {
+ excludeUnknown = true;
+ } else if (isToken(INCLUDE)) {
+ excludeUnknown = false;
+ } else {
+ throw createUnexpectedTokenError();
+ }
+ } <UNKNOWN> <KEY>
+ )?
)
{
IndexType indexType;
@@ -1163,7 +1176,8 @@
fullTextConfigName = null;
}
CreateIndexStatement stmt = new CreateIndexStatement(nameComponents.first, nameComponents.second,
- new Identifier(indexName), indexType, indexedElementList, enforced, gramLength, fullTextConfigName, ifNotExists);
+ new Identifier(indexName), indexType, indexedElementList, enforced, gramLength, fullTextConfigName, ifNotExists,
+ excludeUnknown);
return addSourceLocation(stmt, startStmtToken);
}
}
@@ -1292,7 +1306,7 @@
indexName = "primary_idx_" + nameComponents.second;
}
CreateIndexStatement stmt = new CreateIndexStatement(nameComponents.first, nameComponents.second,
- new Identifier(indexName), IndexType.BTREE, Collections.emptyList(), false, -1, null, ifNotExists);
+ new Identifier(indexName), IndexType.BTREE, Collections.emptyList(), false, -1, null, ifNotExists, false);
return addSourceLocation(stmt, startStmtToken);
}
}
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
index a5dda0d..26423ef 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
@@ -366,7 +366,7 @@
InternalDatasetDetails id = (InternalDatasetDetails) dataset.getDatasetDetails();
Index primaryIndex = new Index(dataset.getDataverseName(), dataset.getDatasetName(),
dataset.getDatasetName(), IndexType.BTREE, id.getPrimaryKey(), id.getKeySourceIndicator(),
- id.getPrimaryKeyType(), false, false, true, dataset.getPendingOp());
+ id.getPrimaryKeyType(), false, false, true, dataset.getPendingOp(), false);
addIndex(txnId, primaryIndex);
}
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java
index 48fb450..1287670 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java
@@ -144,8 +144,8 @@
}
public void dropIndex(DataverseName dataverseName, String datasetName, String indexName) {
- Index index =
- new Index(dataverseName, datasetName, indexName, null, null, false, false, MetadataUtil.PENDING_NO_OP);
+ Index index = new Index(dataverseName, datasetName, indexName, null, null, false, false,
+ MetadataUtil.PENDING_NO_OP, false);
droppedCache.addIndexIfNotExists(index);
logAndApply(new MetadataLogicalOperation(index, false));
}
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
index 6f91572..3ca3750 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
@@ -785,7 +785,7 @@
boolean bulkload, List<List<AlgebricksPipeline>> secondaryKeysPipelines, IOperatorSchema pipelineTopSchema)
throws AlgebricksException {
return getIndexInsertOrDeleteOrUpsertRuntime(IndexOperation.INSERT, dataSourceIndex, propagatedSchema,
- inputSchemas, typeEnv, primaryKeys, secondaryKeys, additionalNonKeyFields, filterExpr, recordDesc,
+ inputSchemas, typeEnv, primaryKeys, secondaryKeys, additionalNonKeyFields, filterExpr, null, recordDesc,
context, spec, bulkload, null, null, null, secondaryKeysPipelines, pipelineTopSchema);
}
@@ -798,7 +798,7 @@
List<List<AlgebricksPipeline>> secondaryKeysPipelines, IOperatorSchema pipelineTopSchema)
throws AlgebricksException {
return getIndexInsertOrDeleteOrUpsertRuntime(IndexOperation.DELETE, dataSourceIndex, propagatedSchema,
- inputSchemas, typeEnv, primaryKeys, secondaryKeys, additionalNonKeyFields, filterExpr, recordDesc,
+ inputSchemas, typeEnv, primaryKeys, secondaryKeys, additionalNonKeyFields, filterExpr, null, recordDesc,
context, spec, false, null, null, null, secondaryKeysPipelines, pipelineTopSchema);
}
@@ -807,12 +807,13 @@
IDataSourceIndex<String, DataSourceId> dataSourceIndex, IOperatorSchema propagatedSchema,
IOperatorSchema[] inputSchemas, IVariableTypeEnvironment typeEnv, List<LogicalVariable> primaryKeys,
List<LogicalVariable> secondaryKeys, List<LogicalVariable> additionalFilteringKeys,
- ILogicalExpression filterExpr, LogicalVariable operationVar, List<LogicalVariable> prevSecondaryKeys,
- LogicalVariable prevAdditionalFilteringKey, RecordDescriptor recordDesc, JobGenContext context,
- JobSpecification spec, List<List<AlgebricksPipeline>> secondaryKeysPipelines) throws AlgebricksException {
+ ILogicalExpression filterExpr, ILogicalExpression prevFilterExpr, LogicalVariable operationVar,
+ List<LogicalVariable> prevSecondaryKeys, LogicalVariable prevAdditionalFilteringKey,
+ RecordDescriptor recordDesc, JobGenContext context, JobSpecification spec,
+ List<List<AlgebricksPipeline>> secondaryKeysPipelines) throws AlgebricksException {
return getIndexInsertOrDeleteOrUpsertRuntime(IndexOperation.UPSERT, dataSourceIndex, propagatedSchema,
- inputSchemas, typeEnv, primaryKeys, secondaryKeys, additionalFilteringKeys, filterExpr, recordDesc,
- context, spec, false, operationVar, prevSecondaryKeys, prevAdditionalFilteringKey,
+ inputSchemas, typeEnv, primaryKeys, secondaryKeys, additionalFilteringKeys, filterExpr, prevFilterExpr,
+ recordDesc, context, spec, false, operationVar, prevSecondaryKeys, prevAdditionalFilteringKey,
secondaryKeysPipelines, null);
}
@@ -1216,10 +1217,11 @@
IOperatorSchema propagatedSchema, IOperatorSchema[] inputSchemas, IVariableTypeEnvironment typeEnv,
List<LogicalVariable> primaryKeys, List<LogicalVariable> secondaryKeys,
List<LogicalVariable> additionalNonKeyFields, ILogicalExpression filterExpr,
- RecordDescriptor inputRecordDesc, JobGenContext context, JobSpecification spec, boolean bulkload,
- LogicalVariable operationVar, List<LogicalVariable> prevSecondaryKeys,
- LogicalVariable prevAdditionalFilteringKey, List<List<AlgebricksPipeline>> secondaryKeysPipelines,
- IOperatorSchema pipelineTopSchema) throws AlgebricksException {
+ ILogicalExpression prevFilterExpr, RecordDescriptor inputRecordDesc, JobGenContext context,
+ JobSpecification spec, boolean bulkload, LogicalVariable operationVar,
+ List<LogicalVariable> prevSecondaryKeys, LogicalVariable prevAdditionalFilteringKey,
+ List<List<AlgebricksPipeline>> secondaryKeysPipelines, IOperatorSchema pipelineTopSchema)
+ throws AlgebricksException {
String indexName = dataSourceIndex.getId();
DataverseName dataverseName = dataSourceIndex.getDataSource().getId().getDataverseName();
String datasetName = dataSourceIndex.getDataSource().getId().getDatasourceName();
@@ -1236,28 +1238,31 @@
// If we have a pipeline, then we need to pass the schema of the pipeline to the filter factory.
AsterixTupleFilterFactory filterFactory;
+ AsterixTupleFilterFactory prevFilterFactory;
if (pipelineTopSchema != null) {
IOperatorSchema[] schemasForFilterFactory = new IOperatorSchema[inputSchemas.length + 1];
System.arraycopy(inputSchemas, 0, schemasForFilterFactory, 0, inputSchemas.length);
schemasForFilterFactory[inputSchemas.length] = pipelineTopSchema;
filterFactory = createTupleFilterFactory(schemasForFilterFactory, typeEnv, filterExpr, context);
-
+ prevFilterFactory = createTupleFilterFactory(schemasForFilterFactory, typeEnv, prevFilterExpr, context);
} else {
filterFactory = createTupleFilterFactory(inputSchemas, typeEnv, filterExpr, context);
+ prevFilterFactory = createTupleFilterFactory(inputSchemas, typeEnv, prevFilterExpr, context);
}
switch (secondaryIndex.getIndexType()) {
case BTREE:
return getBTreeRuntime(dataverseName, datasetName, indexName, propagatedSchema, primaryKeys,
- secondaryKeys, additionalNonKeyFields, filterFactory, inputRecordDesc, context, spec, indexOp,
- bulkload, operationVar, prevSecondaryKeys, prevAdditionalFilteringKeys);
+ secondaryKeys, additionalNonKeyFields, filterFactory, prevFilterFactory, inputRecordDesc,
+ context, spec, indexOp, bulkload, operationVar, prevSecondaryKeys, prevAdditionalFilteringKeys);
case ARRAY:
if (bulkload) {
// In the case of bulk-load, we do not handle any nested plans. We perform the exact same behavior
// as a normal B-Tree bulk load.
return getBTreeRuntime(dataverseName, datasetName, indexName, propagatedSchema, primaryKeys,
- secondaryKeys, additionalNonKeyFields, filterFactory, inputRecordDesc, context, spec,
- indexOp, bulkload, operationVar, prevSecondaryKeys, prevAdditionalFilteringKeys);
+ secondaryKeys, additionalNonKeyFields, filterFactory, prevFilterFactory, inputRecordDesc,
+ context, spec, indexOp, bulkload, operationVar, prevSecondaryKeys,
+ prevAdditionalFilteringKeys);
} else {
return getArrayIndexRuntime(dataverseName, datasetName, indexName, propagatedSchema, primaryKeys,
additionalNonKeyFields, inputRecordDesc, spec, indexOp, operationVar,
@@ -1284,10 +1289,10 @@
private Pair<IOperatorDescriptor, AlgebricksPartitionConstraint> getBTreeRuntime(DataverseName dataverseName,
String datasetName, String indexName, IOperatorSchema propagatedSchema, List<LogicalVariable> primaryKeys,
List<LogicalVariable> secondaryKeys, List<LogicalVariable> additionalNonKeyFields,
- AsterixTupleFilterFactory filterFactory, RecordDescriptor inputRecordDesc, JobGenContext context,
- JobSpecification spec, IndexOperation indexOp, boolean bulkload, LogicalVariable operationVar,
- List<LogicalVariable> prevSecondaryKeys, List<LogicalVariable> prevAdditionalFilteringKeys)
- throws AlgebricksException {
+ AsterixTupleFilterFactory filterFactory, AsterixTupleFilterFactory prevFilterFactory,
+ RecordDescriptor inputRecordDesc, JobGenContext context, JobSpecification spec, IndexOperation indexOp,
+ boolean bulkload, LogicalVariable operationVar, List<LogicalVariable> prevSecondaryKeys,
+ List<LogicalVariable> prevAdditionalFilteringKeys) throws AlgebricksException {
Dataset dataset = MetadataManagerUtil.findExistingDataset(mdTxnCtx, dataverseName, datasetName);
int numKeys = primaryKeys.size() + secondaryKeys.size();
int numFilterFields = DatasetUtil.getFilterField(dataset) == null ? 0 : 1;
@@ -1356,8 +1361,8 @@
} else if (indexOp == IndexOperation.UPSERT) {
int operationFieldIndex = propagatedSchema.findVariable(operationVar);
op = new LSMSecondaryUpsertOperatorDescriptor(spec, inputRecordDesc, fieldPermutation, idfh,
- filterFactory, modificationCallbackFactory, operationFieldIndex, BinaryIntegerInspector.FACTORY,
- prevFieldPermutation);
+ filterFactory, prevFilterFactory, modificationCallbackFactory, operationFieldIndex,
+ BinaryIntegerInspector.FACTORY, prevFieldPermutation);
} else {
op = new LSMTreeInsertDeleteOperatorDescriptor(spec, inputRecordDesc, fieldPermutation, indexOp, idfh,
filterFactory, false, modificationCallbackFactory);
@@ -1511,7 +1516,7 @@
} else if (indexOp == IndexOperation.UPSERT) {
int operationFieldIndex = propagatedSchema.findVariable(operationVar);
op = new LSMSecondaryUpsertOperatorDescriptor(spec, recordDesc, fieldPermutation,
- indexDataflowHelperFactory, filterFactory, modificationCallbackFactory, operationFieldIndex,
+ indexDataflowHelperFactory, filterFactory, null, modificationCallbackFactory, operationFieldIndex,
BinaryIntegerInspector.FACTORY, prevFieldPermutation);
} else {
op = new LSMTreeInsertDeleteOperatorDescriptor(spec, recordDesc, fieldPermutation, indexOp,
@@ -1624,7 +1629,7 @@
} else if (indexOp == IndexOperation.UPSERT) {
int upsertOperationFieldIndex = propagatedSchema.findVariable(operationVar);
op = new LSMSecondaryUpsertOperatorDescriptor(spec, recordDesc, fieldPermutation, indexDataFlowFactory,
- filterFactory, modificationCallbackFactory, upsertOperationFieldIndex,
+ filterFactory, null, modificationCallbackFactory, upsertOperationFieldIndex,
BinaryIntegerInspector.FACTORY, prevFieldPermutation);
} else {
op = new LSMTreeInsertDeleteOperatorDescriptor(spec, recordDesc, fieldPermutation, indexOp,
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Index.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Index.java
index cf09779..edf6c09 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Index.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Index.java
@@ -57,11 +57,13 @@
private final IIndexDetails indexDetails;
private final boolean isPrimaryIndex;
private final boolean isEnforced;
+ private final boolean isSkipUnknowns;
// Type of pending operations with respect to atomic DDL operation
private int pendingOp;
public Index(DataverseName dataverseName, String datasetName, String indexName, IndexType indexType,
- IIndexDetails indexDetails, boolean isEnforced, boolean isPrimaryIndex, int pendingOp) {
+ IIndexDetails indexDetails, boolean isEnforced, boolean isPrimaryIndex, int pendingOp,
+ boolean isSkipUnknowns) {
boolean categoryOk = (indexType == null && indexDetails == null) || (IndexCategory
.of(Objects.requireNonNull(indexType)) == ((AbstractIndexDetails) Objects.requireNonNull(indexDetails))
.getIndexCategory());
@@ -76,16 +78,18 @@
this.isPrimaryIndex = isPrimaryIndex;
this.isEnforced = isEnforced;
this.pendingOp = pendingOp;
+ this.isSkipUnknowns = isSkipUnknowns;
}
@Deprecated
public Index(DataverseName dataverseName, String datasetName, String indexName, IndexType indexType,
List<List<String>> keyFieldNames, List<Integer> keyFieldSourceIndicators, List<IAType> keyFieldTypes,
- boolean overrideKeyFieldTypes, boolean isEnforced, boolean isPrimaryIndex, int pendingOp) {
+ boolean overrideKeyFieldTypes, boolean isEnforced, boolean isPrimaryIndex, int pendingOp,
+ boolean isSkipUnknowns) {
this(dataverseName,
datasetName, indexName, indexType, createSimpleIndexDetails(indexType, keyFieldNames,
keyFieldSourceIndicators, keyFieldTypes, overrideKeyFieldTypes),
- isEnforced, isPrimaryIndex, pendingOp);
+ isEnforced, isPrimaryIndex, pendingOp, isSkipUnknowns);
}
public DataverseName getDataverseName() {
@@ -133,6 +137,10 @@
return indexType == IndexType.BTREE && ((ValueIndexDetails) indexDetails).keyFieldNames.isEmpty();
}
+ public boolean isSkipUnknowns() {
+ return isSkipUnknowns;
+ }
+
public static Pair<IAType, Boolean> getNonNullableType(IAType keyType) {
boolean nullable = false;
IAType actualKeyType = keyType;
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslator.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslator.java
index 32764b2..3d8c456 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslator.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslator.java
@@ -81,6 +81,7 @@
public static final String FULL_TEXT_CONFIG_FIELD_NAME = "FullTextConfig";
public static final String INDEX_SEARCHKEY_TYPE_FIELD_NAME = "SearchKeyType";
public static final String INDEX_ISENFORCED_FIELD_NAME = "IsEnforced";
+ public static final String INDEX_SKIP_UNKNOWNS_FIELD_NAME = "SkipUnknowns";
public static final String INDEX_SEARCHKEY_SOURCE_INDICATOR_FIELD_NAME = "SearchKeySourceIndicator";
public static final String INDEX_SEARCHKEY_ELEMENTS_FIELD_NAME = "SearchKeyElements";
public static final String COMPLEXSEARCHKEY_UNNEST_FIELD_NAME = "UnnestList";
@@ -419,8 +420,13 @@
int pendingOp = ((AInt32) indexRecord.getValueByPos(MetadataRecordTypes.INDEX_ARECORD_PENDINGOP_FIELD_INDEX))
.getIntegerValue();
+ int skipUnknownsPos = indexRecord.getType().getFieldIndex(INDEX_SKIP_UNKNOWNS_FIELD_NAME);
+ Boolean skipUnknowns = false;
+ if (skipUnknownsPos > 0) {
+ skipUnknowns = ((ABoolean) indexRecord.getValueByPos(skipUnknownsPos)).getBoolean();
+ }
return new Index(dataverseName, datasetName, indexName, indexType, indexDetails, isEnforcingKeys,
- isPrimaryIndex, pendingOp);
+ isPrimaryIndex, pendingOp, skipUnknowns);
}
@Override
@@ -549,6 +555,7 @@
writeSearchKeyType(index);
writeEnforced(index);
writeSearchKeySourceIndicator(index);
+ writeSkipUnknowns(index);
}
private void writeComplexSearchKeys(Index.ArrayIndexDetails indexDetails) throws HyracksDataException {
@@ -748,4 +755,15 @@
recordBuilder.addField(nameValue, fieldValue);
}
}
+
+ private void writeSkipUnknowns(Index index) throws HyracksDataException {
+ if (index.isSkipUnknowns()) {
+ fieldValue.reset();
+ nameValue.reset();
+ aString.setValue(INDEX_SKIP_UNKNOWNS_FIELD_NAME);
+ stringSerde.serialize(aString, nameValue.getDataOutput());
+ booleanSerde.serialize(ABoolean.TRUE, fieldValue.getDataOutput());
+ recordBuilder.addField(nameValue, fieldValue);
+ }
+ }
}
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java
index f5b2697..d15ac31 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java
@@ -60,7 +60,7 @@
InternalDatasetDetails id = (InternalDatasetDetails) dataset.getDatasetDetails();
return new Index(dataset.getDataverseName(), dataset.getDatasetName(), dataset.getDatasetName(),
DatasetConfig.IndexType.BTREE, id.getPartitioningKey(), id.getKeySourceIndicator(),
- id.getPrimaryKeyType(), false, false, true, dataset.getPendingOp());
+ id.getPrimaryKeyType(), false, false, true, dataset.getPendingOp(), false);
}
public static int[] getBtreeFieldsIfFiltered(Dataset dataset, Index index) throws AlgebricksException {
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryBTreeOperationsHelper.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryBTreeOperationsHelper.java
index 9645e7a..e265af1 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryBTreeOperationsHelper.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryBTreeOperationsHelper.java
@@ -70,6 +70,7 @@
int[] fieldPermutation = createFieldPermutationForBulkLoadOp(indexDetails.getKeyFieldNames().size());
IIndexDataflowHelperFactory dataflowHelperFactory = new IndexDataflowHelperFactory(
metadataProvider.getStorageComponentProvider().getStorageManager(), secondaryFileSplitProvider);
+ boolean filterUnknowns = index.isSkipUnknowns() && (anySecondaryKeyIsNullable || isOverridingKeyFieldTypes);
if (dataset.getDatasetType() == DatasetType.EXTERNAL) {
/*
* In case of external data,
@@ -89,6 +90,11 @@
AlgebricksMetaOperatorDescriptor asterixAssignOp =
createExternalAssignOp(spec, indexDetails.getKeyFieldNames().size(), secondaryRecDesc);
+ // If any of the secondary fields are nullable, then add a select op that filters nulls.
+ AlgebricksMetaOperatorDescriptor selectOp = null;
+ if (filterUnknowns) {
+ selectOp = createFilterNullsSelectOp(spec, indexDetails.getKeyFieldNames().size(), secondaryRecDesc);
+ }
// Sort by secondary keys.
ExternalSortOperatorDescriptor sortOp = createSortOp(spec, secondaryComparatorFactories, secondaryRecDesc);
// Create secondary BTree bulk load op.
@@ -111,7 +117,12 @@
spec.connect(new OneToOneConnectorDescriptor(spec), secondaryBulkLoadOp, 0, metaOp, 0);
root = metaOp;
spec.connect(new OneToOneConnectorDescriptor(spec), sourceOp, 0, asterixAssignOp, 0);
- spec.connect(new OneToOneConnectorDescriptor(spec), asterixAssignOp, 0, sortOp, 0);
+ if (filterUnknowns) {
+ spec.connect(new OneToOneConnectorDescriptor(spec), asterixAssignOp, 0, selectOp, 0);
+ spec.connect(new OneToOneConnectorDescriptor(spec), selectOp, 0, sortOp, 0);
+ } else {
+ spec.connect(new OneToOneConnectorDescriptor(spec), asterixAssignOp, 0, sortOp, 0);
+ }
spec.connect(new OneToOneConnectorDescriptor(spec), sortOp, 0, secondaryBulkLoadOp, 0);
spec.addRoot(root);
spec.setConnectorPolicyAssignmentPolicy(new ConnectorPolicyAssignmentPolicy());
@@ -138,7 +149,13 @@
spec.connect(new OneToOneConnectorDescriptor(spec), sourceOp, 0, targetOp, 0);
sourceOp = targetOp;
-
+ if (filterUnknowns) {
+ // if any of the secondary fields are nullable, then add a select op that filters nulls.
+ // assign op ----> select op
+ targetOp = createFilterNullsSelectOp(spec, indexDetails.getKeyFieldNames().size(), secondaryRecDesc);
+ spec.connect(new OneToOneConnectorDescriptor(spec), sourceOp, 0, targetOp, 0);
+ sourceOp = targetOp;
+ }
// no need to sort if the index is secondary primary index
if (!indexDetails.getKeyFieldNames().isEmpty()) {
// sort by secondary keys.
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java
index a6e3087..544071b 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java
@@ -47,9 +47,9 @@
import org.apache.asterix.om.functions.IFunctionDescriptor;
import org.apache.asterix.om.types.ARecordType;
import org.apache.asterix.om.types.IAType;
-import org.apache.asterix.runtime.evaluators.functions.AndDescriptor;
import org.apache.asterix.runtime.evaluators.functions.IsUnknownDescriptor;
import org.apache.asterix.runtime.evaluators.functions.NotDescriptor;
+import org.apache.asterix.runtime.evaluators.functions.OrDescriptor;
import org.apache.asterix.runtime.operators.LSMIndexBulkLoadOperatorDescriptor;
import org.apache.asterix.runtime.operators.LSMIndexBulkLoadOperatorDescriptor.BulkLoadUsage;
import org.apache.hyracks.algebricks.common.constraints.AlgebricksPartitionConstraint;
@@ -404,11 +404,10 @@
}
IScalarEvaluatorFactory selectCond;
if (numSecondaryKeyFields > 1) {
- // Create conjunctive condition where all secondary index keys must
- // satisfy 'is not null'.
- AndDescriptor andDesc = new AndDescriptor();
- andDesc.setSourceLocation(sourceLoc);
- selectCond = andDesc.createEvaluatorFactory(andArgsEvalFactories);
+ // create disjunctive condition where at least one key must be known to put entry into the index
+ OrDescriptor orDesc = new OrDescriptor();
+ orDesc.setSourceLocation(sourceLoc);
+ selectCond = orDesc.createEvaluatorFactory(andArgsEvalFactories);
} else {
selectCond = andArgsEvalFactories[0];
}
diff --git a/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslatorTest.java b/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslatorTest.java
index f690018..ad04bbe 100644
--- a/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslatorTest.java
+++ b/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslatorTest.java
@@ -73,7 +73,7 @@
Index index = new Index(dvTest, "d1", "i1", IndexType.BTREE,
Collections.singletonList(Collections.singletonList("row_id")),
indicator == null ? null : Collections.singletonList(indicator),
- Collections.singletonList(BuiltinType.AINT64), false, false, false, 0);
+ Collections.singletonList(BuiltinType.AINT64), false, false, false, 0, false);
MetadataNode mockMetadataNode = mock(MetadataNode.class);
when(mockMetadataNode.getDatatype(any(), any(DataverseName.class), anyString())).thenReturn(new Datatype(
diff --git a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorDescriptor.java b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorDescriptor.java
index 3231162..661b820 100644
--- a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorDescriptor.java
+++ b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorDescriptor.java
@@ -37,17 +37,19 @@
private final int[] prevValuePermutation;
protected final int operationFieldIndex;
protected final IBinaryIntegerInspectorFactory operationInspectorFactory;
+ private final ITupleFilterFactory prevTupleFilterFactory;
public LSMSecondaryUpsertOperatorDescriptor(IOperatorDescriptorRegistry spec, RecordDescriptor outRecDesc,
int[] fieldPermutation, IIndexDataflowHelperFactory indexHelperFactory,
- ITupleFilterFactory tupleFilterFactory, IModificationOperationCallbackFactory modificationOpCallbackFactory,
- int operationFieldIndex, IBinaryIntegerInspectorFactory operationInspectorFactory,
- int[] prevValuePermutation) {
+ ITupleFilterFactory tupleFilterFactory, ITupleFilterFactory prevTupleFilterFactory,
+ IModificationOperationCallbackFactory modificationOpCallbackFactory, int operationFieldIndex,
+ IBinaryIntegerInspectorFactory operationInspectorFactory, int[] prevValuePermutation) {
super(spec, outRecDesc, fieldPermutation, IndexOperation.UPSERT, indexHelperFactory, tupleFilterFactory, false,
modificationOpCallbackFactory);
this.prevValuePermutation = prevValuePermutation;
this.operationFieldIndex = operationFieldIndex;
this.operationInspectorFactory = operationInspectorFactory;
+ this.prevTupleFilterFactory = prevTupleFilterFactory;
}
@Override
@@ -55,7 +57,7 @@
IRecordDescriptorProvider recordDescProvider, int partition, int nPartitions) throws HyracksDataException {
RecordDescriptor intputRecDesc = recordDescProvider.getInputRecordDescriptor(getActivityId(), 0);
return new LSMSecondaryUpsertOperatorNodePushable(ctx, partition, indexHelperFactory, modCallbackFactory,
- tupleFilterFactory, fieldPermutation, intputRecDesc, operationFieldIndex, operationInspectorFactory,
- prevValuePermutation);
+ tupleFilterFactory, prevTupleFilterFactory, fieldPermutation, intputRecDesc, operationFieldIndex,
+ operationInspectorFactory, prevValuePermutation);
}
}
diff --git a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorNodePushable.java b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorNodePushable.java
index 482be06..68824b2 100644
--- a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorNodePushable.java
+++ b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorNodePushable.java
@@ -34,6 +34,7 @@
import org.apache.hyracks.dataflow.common.data.accessors.PermutingFrameTupleReference;
import org.apache.hyracks.dataflow.common.utils.TupleUtils;
import org.apache.hyracks.storage.am.common.api.IModificationOperationCallbackFactory;
+import org.apache.hyracks.storage.am.common.api.ITupleFilter;
import org.apache.hyracks.storage.am.common.api.ITupleFilterFactory;
import org.apache.hyracks.storage.am.common.dataflow.IIndexDataflowHelperFactory;
import org.apache.hyracks.storage.am.common.ophelpers.IndexOperation;
@@ -57,6 +58,8 @@
private final PermutingFrameTupleReference prevTuple = new PermutingFrameTupleReference();
private final int numberOfFields;
+ private final ITupleFilterFactory prevTupleFilterFactory;
+ private ITupleFilter prevTupleFilter;
protected final int operationFieldIndex;
protected final IBinaryIntegerInspector operationInspector;
@@ -64,15 +67,18 @@
public LSMSecondaryUpsertOperatorNodePushable(IHyracksTaskContext ctx, int partition,
IIndexDataflowHelperFactory indexHelperFactory, IModificationOperationCallbackFactory modCallbackFactory,
- ITupleFilterFactory tupleFilterFactory, int[] fieldPermutation, RecordDescriptor inputRecDesc,
- int operationFieldIndex, IBinaryIntegerInspectorFactory operationInspectorFactory,
- int[] prevTuplePermutation) throws HyracksDataException {
+ ITupleFilterFactory tupleFilterFactory, ITupleFilterFactory prevTupleFilterFactory, int[] fieldPermutation,
+ RecordDescriptor inputRecDesc, int operationFieldIndex,
+ IBinaryIntegerInspectorFactory operationInspectorFactory, int[] prevTuplePermutation)
+ throws HyracksDataException {
super(ctx, partition, indexHelperFactory, fieldPermutation, inputRecDesc, IndexOperation.UPSERT,
modCallbackFactory, tupleFilterFactory);
this.prevTuple.setFieldPermutation(prevTuplePermutation);
this.operationFieldIndex = operationFieldIndex;
this.operationInspector = operationInspectorFactory.createBinaryIntegerInspector(ctx);
this.numberOfFields = fieldPermutation.length;
+ this.prevTupleFilterFactory = prevTupleFilterFactory;
+
}
@Override
@@ -80,6 +86,9 @@
super.open();
frameTuple = new FrameTupleReference();
abstractModCallback = (AbstractIndexModificationOperationCallback) modCallback;
+ if (prevTupleFilterFactory != null) {
+ prevTupleFilter = prevTupleFilterFactory.createTupleFilter(ctx);
+ }
}
@Override
@@ -96,18 +105,30 @@
prevTuple.reset(accessor, i);
if (operation == UPSERT_NEW) {
- abstractModCallback.setOp(Operation.INSERT);
- lsmAccessor.forceInsert(tuple);
- } else if (operation == UPSERT_EXISTING) {
- if (!TupleUtils.equalTuples(tuple, prevTuple, numberOfFields)) {
- abstractModCallback.setOp(Operation.DELETE);
- lsmAccessor.forceDelete(prevTuple);
+ boolean processNewTuple = tupleFilter == null || tupleFilter.accept(frameTuple);
+ if (processNewTuple) {
abstractModCallback.setOp(Operation.INSERT);
lsmAccessor.forceInsert(tuple);
}
+ } else if (operation == UPSERT_EXISTING) {
+ if (!TupleUtils.equalTuples(tuple, prevTuple, numberOfFields)) {
+ boolean processOldTuple = prevTupleFilter == null || prevTupleFilter.accept(frameTuple);
+ if (processOldTuple) {
+ abstractModCallback.setOp(Operation.DELETE);
+ lsmAccessor.forceDelete(prevTuple);
+ }
+ boolean processNewTuple = tupleFilter == null || tupleFilter.accept(frameTuple);
+ if (processNewTuple) {
+ abstractModCallback.setOp(Operation.INSERT);
+ lsmAccessor.forceInsert(tuple);
+ }
+ }
} else if (operation == DELETE_EXISTING) {
- abstractModCallback.setOp(Operation.DELETE);
- lsmAccessor.forceDelete(prevTuple);
+ boolean processOldTuple = prevTupleFilter == null || prevTupleFilter.accept(frameTuple);
+ if (processOldTuple) {
+ abstractModCallback.setOp(Operation.DELETE);
+ lsmAccessor.forceDelete(prevTuple);
+ }
}
} catch (Exception e) {
throw HyracksDataException.create(e);
diff --git a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorDescriptor.java b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorDescriptor.java
index bbf0af1..41bd0fb 100644
--- a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorDescriptor.java
+++ b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorDescriptor.java
@@ -42,8 +42,8 @@
IModificationOperationCallbackFactory modCallbackFactory, int operationFieldIndex,
IBinaryIntegerInspectorFactory operationInspectorFactory, List<AlgebricksPipeline> secondaryKeysPipeline,
List<AlgebricksPipeline> prevSecondaryKeysPipeline) {
- super(spec, outRecDesc, fieldPermutation, indexHelperFactory, null, modCallbackFactory, operationFieldIndex,
- operationInspectorFactory, null);
+ super(spec, outRecDesc, fieldPermutation, indexHelperFactory, null, null, modCallbackFactory,
+ operationFieldIndex, operationInspectorFactory, null);
this.secondaryKeysPipeline = secondaryKeysPipeline;
this.prevSecondaryKeysPipeline = prevSecondaryKeysPipeline;
}
diff --git a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorNodePushable.java b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorNodePushable.java
index 3fe3e85..08fd566 100644
--- a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorNodePushable.java
+++ b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorNodePushable.java
@@ -51,7 +51,7 @@
int[] fieldPermutation, RecordDescriptor inputRecDesc, int operationFieldIndex,
IBinaryIntegerInspectorFactory operationInspectorFactory, List<AlgebricksPipeline> secondaryKeysPipeline,
List<AlgebricksPipeline> prevSecondaryKeysPipeline) throws HyracksDataException {
- super(ctx, partition, indexHelperFactory, modCallbackFactory, null, fieldPermutation, inputRecDesc,
+ super(ctx, partition, indexHelperFactory, modCallbackFactory, null, null, fieldPermutation, inputRecDesc,
operationFieldIndex, operationInspectorFactory, null);
this.numberOfPrimaryKeyAndFilterFields = fieldPermutation.length;
this.startOfNewKeyPipelines = buildStartOfPipelines(secondaryKeysPipeline, inputRecDesc, false);
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/metadata/IMetadataProvider.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/metadata/IMetadataProvider.java
index 9bafe3e..77fdd74 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/metadata/IMetadataProvider.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/metadata/IMetadataProvider.java
@@ -218,10 +218,10 @@
public Pair<IOperatorDescriptor, AlgebricksPartitionConstraint> getIndexUpsertRuntime(
IDataSourceIndex<I, S> dataSourceIndex, IOperatorSchema propagatedSchema, IOperatorSchema[] inputSchemas,
IVariableTypeEnvironment typeEnv, List<LogicalVariable> primaryKeys, List<LogicalVariable> secondaryKeys,
- List<LogicalVariable> additionalFilteringKeys, ILogicalExpression filterExpr, LogicalVariable operationVar,
- List<LogicalVariable> prevSecondaryKeys, LogicalVariable prevAdditionalFilteringKeys,
- RecordDescriptor inputDesc, JobGenContext context, JobSpecification spec,
- List<List<AlgebricksPipeline>> secondaryKeysPipelines) throws AlgebricksException;
+ List<LogicalVariable> additionalFilteringKeys, ILogicalExpression filterExpr,
+ ILogicalExpression prevFilterExpr, LogicalVariable operationVar, List<LogicalVariable> prevSecondaryKeys,
+ LogicalVariable prevAdditionalFilteringKeys, RecordDescriptor inputDesc, JobGenContext context,
+ JobSpecification spec, List<List<AlgebricksPipeline>> secondaryKeysPipelines) throws AlgebricksException;
public ITupleFilterFactory createTupleFilterFactory(IOperatorSchema[] inputSchemas,
IVariableTypeEnvironment typeEnv, ILogicalExpression filterExpr, JobGenContext context)
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/IndexInsertDeleteUpsertOperator.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/IndexInsertDeleteUpsertOperator.java
index f2ac41a..0098817 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/IndexInsertDeleteUpsertOperator.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/IndexInsertDeleteUpsertOperator.java
@@ -78,6 +78,7 @@
// Otherwise, it contains secondary key information.
private List<Mutable<ILogicalExpression>> secondaryKeyExprs;
private Mutable<ILogicalExpression> filterExpr;
+ private Mutable<ILogicalExpression> beforeOpFilterExpr;
private final Kind operation;
private final boolean bulkload;
private List<Mutable<ILogicalExpression>> additionalFilteringExpressions;
@@ -89,12 +90,13 @@
public IndexInsertDeleteUpsertOperator(IDataSourceIndex<?, ?> dataSourceIndex,
List<Mutable<ILogicalExpression>> primaryKeyExprs, List<Mutable<ILogicalExpression>> secondaryKeyExprs,
- Mutable<ILogicalExpression> filterExpr, Kind operation, boolean bulkload,
- int numberOfAdditionalNonFilteringFields) {
+ Mutable<ILogicalExpression> filterExpr, Mutable<ILogicalExpression> beforeOpFilterExpr, Kind operation,
+ boolean bulkload, int numberOfAdditionalNonFilteringFields) {
this.dataSourceIndex = dataSourceIndex;
this.primaryKeyExprs = primaryKeyExprs;
this.secondaryKeyExprs = secondaryKeyExprs;
this.filterExpr = filterExpr;
+ this.beforeOpFilterExpr = beforeOpFilterExpr;
this.operation = operation;
this.bulkload = bulkload;
this.numberOfAdditionalNonFilteringFields = numberOfAdditionalNonFilteringFields;
@@ -121,6 +123,12 @@
b = true;
}
}
+ // Old Filtering <For upsert>
+ if (beforeOpFilterExpr != null) {
+ if (visitor.transform(beforeOpFilterExpr)) {
+ b = true;
+ }
+ }
// Additional Filtering <For upsert>
if (additionalFilteringExpressions != null) {
for (int i = 0; i < additionalFilteringExpressions.size(); i++) {
@@ -141,7 +149,7 @@
}
}
}
- // Old Filtering <For upsert>
+ // Old Additional Filtering <For upsert>
if (prevAdditionalFilteringExpression != null) {
visitor.transform(prevAdditionalFilteringExpression);
}
@@ -164,6 +172,9 @@
if (getFilterExpression() != null) {
getFilterExpression().getValue().getUsedVariables(vars);
}
+ if (getBeforeOpFilterExpression() != null) {
+ getBeforeOpFilterExpression().getValue().getUsedVariables(vars);
+ }
if (getAdditionalFilteringExpressions() != null) {
for (Mutable<ILogicalExpression> e : getAdditionalFilteringExpressions()) {
e.getValue().getUsedVariables(vars);
@@ -232,10 +243,18 @@
return filterExpr;
}
+ public Mutable<ILogicalExpression> getBeforeOpFilterExpression() {
+ return beforeOpFilterExpr;
+ }
+
public void setFilterExpression(Mutable<ILogicalExpression> filterExpr) {
this.filterExpr = filterExpr;
}
+ public void setBeforeOpFilterExpression(Mutable<ILogicalExpression> beforeOpFilterExpr) {
+ this.beforeOpFilterExpr = beforeOpFilterExpr;
+ }
+
public Kind getOperation() {
return operation;
}
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java
index c4b4e47..aa5b5b4 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java
@@ -610,6 +610,7 @@
|| !Objects.equals(op.getPrimaryKeyExpressions(), insertOpArg.getPrimaryKeyExpressions())
|| !Objects.equals(op.getSecondaryKeyExpressions(), insertOpArg.getSecondaryKeyExpressions())
|| !Objects.equals(op.getFilterExpression(), insertOpArg.getFilterExpression())
+ || !Objects.equals(op.getBeforeOpFilterExpression(), insertOpArg.getBeforeOpFilterExpression())
|| !Objects.equals(op.getOperation(), insertOpArg.getOperation())
|| (op.isBulkload() != insertOpArg.isBulkload())
|| !Objects.equals(op.getAdditionalFilteringExpressions(),
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java
index c06ad5c..1c91d7b 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java
@@ -324,11 +324,14 @@
deepCopyExpressionRefs(newSecondaryKeyExpressions, op.getSecondaryKeyExpressions());
Mutable<ILogicalExpression> newFilterExpression =
new MutableObject<>(((AbstractLogicalExpression) op.getFilterExpression()).cloneExpression());
+ Mutable<ILogicalExpression> newBeforeOpFilterExpression =
+ new MutableObject<>(((AbstractLogicalExpression) op.getBeforeOpFilterExpression()).cloneExpression());
List<Mutable<ILogicalExpression>> newLSMComponentFilterExpressions = new ArrayList<>();
deepCopyExpressionRefs(newLSMComponentFilterExpressions, op.getAdditionalFilteringExpressions());
- IndexInsertDeleteUpsertOperator indexInsertDeleteOp = new IndexInsertDeleteUpsertOperator(
- op.getDataSourceIndex(), newPrimaryKeyExpressions, newSecondaryKeyExpressions, newFilterExpression,
- op.getOperation(), op.isBulkload(), op.getNumberOfAdditionalNonFilteringFields());
+ IndexInsertDeleteUpsertOperator indexInsertDeleteOp =
+ new IndexInsertDeleteUpsertOperator(op.getDataSourceIndex(), newPrimaryKeyExpressions,
+ newSecondaryKeyExpressions, newFilterExpression, newBeforeOpFilterExpression, op.getOperation(),
+ op.isBulkload(), op.getNumberOfAdditionalNonFilteringFields());
indexInsertDeleteOp.setAdditionalFilteringExpressions(newLSMComponentFilterExpressions);
for (ILogicalPlan plan : op.getNestedPlans()) {
indexInsertDeleteOp.getNestedPlans().add(OperatorManipulationUtil.deepCopy(plan, indexInsertDeleteOp));
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java
index 5aa63ae..439e493 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java
@@ -452,6 +452,7 @@
substUsedVariablesInExpr(op.getPrimaryKeyExpressions(), pair.first, pair.second);
substUsedVariablesInExpr(op.getSecondaryKeyExpressions(), pair.first, pair.second);
substUsedVariablesInExpr(op.getFilterExpression(), pair.first, pair.second);
+ substUsedVariablesInExpr(op.getBeforeOpFilterExpression(), pair.first, pair.second);
substUsedVariablesInExpr(op.getAdditionalFilteringExpressions(), pair.first, pair.second);
substUsedVariablesInExpr(op.getOperationExpr(), pair.first, pair.second);
substUsedVariablesInExpr(op.getPrevSecondaryKeyExprs(), pair.first, pair.second);
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java
index 4fb30b4..174b184 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java
@@ -416,6 +416,9 @@
if (op.getFilterExpression() != null) {
op.getFilterExpression().getValue().getUsedVariables(usedVariables);
}
+ if (op.getBeforeOpFilterExpression() != null) {
+ op.getBeforeOpFilterExpression().getValue().getUsedVariables(usedVariables);
+ }
if (op.getAdditionalFilteringExpressions() != null) {
for (Mutable<ILogicalExpression> e : op.getAdditionalFilteringExpressions()) {
e.getValue().getUsedVariables(usedVariables);
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/IndexInsertDeleteUpsertPOperator.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/IndexInsertDeleteUpsertPOperator.java
index 26581b1..3416163 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/IndexInsertDeleteUpsertPOperator.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/IndexInsertDeleteUpsertPOperator.java
@@ -55,6 +55,7 @@
private final List<LogicalVariable> primaryKeys;
private final List<LogicalVariable> secondaryKeys;
private final ILogicalExpression filterExpr;
+ private final ILogicalExpression prevFilterExpr;
private final IDataSourceIndex<?, ?> dataSourceIndex;
private final List<LogicalVariable> additionalFilteringKeys;
private final LogicalVariable operationVar;
@@ -64,9 +65,9 @@
public IndexInsertDeleteUpsertPOperator(List<LogicalVariable> primaryKeys, List<LogicalVariable> secondaryKeys,
List<LogicalVariable> additionalFilteringKeys, Mutable<ILogicalExpression> filterExpr,
- IDataSourceIndex<?, ?> dataSourceIndex, LogicalVariable operationVar,
- List<LogicalVariable> prevSecondaryKeys, LogicalVariable prevAdditionalFilteringKey,
- int numOfAdditionalNonFilteringFields) {
+ Mutable<ILogicalExpression> prevFilterExpr, IDataSourceIndex<?, ?> dataSourceIndex,
+ LogicalVariable operationVar, List<LogicalVariable> prevSecondaryKeys,
+ LogicalVariable prevAdditionalFilteringKey, int numOfAdditionalNonFilteringFields) {
this.primaryKeys = primaryKeys;
this.secondaryKeys = secondaryKeys;
if (filterExpr != null) {
@@ -74,6 +75,11 @@
} else {
this.filterExpr = null;
}
+ if (prevFilterExpr != null) {
+ this.prevFilterExpr = prevFilterExpr.getValue();
+ } else {
+ this.prevFilterExpr = null;
+ }
this.dataSourceIndex = dataSourceIndex;
this.additionalFilteringKeys = additionalFilteringKeys;
this.operationVar = operationVar;
@@ -157,8 +163,9 @@
break;
case UPSERT:
runtimeAndConstraints = mp.getIndexUpsertRuntime(dataSourceIndex, propagatedSchema, inputSchemas,
- typeEnv, primaryKeys, secondaryKeys, additionalFilteringKeys, filterExpr, operationVar,
- prevSecondaryKeys, prevAdditionalFilteringKey, inputDesc, context, spec, secondaryKeyPipelines);
+ typeEnv, primaryKeys, secondaryKeys, additionalFilteringKeys, filterExpr, prevFilterExpr,
+ operationVar, prevSecondaryKeys, prevAdditionalFilteringKey, inputDesc, context, spec,
+ secondaryKeyPipelines);
break;
default:
throw new AlgebricksException("Unsupported Operation " + operation);
diff --git a/hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/SetAlgebricksPhysicalOperatorsRule.java b/hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/SetAlgebricksPhysicalOperatorsRule.java
index cd0d996..a6fe495 100644
--- a/hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/SetAlgebricksPhysicalOperatorsRule.java
+++ b/hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/SetAlgebricksPhysicalOperatorsRule.java
@@ -442,8 +442,9 @@
}
}
return new IndexInsertDeleteUpsertPOperator(primaryKeys, secondaryKeys, additionalFilteringKeys,
- opInsDel.getFilterExpression(), opInsDel.getDataSourceIndex(), operationVar, prevSecondaryKeys,
- prevAdditionalFilteringKey, opInsDel.getNumberOfAdditionalNonFilteringFields());
+ opInsDel.getFilterExpression(), opInsDel.getBeforeOpFilterExpression(),
+ opInsDel.getDataSourceIndex(), operationVar, prevSecondaryKeys, prevAdditionalFilteringKey,
+ opInsDel.getNumberOfAdditionalNonFilteringFields());
}
}
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/12583
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I5a5111f9c8c4f1ae1c311609a64d7ebd413ca18f
Gerrit-Change-Number: 12583
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <al...@gmail.com>
Gerrit-MessageType: newchange
Change in asterixdb[master]: WIP: option to exclude unknowns in secondary indexes
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Ali Alsuliman <al...@gmail.com>:
Ali Alsuliman has abandoned this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/12583 )
Change subject: WIP: option to exclude unknowns in secondary indexes
......................................................................
Abandoned
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/12583
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I5a5111f9c8c4f1ae1c311609a64d7ebd413ca18f
Gerrit-Change-Number: 12583
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <al...@gmail.com>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-MessageType: abandon
Change in asterixdb[master]: WIP: option to exclude unknowns in secondary indexes
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Ali Alsuliman <al...@gmail.com>:
Ali Alsuliman has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/12583 )
Change subject: WIP: option to exclude unknowns in secondary indexes
......................................................................
WIP: option to exclude unknowns in secondary indexes
Change-Id: I5a5111f9c8c4f1ae1c311609a64d7ebd413ca18f
---
M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java
M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ConcurrentInsertTest.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-open-index/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-enforced/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateIndexStatement.java
M asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Index.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslator.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryBTreeOperationsHelper.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java
M asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslatorTest.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorNodePushable.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorNodePushable.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/metadata/IMetadataProvider.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/IndexInsertDeleteUpsertOperator.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/IndexInsertDeleteUpsertPOperator.java
M hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/SetAlgebricksPhysicalOperatorsRule.java
36 files changed, 263 insertions(+), 125 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/83/12583/1
diff --git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java
index a6ae0af..e3d5100 100644
--- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java
+++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java
@@ -313,6 +313,7 @@
// Set our key variables and expressions for non-array indexes. Our secondary keys for array indexes will
// always be an empty list.
List<LogicalVariable> secondaryKeyVars = new ArrayList<>();
+ List<LogicalVariable> beforeOpSecondaryKeyVars = new ArrayList<>();
List<Mutable<ILogicalExpression>> secondaryExpressions = new ArrayList<>();
List<Mutable<ILogicalExpression>> beforeOpSecondaryExpressions = new ArrayList<>();
ILogicalOperator replicateOutput;
@@ -324,23 +325,30 @@
secondaryKeyVars.add(skVar);
VariableReferenceExpression skVarRef = new VariableReferenceExpression(skVar);
skVarRef.setSourceLocation(sourceLoc);
- secondaryExpressions.add(new MutableObject<ILogicalExpression>(skVarRef));
+ secondaryExpressions.add(new MutableObject<>(skVarRef));
if (primaryIndexModificationOp.getOperation() == Kind.UPSERT) {
- VariableReferenceExpression varRef =
- new VariableReferenceExpression(fieldVarsForBeforeOperation.get(indexFieldId));
+ LogicalVariable beforeKeyVar = fieldVarsForBeforeOperation.get(indexFieldId);
+ beforeOpSecondaryKeyVars.add(beforeKeyVar);
+ VariableReferenceExpression varRef = new VariableReferenceExpression(beforeKeyVar);
varRef.setSourceLocation(sourceLoc);
- beforeOpSecondaryExpressions.add(new MutableObject<ILogicalExpression>(varRef));
+ beforeOpSecondaryExpressions.add(new MutableObject<>(varRef));
}
}
}
IndexInsertDeleteUpsertOperator indexUpdate;
+ boolean skipUnknowns = index.isSkipUnknowns();
if (index.getIndexType() != IndexType.RTREE) {
// Create an expression per key
- Mutable<ILogicalExpression> filterExpression = (primaryIndexModificationOp.getOperation() == Kind.UPSERT
- || index.getIndexType() == IndexType.BTREE) ? null
- : createFilterExpression(secondaryKeyVars, context.getOutputTypeEnvironment(currentTop),
- index.getIndexDetails().isOverridingKeyFieldTypes());
+ Mutable<ILogicalExpression> filterExpression =
+ createFilterExpression(secondaryKeyVars, context.getOutputTypeEnvironment(currentTop),
+ index.getIndexDetails().isOverridingKeyFieldTypes(), skipUnknowns);
+ Mutable<ILogicalExpression> beforeOpFilterExpression = null;
+ if (primaryIndexModificationOp.getOperation() == Kind.UPSERT) {
+ beforeOpFilterExpression = createFilterExpression(beforeOpSecondaryKeyVars,
+ context.getOutputTypeEnvironment(currentTop),
+ index.getIndexDetails().isOverridingKeyFieldTypes(), skipUnknowns);
+ }
DataSourceIndex dataSourceIndex = new DataSourceIndex(index, dataverseName, datasetName, mp);
// Introduce the TokenizeOperator only when doing bulk-load,
@@ -406,8 +414,8 @@
indexUpdate = new IndexInsertDeleteUpsertOperator(dataSourceIndex,
OperatorManipulationUtil
.cloneExpressions(primaryIndexModificationOp.getPrimaryKeyExpressions()),
- tokenizeKeyExprs, filterExpression, primaryIndexModificationOp.getOperation(),
- primaryIndexModificationOp.isBulkload(),
+ tokenizeKeyExprs, filterExpression, beforeOpFilterExpression,
+ primaryIndexModificationOp.getOperation(), primaryIndexModificationOp.isBulkload(),
primaryIndexModificationOp.getAdditionalNonFilteringExpressions() == null ? 0
: primaryIndexModificationOp.getAdditionalNonFilteringExpressions().size());
indexUpdate.setSourceLocation(sourceLoc);
@@ -419,8 +427,8 @@
indexUpdate = new IndexInsertDeleteUpsertOperator(dataSourceIndex,
OperatorManipulationUtil
.cloneExpressions(primaryIndexModificationOp.getPrimaryKeyExpressions()),
- secondaryExpressions, filterExpression, primaryIndexModificationOp.getOperation(),
- primaryIndexModificationOp.isBulkload(),
+ secondaryExpressions, filterExpression, beforeOpFilterExpression,
+ primaryIndexModificationOp.getOperation(), primaryIndexModificationOp.isBulkload(),
primaryIndexModificationOp.getAdditionalNonFilteringExpressions() == null ? 0
: primaryIndexModificationOp.getAdditionalNonFilteringExpressions().size());
indexUpdate.setSourceLocation(sourceLoc);
@@ -455,7 +463,7 @@
filterExpression = (primaryIndexModificationOp.getOperation() == Kind.UPSERT) ? null
: createFilterExpression(unnestSIDXBranch.lastFieldVars,
context.getOutputTypeEnvironment(unnestSIDXBranch.currentTop),
- index.getIndexDetails().isOverridingKeyFieldTypes());
+ index.getIndexDetails().isOverridingKeyFieldTypes(), skipUnknowns);
if (filterExpression != null) {
unnestSIDXBranch.applyFilteringExpression(filterExpression);
}
@@ -500,7 +508,7 @@
// Update the filter expression to include these new keys.
filterExpression = createFilterExpression(unnestSIDXBranch.lastFieldVars,
context.getOutputTypeEnvironment(unnestSIDXBranch.currentTop),
- index.getIndexDetails().isOverridingKeyFieldTypes());
+ index.getIndexDetails().isOverridingKeyFieldTypes(), skipUnknowns);
indexUpdate.setFilterExpression(filterExpression);
if (replicateOp != null) {
@@ -560,8 +568,11 @@
assignCoordinates.getInputs().add(new MutableObject<ILogicalOperator>(currentTop));
context.computeAndSetTypeEnvironmentForOperator(assignCoordinates);
replicateOutput = assignCoordinates;
- Mutable<ILogicalExpression> filterExpression = null;
+ boolean forceFilter = keyPairType.second;
+ Mutable<ILogicalExpression> filterExpression = createFilterExpression(keyVarList,
+ context.getOutputTypeEnvironment(assignCoordinates), forceFilter, skipUnknowns);
AssignOperator originalAssignCoordinates = null;
+ Mutable<ILogicalExpression> beforeOpFilterExpression = null;
// We do something similar for beforeOp key if the operation is an upsert
if (primaryIndexModificationOp.getOperation() == Kind.UPSERT) {
List<LogicalVariable> originalKeyVarList = new ArrayList<>();
@@ -592,19 +603,15 @@
originalAssignCoordinates.setSourceLocation(sourceLoc);
originalAssignCoordinates.getInputs().add(new MutableObject<ILogicalOperator>(assignCoordinates));
context.computeAndSetTypeEnvironmentForOperator(originalAssignCoordinates);
- } else {
- // We must enforce the filter if the originating spatial type is
- // nullable.
- boolean forceFilter = keyPairType.second;
- filterExpression = createFilterExpression(keyVarList,
- context.getOutputTypeEnvironment(assignCoordinates), forceFilter);
+ beforeOpFilterExpression = createFilterExpression(originalKeyVarList,
+ context.getOutputTypeEnvironment(originalAssignCoordinates), forceFilter, skipUnknowns);
}
DataSourceIndex dataSourceIndex = new DataSourceIndex(index, dataverseName, datasetName, mp);
indexUpdate = new IndexInsertDeleteUpsertOperator(dataSourceIndex,
OperatorManipulationUtil
.cloneExpressions(primaryIndexModificationOp.getPrimaryKeyExpressions()),
- secondaryExpressions, filterExpression, primaryIndexModificationOp.getOperation(),
- primaryIndexModificationOp.isBulkload(),
+ secondaryExpressions, filterExpression, beforeOpFilterExpression,
+ primaryIndexModificationOp.getOperation(), primaryIndexModificationOp.isBulkload(),
primaryIndexModificationOp.getAdditionalNonFilteringExpressions() == null ? 0
: primaryIndexModificationOp.getAdditionalNonFilteringExpressions().size());
indexUpdate.setSourceLocation(sourceLoc);
@@ -943,7 +950,10 @@
}
private Mutable<ILogicalExpression> createFilterExpression(List<LogicalVariable> secondaryKeyVars,
- IVariableTypeEnvironment typeEnv, boolean forceFilter) throws AlgebricksException {
+ IVariableTypeEnvironment typeEnv, boolean forceFilter, boolean skipUnknowns) throws AlgebricksException {
+ if (!skipUnknowns) {
+ return null;
+ }
List<Mutable<ILogicalExpression>> filterExpressions = new ArrayList<>();
// Add 'is not null' to all nullable secondary index keys as a filtering
// condition.
@@ -971,10 +981,10 @@
Mutable<ILogicalExpression> filterExpression;
if (filterExpressions.size() > 1) {
// Create a conjunctive condition.
- ScalarFunctionCallExpression andExpr = new ScalarFunctionCallExpression(
- FunctionUtil.getFunctionInfo(BuiltinFunctions.AND), filterExpressions);
- andExpr.setSourceLocation(sourceLoc);
- filterExpression = new MutableObject<>(andExpr);
+ ScalarFunctionCallExpression orExpr = new ScalarFunctionCallExpression(
+ FunctionUtil.getFunctionInfo(BuiltinFunctions.OR), filterExpressions);
+ orExpr.setSourceLocation(sourceLoc);
+ filterExpression = new MutableObject<>(orExpr);
} else {
filterExpression = filterExpressions.get(0);
}
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
index 267bd76..9d5c5f5 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
@@ -1291,7 +1291,7 @@
}
Index newIndex = new Index(dataverseName, datasetName, indexName, indexType, indexDetails,
- stmtCreateIndex.isEnforced(), false, MetadataUtil.PENDING_ADD_OP);
+ stmtCreateIndex.isEnforced(), false, MetadataUtil.PENDING_ADD_OP, stmtCreateIndex.isSkipUnknowns());
bActiveTxn = false; // doCreateIndexImpl() takes over the current transaction
doCreateIndexImpl(hcc, metadataProvider, ds, newIndex, jobFlags, sourceLoc);
@@ -1489,7 +1489,7 @@
IndexingConstants.getFilesIndexName(index.getDatasetName()), IndexType.BTREE,
new Index.ValueIndexDetails(ExternalIndexingOperations.FILE_INDEX_FIELD_NAMES, null,
ExternalIndexingOperations.FILE_INDEX_FIELD_TYPES, false),
- false, false, MetadataUtil.PENDING_ADD_OP);
+ false, false, MetadataUtil.PENDING_ADD_OP, index.isSkipUnknowns());
MetadataManager.INSTANCE.addIndex(metadataProvider.getMetadataTxnContext(), filesIndex);
// Add files to the external files index
for (ExternalFile file : externalFilesSnapshot) {
@@ -2137,7 +2137,8 @@
MetadataManager.INSTANCE.dropIndex(mdTxnCtx, dataverseName, datasetName, indexName);
MetadataManager.INSTANCE.addIndex(mdTxnCtx,
new Index(dataverseName, datasetName, indexName, index.getIndexType(), index.getIndexDetails(),
- index.isEnforced(), index.isPrimaryIndex(), MetadataUtil.PENDING_DROP_OP));
+ index.isEnforced(), index.isPrimaryIndex(), MetadataUtil.PENDING_DROP_OP,
+ index.isSkipUnknowns()));
// #. commit the existing transaction before calling runJob.
MetadataManager.INSTANCE.commitTransaction(mdTxnCtx);
@@ -2189,7 +2190,7 @@
new Index(dataverseName, datasetName, externalIndex.getIndexName(),
externalIndex.getIndexType(), externalIndex.getIndexDetails(),
externalIndex.isEnforced(), externalIndex.isPrimaryIndex(),
- MetadataUtil.PENDING_DROP_OP));
+ MetadataUtil.PENDING_DROP_OP, externalIndex.isSkipUnknowns()));
}
}
}
@@ -2198,7 +2199,8 @@
MetadataManager.INSTANCE.dropIndex(mdTxnCtx, dataverseName, datasetName, indexName);
MetadataManager.INSTANCE.addIndex(mdTxnCtx,
new Index(dataverseName, datasetName, indexName, index.getIndexType(), index.getIndexDetails(),
- index.isEnforced(), index.isPrimaryIndex(), MetadataUtil.PENDING_DROP_OP));
+ index.isEnforced(), index.isPrimaryIndex(), MetadataUtil.PENDING_DROP_OP,
+ index.isSkipUnknowns()));
// #. commit the existing transaction before calling runJob.
MetadataManager.INSTANCE.commitTransaction(mdTxnCtx);
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
index c668fb6..9e8edd3 100644
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
@@ -721,7 +721,7 @@
index = new Index(dataset.getDataverseName(), dataset.getDatasetName(), dataset.getDatasetName(),
IndexType.BTREE,
new Index.ValueIndexDetails(keyFieldNames, primaryKeyIndicators, keyFieldTypes, false), false, true,
- MetadataUtil.PENDING_NO_OP);
+ MetadataUtil.PENDING_NO_OP, false);
List<String> nodes = Collections.singletonList(ExecutionTestUtil.integrationUtil.ncs[0].getId());
CcApplicationContext appCtx =
(CcApplicationContext) ExecutionTestUtil.integrationUtil.cc.getApplicationContext();
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java
index 1913233..caab0b8 100644
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java
@@ -171,7 +171,7 @@
partitioningKeys, null, null, null, false, null, null),
null, DatasetType.INTERNAL, DATASET_ID, 0);
secondaryIndex = new Index(dvName, DATASET_NAME, INDEX_NAME, INDEX_TYPE, INDEX_FIELD_NAMES,
- INDEX_FIELD_INDICATORS, INDEX_FIELD_TYPES, false, false, false, 0);
+ INDEX_FIELD_INDICATORS, INDEX_FIELD_TYPES, false, false, false, 0, false);
taskCtx = null;
primaryIndexDataflowHelper = null;
secondaryIndexDataflowHelper = null;
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ConcurrentInsertTest.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ConcurrentInsertTest.java
index 76dd035..841536e 100644
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ConcurrentInsertTest.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ConcurrentInsertTest.java
@@ -128,7 +128,8 @@
Index secondaryIndexEntity = new Index(StorageTestUtils.DATASET.getDataverseName(),
StorageTestUtils.DATASET.getDatasetName(), "TestIndex", IndexType.BTREE,
Arrays.asList(Arrays.asList(StorageTestUtils.RECORD_TYPE.getFieldNames()[1])),
- Arrays.asList(Index.RECORD_INDICATOR), Arrays.asList(BuiltinType.AINT64), false, false, false, 0);
+ Arrays.asList(Index.RECORD_INDICATOR), Arrays.asList(BuiltinType.AINT64), false, false, false, 0,
+ false);
SecondaryIndexInfo secondaryIndexInfo =
nc.createSecondaryIndex(primaryIndexInfo, secondaryIndexEntity, StorageTestUtils.STORAGE_MANAGER, 0);
@@ -142,7 +143,7 @@
Index primaryKeyIndexEntity = new Index(StorageTestUtils.DATASET.getDataverseName(),
StorageTestUtils.DATASET.getDatasetName(), "PrimaryKeyIndex", IndexType.BTREE, Arrays.asList(),
- Arrays.asList(), Arrays.asList(), false, false, false, 0);
+ Arrays.asList(), Arrays.asList(), false, false, false, 0, false);
SecondaryIndexInfo primaryKeyIndexInfo =
nc.createSecondaryIndex(primaryIndexInfo, primaryKeyIndexEntity, StorageTestUtils.STORAGE_MANAGER, 0);
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java
index 2aad416..f408758 100644
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java
@@ -187,7 +187,7 @@
dataset = StorageTestUtils.DATASET;
secondaryIndexEntity = new Index(dataset.getDataverseName(), dataset.getDatasetName(), SECONDARY_INDEX_NAME,
SECONDARY_INDEX_TYPE, SECONDARY_INDEX_FIELD_NAMES, SECONDARY_INDEX_FIELD_INDICATORS,
- SECONDARY_INDEX_FIELD_TYPES, false, false, false, 0);
+ SECONDARY_INDEX_FIELD_TYPES, false, false, false, 0, false);
primaryIndexInfos = new PrimaryIndexInfo[NUM_PARTITIONS];
secondaryIndexInfo = new SecondaryIndexInfo[NUM_PARTITIONS];
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java
index ef3a7cb..42ddd95 100644
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java
@@ -161,7 +161,7 @@
partitioningKeys, null, null, null, false, null, null),
null, DatasetType.INTERNAL, DATASET_ID, 0);
secondaryIndex = new Index(dvName, DATASET_NAME, INDEX_NAME, INDEX_TYPE, INDEX_FIELD_NAMES,
- INDEX_FIELD_INDICATORS, INDEX_FIELD_TYPES, false, false, false, 0);
+ INDEX_FIELD_INDICATORS, INDEX_FIELD_TYPES, false, false, false, 0, false);
taskCtxs = new IHyracksTaskContext[NUM_PARTITIONS];
primaryIndexDataflowHelpers = new IIndexDataflowHelper[NUM_PARTITIONS];
secondaryIndexDataflowHelpers = new IIndexDataflowHelper[NUM_PARTITIONS];
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
index 62dfbe2..606c2e7 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
@@ -26,5 +26,5 @@
use test;
-create index ngram_index on DBLP (title) type ngram (3);
+create index ngram_index on DBLP (title) type ngram (3) exclude unknown key;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
index 662b250..5e6021c 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
@@ -20,5 +20,5 @@
use test;
-create index rtree_index_point on MyData (point) type rtree;
+create index rtree_index_point on MyData (point) type rtree exclude unknown key;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
index c57941c..6876008 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-inverted-index-ngram-secondary-index-nullable/scan-delete-inverted-index-ngram-secondary-index-nullable.3.ddl.sqlpp
@@ -26,5 +26,5 @@
use test;
-create index ngram_index on DBLP (nested.title) type ngram (3);
+create index ngram_index on DBLP (nested.title) type ngram (3) exclude unknown key;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
index 5683fe0..4eae583 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-index-dml/scan-delete-rtree-secondary-index-nullable/scan-delete-rtree-secondary-index-nullable.3.ddl.sqlpp
@@ -20,5 +20,5 @@
use test;
-create index rtree_index_point on MyData (nested.point) type rtree;
+create index rtree_index_point on MyData (nested.point) type rtree exclude unknown key;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-open-index/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-open-index/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
index 01e97f6..5603f55 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-open-index/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/nested-open-index/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
@@ -59,5 +59,5 @@
create dataset TweetMessagesTmp(TweetMessageNestedType) primary key tweetid;
-create index msgNgramIx on TweetMessages (nested.`message-text`:string?) type ngram (3) enforced;
+create index msgNgramIx on TweetMessages (nested.`message-text`:string?) type ngram (3) enforced exclude unknown key;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-enforced/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-enforced/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
index 4ff0a38..ac7843d 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-enforced/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-enforced/index-leftouterjoin/probe-pidx-with-join-invidx-sidx2/probe-pidx-with-join-invidx-sidx2.1.ddl.sqlpp
@@ -54,5 +54,5 @@
create dataset TweetMessagesTmp(TweetMessageType) primary key tweetid;
-create index msgNgramIx on TweetMessages (`message-text`:string?) type ngram (3) enforced;
+create index msgNgramIx on TweetMessages (`message-text`:string?) type ngram (3) enforced exclude unknown key;
diff --git a/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateIndexStatement.java b/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateIndexStatement.java
index 468006c..d7d66a6 100644
--- a/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateIndexStatement.java
+++ b/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateIndexStatement.java
@@ -48,10 +48,11 @@
private final int gramLength;
// Specific to FullText indexes.
private final String fullTextConfigName;
+ private final boolean skipUnknowns;
public CreateIndexStatement(DataverseName dataverseName, Identifier datasetName, Identifier indexName,
IndexType indexType, List<IndexedElement> indexedElements, boolean enforced, int gramLength,
- String fullTextConfigName, boolean ifNotExists) {
+ String fullTextConfigName, boolean ifNotExists, boolean skipUnknowns) {
this.dataverseName = dataverseName;
this.datasetName = Objects.requireNonNull(datasetName);
this.indexName = Objects.requireNonNull(indexName);
@@ -61,6 +62,7 @@
this.gramLength = gramLength;
this.ifNotExists = ifNotExists;
this.fullTextConfigName = fullTextConfigName;
+ this.skipUnknowns = skipUnknowns;
}
public String getFullTextConfigName() {
@@ -91,6 +93,10 @@
return enforced;
}
+ public boolean isSkipUnknowns() {
+ return skipUnknowns;
+ }
+
public int getGramLength() {
return gramLength;
}
diff --git a/asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj b/asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
index de9ea7a..fb24366 100644
--- a/asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
+++ b/asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
@@ -227,6 +227,7 @@
private static final String CURRENT = "CURRENT";
private static final String DEFAULT = "DEFAULT";
private static final String EXCLUDE = "EXCLUDE";
+ private static final String INCLUDE = "INCLUDE";
private static final String FIRST = "FIRST";
private static final String FOLLOWING = "FOLLOWING";
private static final String GROUPING = "GROUPING";
@@ -1131,6 +1132,7 @@
boolean hasUnnest = false;
String fullTextConfigName = null;
Token startElementToken = null;
+ boolean excludeUnknown = false;
}
{
(
@@ -1149,6 +1151,17 @@
)*
<RIGHTPAREN>
( <TYPE> indexParams = IndexType() )? ( <ENFORCED> { enforced = true; } )?
+ ( <IDENTIFIER>
+ {
+ if (isToken(EXCLUDE)) {
+ excludeUnknown = true;
+ } else if (isToken(INCLUDE)) {
+ excludeUnknown = false;
+ } else {
+ throw createUnexpectedTokenError();
+ }
+ } <UNKNOWN> <KEY>
+ )?
)
{
IndexType indexType;
@@ -1163,7 +1176,8 @@
fullTextConfigName = null;
}
CreateIndexStatement stmt = new CreateIndexStatement(nameComponents.first, nameComponents.second,
- new Identifier(indexName), indexType, indexedElementList, enforced, gramLength, fullTextConfigName, ifNotExists);
+ new Identifier(indexName), indexType, indexedElementList, enforced, gramLength, fullTextConfigName, ifNotExists,
+ excludeUnknown);
return addSourceLocation(stmt, startStmtToken);
}
}
@@ -1292,7 +1306,7 @@
indexName = "primary_idx_" + nameComponents.second;
}
CreateIndexStatement stmt = new CreateIndexStatement(nameComponents.first, nameComponents.second,
- new Identifier(indexName), IndexType.BTREE, Collections.emptyList(), false, -1, null, ifNotExists);
+ new Identifier(indexName), IndexType.BTREE, Collections.emptyList(), false, -1, null, ifNotExists, false);
return addSourceLocation(stmt, startStmtToken);
}
}
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
index a5dda0d..26423ef 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java
@@ -366,7 +366,7 @@
InternalDatasetDetails id = (InternalDatasetDetails) dataset.getDatasetDetails();
Index primaryIndex = new Index(dataset.getDataverseName(), dataset.getDatasetName(),
dataset.getDatasetName(), IndexType.BTREE, id.getPrimaryKey(), id.getKeySourceIndicator(),
- id.getPrimaryKeyType(), false, false, true, dataset.getPendingOp());
+ id.getPrimaryKeyType(), false, false, true, dataset.getPendingOp(), false);
addIndex(txnId, primaryIndex);
}
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java
index 48fb450..1287670 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java
@@ -144,8 +144,8 @@
}
public void dropIndex(DataverseName dataverseName, String datasetName, String indexName) {
- Index index =
- new Index(dataverseName, datasetName, indexName, null, null, false, false, MetadataUtil.PENDING_NO_OP);
+ Index index = new Index(dataverseName, datasetName, indexName, null, null, false, false,
+ MetadataUtil.PENDING_NO_OP, false);
droppedCache.addIndexIfNotExists(index);
logAndApply(new MetadataLogicalOperation(index, false));
}
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
index 6f91572..3ca3750 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
@@ -785,7 +785,7 @@
boolean bulkload, List<List<AlgebricksPipeline>> secondaryKeysPipelines, IOperatorSchema pipelineTopSchema)
throws AlgebricksException {
return getIndexInsertOrDeleteOrUpsertRuntime(IndexOperation.INSERT, dataSourceIndex, propagatedSchema,
- inputSchemas, typeEnv, primaryKeys, secondaryKeys, additionalNonKeyFields, filterExpr, recordDesc,
+ inputSchemas, typeEnv, primaryKeys, secondaryKeys, additionalNonKeyFields, filterExpr, null, recordDesc,
context, spec, bulkload, null, null, null, secondaryKeysPipelines, pipelineTopSchema);
}
@@ -798,7 +798,7 @@
List<List<AlgebricksPipeline>> secondaryKeysPipelines, IOperatorSchema pipelineTopSchema)
throws AlgebricksException {
return getIndexInsertOrDeleteOrUpsertRuntime(IndexOperation.DELETE, dataSourceIndex, propagatedSchema,
- inputSchemas, typeEnv, primaryKeys, secondaryKeys, additionalNonKeyFields, filterExpr, recordDesc,
+ inputSchemas, typeEnv, primaryKeys, secondaryKeys, additionalNonKeyFields, filterExpr, null, recordDesc,
context, spec, false, null, null, null, secondaryKeysPipelines, pipelineTopSchema);
}
@@ -807,12 +807,13 @@
IDataSourceIndex<String, DataSourceId> dataSourceIndex, IOperatorSchema propagatedSchema,
IOperatorSchema[] inputSchemas, IVariableTypeEnvironment typeEnv, List<LogicalVariable> primaryKeys,
List<LogicalVariable> secondaryKeys, List<LogicalVariable> additionalFilteringKeys,
- ILogicalExpression filterExpr, LogicalVariable operationVar, List<LogicalVariable> prevSecondaryKeys,
- LogicalVariable prevAdditionalFilteringKey, RecordDescriptor recordDesc, JobGenContext context,
- JobSpecification spec, List<List<AlgebricksPipeline>> secondaryKeysPipelines) throws AlgebricksException {
+ ILogicalExpression filterExpr, ILogicalExpression prevFilterExpr, LogicalVariable operationVar,
+ List<LogicalVariable> prevSecondaryKeys, LogicalVariable prevAdditionalFilteringKey,
+ RecordDescriptor recordDesc, JobGenContext context, JobSpecification spec,
+ List<List<AlgebricksPipeline>> secondaryKeysPipelines) throws AlgebricksException {
return getIndexInsertOrDeleteOrUpsertRuntime(IndexOperation.UPSERT, dataSourceIndex, propagatedSchema,
- inputSchemas, typeEnv, primaryKeys, secondaryKeys, additionalFilteringKeys, filterExpr, recordDesc,
- context, spec, false, operationVar, prevSecondaryKeys, prevAdditionalFilteringKey,
+ inputSchemas, typeEnv, primaryKeys, secondaryKeys, additionalFilteringKeys, filterExpr, prevFilterExpr,
+ recordDesc, context, spec, false, operationVar, prevSecondaryKeys, prevAdditionalFilteringKey,
secondaryKeysPipelines, null);
}
@@ -1216,10 +1217,11 @@
IOperatorSchema propagatedSchema, IOperatorSchema[] inputSchemas, IVariableTypeEnvironment typeEnv,
List<LogicalVariable> primaryKeys, List<LogicalVariable> secondaryKeys,
List<LogicalVariable> additionalNonKeyFields, ILogicalExpression filterExpr,
- RecordDescriptor inputRecordDesc, JobGenContext context, JobSpecification spec, boolean bulkload,
- LogicalVariable operationVar, List<LogicalVariable> prevSecondaryKeys,
- LogicalVariable prevAdditionalFilteringKey, List<List<AlgebricksPipeline>> secondaryKeysPipelines,
- IOperatorSchema pipelineTopSchema) throws AlgebricksException {
+ ILogicalExpression prevFilterExpr, RecordDescriptor inputRecordDesc, JobGenContext context,
+ JobSpecification spec, boolean bulkload, LogicalVariable operationVar,
+ List<LogicalVariable> prevSecondaryKeys, LogicalVariable prevAdditionalFilteringKey,
+ List<List<AlgebricksPipeline>> secondaryKeysPipelines, IOperatorSchema pipelineTopSchema)
+ throws AlgebricksException {
String indexName = dataSourceIndex.getId();
DataverseName dataverseName = dataSourceIndex.getDataSource().getId().getDataverseName();
String datasetName = dataSourceIndex.getDataSource().getId().getDatasourceName();
@@ -1236,28 +1238,31 @@
// If we have a pipeline, then we need to pass the schema of the pipeline to the filter factory.
AsterixTupleFilterFactory filterFactory;
+ AsterixTupleFilterFactory prevFilterFactory;
if (pipelineTopSchema != null) {
IOperatorSchema[] schemasForFilterFactory = new IOperatorSchema[inputSchemas.length + 1];
System.arraycopy(inputSchemas, 0, schemasForFilterFactory, 0, inputSchemas.length);
schemasForFilterFactory[inputSchemas.length] = pipelineTopSchema;
filterFactory = createTupleFilterFactory(schemasForFilterFactory, typeEnv, filterExpr, context);
-
+ prevFilterFactory = createTupleFilterFactory(schemasForFilterFactory, typeEnv, prevFilterExpr, context);
} else {
filterFactory = createTupleFilterFactory(inputSchemas, typeEnv, filterExpr, context);
+ prevFilterFactory = createTupleFilterFactory(inputSchemas, typeEnv, prevFilterExpr, context);
}
switch (secondaryIndex.getIndexType()) {
case BTREE:
return getBTreeRuntime(dataverseName, datasetName, indexName, propagatedSchema, primaryKeys,
- secondaryKeys, additionalNonKeyFields, filterFactory, inputRecordDesc, context, spec, indexOp,
- bulkload, operationVar, prevSecondaryKeys, prevAdditionalFilteringKeys);
+ secondaryKeys, additionalNonKeyFields, filterFactory, prevFilterFactory, inputRecordDesc,
+ context, spec, indexOp, bulkload, operationVar, prevSecondaryKeys, prevAdditionalFilteringKeys);
case ARRAY:
if (bulkload) {
// In the case of bulk-load, we do not handle any nested plans. We perform the exact same behavior
// as a normal B-Tree bulk load.
return getBTreeRuntime(dataverseName, datasetName, indexName, propagatedSchema, primaryKeys,
- secondaryKeys, additionalNonKeyFields, filterFactory, inputRecordDesc, context, spec,
- indexOp, bulkload, operationVar, prevSecondaryKeys, prevAdditionalFilteringKeys);
+ secondaryKeys, additionalNonKeyFields, filterFactory, prevFilterFactory, inputRecordDesc,
+ context, spec, indexOp, bulkload, operationVar, prevSecondaryKeys,
+ prevAdditionalFilteringKeys);
} else {
return getArrayIndexRuntime(dataverseName, datasetName, indexName, propagatedSchema, primaryKeys,
additionalNonKeyFields, inputRecordDesc, spec, indexOp, operationVar,
@@ -1284,10 +1289,10 @@
private Pair<IOperatorDescriptor, AlgebricksPartitionConstraint> getBTreeRuntime(DataverseName dataverseName,
String datasetName, String indexName, IOperatorSchema propagatedSchema, List<LogicalVariable> primaryKeys,
List<LogicalVariable> secondaryKeys, List<LogicalVariable> additionalNonKeyFields,
- AsterixTupleFilterFactory filterFactory, RecordDescriptor inputRecordDesc, JobGenContext context,
- JobSpecification spec, IndexOperation indexOp, boolean bulkload, LogicalVariable operationVar,
- List<LogicalVariable> prevSecondaryKeys, List<LogicalVariable> prevAdditionalFilteringKeys)
- throws AlgebricksException {
+ AsterixTupleFilterFactory filterFactory, AsterixTupleFilterFactory prevFilterFactory,
+ RecordDescriptor inputRecordDesc, JobGenContext context, JobSpecification spec, IndexOperation indexOp,
+ boolean bulkload, LogicalVariable operationVar, List<LogicalVariable> prevSecondaryKeys,
+ List<LogicalVariable> prevAdditionalFilteringKeys) throws AlgebricksException {
Dataset dataset = MetadataManagerUtil.findExistingDataset(mdTxnCtx, dataverseName, datasetName);
int numKeys = primaryKeys.size() + secondaryKeys.size();
int numFilterFields = DatasetUtil.getFilterField(dataset) == null ? 0 : 1;
@@ -1356,8 +1361,8 @@
} else if (indexOp == IndexOperation.UPSERT) {
int operationFieldIndex = propagatedSchema.findVariable(operationVar);
op = new LSMSecondaryUpsertOperatorDescriptor(spec, inputRecordDesc, fieldPermutation, idfh,
- filterFactory, modificationCallbackFactory, operationFieldIndex, BinaryIntegerInspector.FACTORY,
- prevFieldPermutation);
+ filterFactory, prevFilterFactory, modificationCallbackFactory, operationFieldIndex,
+ BinaryIntegerInspector.FACTORY, prevFieldPermutation);
} else {
op = new LSMTreeInsertDeleteOperatorDescriptor(spec, inputRecordDesc, fieldPermutation, indexOp, idfh,
filterFactory, false, modificationCallbackFactory);
@@ -1511,7 +1516,7 @@
} else if (indexOp == IndexOperation.UPSERT) {
int operationFieldIndex = propagatedSchema.findVariable(operationVar);
op = new LSMSecondaryUpsertOperatorDescriptor(spec, recordDesc, fieldPermutation,
- indexDataflowHelperFactory, filterFactory, modificationCallbackFactory, operationFieldIndex,
+ indexDataflowHelperFactory, filterFactory, null, modificationCallbackFactory, operationFieldIndex,
BinaryIntegerInspector.FACTORY, prevFieldPermutation);
} else {
op = new LSMTreeInsertDeleteOperatorDescriptor(spec, recordDesc, fieldPermutation, indexOp,
@@ -1624,7 +1629,7 @@
} else if (indexOp == IndexOperation.UPSERT) {
int upsertOperationFieldIndex = propagatedSchema.findVariable(operationVar);
op = new LSMSecondaryUpsertOperatorDescriptor(spec, recordDesc, fieldPermutation, indexDataFlowFactory,
- filterFactory, modificationCallbackFactory, upsertOperationFieldIndex,
+ filterFactory, null, modificationCallbackFactory, upsertOperationFieldIndex,
BinaryIntegerInspector.FACTORY, prevFieldPermutation);
} else {
op = new LSMTreeInsertDeleteOperatorDescriptor(spec, recordDesc, fieldPermutation, indexOp,
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Index.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Index.java
index cf09779..edf6c09 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Index.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Index.java
@@ -57,11 +57,13 @@
private final IIndexDetails indexDetails;
private final boolean isPrimaryIndex;
private final boolean isEnforced;
+ private final boolean isSkipUnknowns;
// Type of pending operations with respect to atomic DDL operation
private int pendingOp;
public Index(DataverseName dataverseName, String datasetName, String indexName, IndexType indexType,
- IIndexDetails indexDetails, boolean isEnforced, boolean isPrimaryIndex, int pendingOp) {
+ IIndexDetails indexDetails, boolean isEnforced, boolean isPrimaryIndex, int pendingOp,
+ boolean isSkipUnknowns) {
boolean categoryOk = (indexType == null && indexDetails == null) || (IndexCategory
.of(Objects.requireNonNull(indexType)) == ((AbstractIndexDetails) Objects.requireNonNull(indexDetails))
.getIndexCategory());
@@ -76,16 +78,18 @@
this.isPrimaryIndex = isPrimaryIndex;
this.isEnforced = isEnforced;
this.pendingOp = pendingOp;
+ this.isSkipUnknowns = isSkipUnknowns;
}
@Deprecated
public Index(DataverseName dataverseName, String datasetName, String indexName, IndexType indexType,
List<List<String>> keyFieldNames, List<Integer> keyFieldSourceIndicators, List<IAType> keyFieldTypes,
- boolean overrideKeyFieldTypes, boolean isEnforced, boolean isPrimaryIndex, int pendingOp) {
+ boolean overrideKeyFieldTypes, boolean isEnforced, boolean isPrimaryIndex, int pendingOp,
+ boolean isSkipUnknowns) {
this(dataverseName,
datasetName, indexName, indexType, createSimpleIndexDetails(indexType, keyFieldNames,
keyFieldSourceIndicators, keyFieldTypes, overrideKeyFieldTypes),
- isEnforced, isPrimaryIndex, pendingOp);
+ isEnforced, isPrimaryIndex, pendingOp, isSkipUnknowns);
}
public DataverseName getDataverseName() {
@@ -133,6 +137,10 @@
return indexType == IndexType.BTREE && ((ValueIndexDetails) indexDetails).keyFieldNames.isEmpty();
}
+ public boolean isSkipUnknowns() {
+ return isSkipUnknowns;
+ }
+
public static Pair<IAType, Boolean> getNonNullableType(IAType keyType) {
boolean nullable = false;
IAType actualKeyType = keyType;
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslator.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslator.java
index 32764b2..3d8c456 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslator.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslator.java
@@ -81,6 +81,7 @@
public static final String FULL_TEXT_CONFIG_FIELD_NAME = "FullTextConfig";
public static final String INDEX_SEARCHKEY_TYPE_FIELD_NAME = "SearchKeyType";
public static final String INDEX_ISENFORCED_FIELD_NAME = "IsEnforced";
+ public static final String INDEX_SKIP_UNKNOWNS_FIELD_NAME = "SkipUnknowns";
public static final String INDEX_SEARCHKEY_SOURCE_INDICATOR_FIELD_NAME = "SearchKeySourceIndicator";
public static final String INDEX_SEARCHKEY_ELEMENTS_FIELD_NAME = "SearchKeyElements";
public static final String COMPLEXSEARCHKEY_UNNEST_FIELD_NAME = "UnnestList";
@@ -419,8 +420,13 @@
int pendingOp = ((AInt32) indexRecord.getValueByPos(MetadataRecordTypes.INDEX_ARECORD_PENDINGOP_FIELD_INDEX))
.getIntegerValue();
+ int skipUnknownsPos = indexRecord.getType().getFieldIndex(INDEX_SKIP_UNKNOWNS_FIELD_NAME);
+ Boolean skipUnknowns = false;
+ if (skipUnknownsPos > 0) {
+ skipUnknowns = ((ABoolean) indexRecord.getValueByPos(skipUnknownsPos)).getBoolean();
+ }
return new Index(dataverseName, datasetName, indexName, indexType, indexDetails, isEnforcingKeys,
- isPrimaryIndex, pendingOp);
+ isPrimaryIndex, pendingOp, skipUnknowns);
}
@Override
@@ -549,6 +555,7 @@
writeSearchKeyType(index);
writeEnforced(index);
writeSearchKeySourceIndicator(index);
+ writeSkipUnknowns(index);
}
private void writeComplexSearchKeys(Index.ArrayIndexDetails indexDetails) throws HyracksDataException {
@@ -748,4 +755,15 @@
recordBuilder.addField(nameValue, fieldValue);
}
}
+
+ private void writeSkipUnknowns(Index index) throws HyracksDataException {
+ if (index.isSkipUnknowns()) {
+ fieldValue.reset();
+ nameValue.reset();
+ aString.setValue(INDEX_SKIP_UNKNOWNS_FIELD_NAME);
+ stringSerde.serialize(aString, nameValue.getDataOutput());
+ booleanSerde.serialize(ABoolean.TRUE, fieldValue.getDataOutput());
+ recordBuilder.addField(nameValue, fieldValue);
+ }
+ }
}
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java
index f5b2697..d15ac31 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java
@@ -60,7 +60,7 @@
InternalDatasetDetails id = (InternalDatasetDetails) dataset.getDatasetDetails();
return new Index(dataset.getDataverseName(), dataset.getDatasetName(), dataset.getDatasetName(),
DatasetConfig.IndexType.BTREE, id.getPartitioningKey(), id.getKeySourceIndicator(),
- id.getPrimaryKeyType(), false, false, true, dataset.getPendingOp());
+ id.getPrimaryKeyType(), false, false, true, dataset.getPendingOp(), false);
}
public static int[] getBtreeFieldsIfFiltered(Dataset dataset, Index index) throws AlgebricksException {
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryBTreeOperationsHelper.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryBTreeOperationsHelper.java
index 9645e7a..e265af1 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryBTreeOperationsHelper.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryBTreeOperationsHelper.java
@@ -70,6 +70,7 @@
int[] fieldPermutation = createFieldPermutationForBulkLoadOp(indexDetails.getKeyFieldNames().size());
IIndexDataflowHelperFactory dataflowHelperFactory = new IndexDataflowHelperFactory(
metadataProvider.getStorageComponentProvider().getStorageManager(), secondaryFileSplitProvider);
+ boolean filterUnknowns = index.isSkipUnknowns() && (anySecondaryKeyIsNullable || isOverridingKeyFieldTypes);
if (dataset.getDatasetType() == DatasetType.EXTERNAL) {
/*
* In case of external data,
@@ -89,6 +90,11 @@
AlgebricksMetaOperatorDescriptor asterixAssignOp =
createExternalAssignOp(spec, indexDetails.getKeyFieldNames().size(), secondaryRecDesc);
+ // If any of the secondary fields are nullable, then add a select op that filters nulls.
+ AlgebricksMetaOperatorDescriptor selectOp = null;
+ if (filterUnknowns) {
+ selectOp = createFilterNullsSelectOp(spec, indexDetails.getKeyFieldNames().size(), secondaryRecDesc);
+ }
// Sort by secondary keys.
ExternalSortOperatorDescriptor sortOp = createSortOp(spec, secondaryComparatorFactories, secondaryRecDesc);
// Create secondary BTree bulk load op.
@@ -111,7 +117,12 @@
spec.connect(new OneToOneConnectorDescriptor(spec), secondaryBulkLoadOp, 0, metaOp, 0);
root = metaOp;
spec.connect(new OneToOneConnectorDescriptor(spec), sourceOp, 0, asterixAssignOp, 0);
- spec.connect(new OneToOneConnectorDescriptor(spec), asterixAssignOp, 0, sortOp, 0);
+ if (filterUnknowns) {
+ spec.connect(new OneToOneConnectorDescriptor(spec), asterixAssignOp, 0, selectOp, 0);
+ spec.connect(new OneToOneConnectorDescriptor(spec), selectOp, 0, sortOp, 0);
+ } else {
+ spec.connect(new OneToOneConnectorDescriptor(spec), asterixAssignOp, 0, sortOp, 0);
+ }
spec.connect(new OneToOneConnectorDescriptor(spec), sortOp, 0, secondaryBulkLoadOp, 0);
spec.addRoot(root);
spec.setConnectorPolicyAssignmentPolicy(new ConnectorPolicyAssignmentPolicy());
@@ -138,7 +149,13 @@
spec.connect(new OneToOneConnectorDescriptor(spec), sourceOp, 0, targetOp, 0);
sourceOp = targetOp;
-
+ if (filterUnknowns) {
+ // if any of the secondary fields are nullable, then add a select op that filters nulls.
+ // assign op ----> select op
+ targetOp = createFilterNullsSelectOp(spec, indexDetails.getKeyFieldNames().size(), secondaryRecDesc);
+ spec.connect(new OneToOneConnectorDescriptor(spec), sourceOp, 0, targetOp, 0);
+ sourceOp = targetOp;
+ }
// no need to sort if the index is secondary primary index
if (!indexDetails.getKeyFieldNames().isEmpty()) {
// sort by secondary keys.
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java
index a6e3087..544071b 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java
@@ -47,9 +47,9 @@
import org.apache.asterix.om.functions.IFunctionDescriptor;
import org.apache.asterix.om.types.ARecordType;
import org.apache.asterix.om.types.IAType;
-import org.apache.asterix.runtime.evaluators.functions.AndDescriptor;
import org.apache.asterix.runtime.evaluators.functions.IsUnknownDescriptor;
import org.apache.asterix.runtime.evaluators.functions.NotDescriptor;
+import org.apache.asterix.runtime.evaluators.functions.OrDescriptor;
import org.apache.asterix.runtime.operators.LSMIndexBulkLoadOperatorDescriptor;
import org.apache.asterix.runtime.operators.LSMIndexBulkLoadOperatorDescriptor.BulkLoadUsage;
import org.apache.hyracks.algebricks.common.constraints.AlgebricksPartitionConstraint;
@@ -404,11 +404,10 @@
}
IScalarEvaluatorFactory selectCond;
if (numSecondaryKeyFields > 1) {
- // Create conjunctive condition where all secondary index keys must
- // satisfy 'is not null'.
- AndDescriptor andDesc = new AndDescriptor();
- andDesc.setSourceLocation(sourceLoc);
- selectCond = andDesc.createEvaluatorFactory(andArgsEvalFactories);
+ // create disjunctive condition where at least one key must be known to put entry into the index
+ OrDescriptor orDesc = new OrDescriptor();
+ orDesc.setSourceLocation(sourceLoc);
+ selectCond = orDesc.createEvaluatorFactory(andArgsEvalFactories);
} else {
selectCond = andArgsEvalFactories[0];
}
diff --git a/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslatorTest.java b/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslatorTest.java
index f690018..ad04bbe 100644
--- a/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslatorTest.java
+++ b/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslatorTest.java
@@ -73,7 +73,7 @@
Index index = new Index(dvTest, "d1", "i1", IndexType.BTREE,
Collections.singletonList(Collections.singletonList("row_id")),
indicator == null ? null : Collections.singletonList(indicator),
- Collections.singletonList(BuiltinType.AINT64), false, false, false, 0);
+ Collections.singletonList(BuiltinType.AINT64), false, false, false, 0, false);
MetadataNode mockMetadataNode = mock(MetadataNode.class);
when(mockMetadataNode.getDatatype(any(), any(DataverseName.class), anyString())).thenReturn(new Datatype(
diff --git a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorDescriptor.java b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorDescriptor.java
index 3231162..661b820 100644
--- a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorDescriptor.java
+++ b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorDescriptor.java
@@ -37,17 +37,19 @@
private final int[] prevValuePermutation;
protected final int operationFieldIndex;
protected final IBinaryIntegerInspectorFactory operationInspectorFactory;
+ private final ITupleFilterFactory prevTupleFilterFactory;
public LSMSecondaryUpsertOperatorDescriptor(IOperatorDescriptorRegistry spec, RecordDescriptor outRecDesc,
int[] fieldPermutation, IIndexDataflowHelperFactory indexHelperFactory,
- ITupleFilterFactory tupleFilterFactory, IModificationOperationCallbackFactory modificationOpCallbackFactory,
- int operationFieldIndex, IBinaryIntegerInspectorFactory operationInspectorFactory,
- int[] prevValuePermutation) {
+ ITupleFilterFactory tupleFilterFactory, ITupleFilterFactory prevTupleFilterFactory,
+ IModificationOperationCallbackFactory modificationOpCallbackFactory, int operationFieldIndex,
+ IBinaryIntegerInspectorFactory operationInspectorFactory, int[] prevValuePermutation) {
super(spec, outRecDesc, fieldPermutation, IndexOperation.UPSERT, indexHelperFactory, tupleFilterFactory, false,
modificationOpCallbackFactory);
this.prevValuePermutation = prevValuePermutation;
this.operationFieldIndex = operationFieldIndex;
this.operationInspectorFactory = operationInspectorFactory;
+ this.prevTupleFilterFactory = prevTupleFilterFactory;
}
@Override
@@ -55,7 +57,7 @@
IRecordDescriptorProvider recordDescProvider, int partition, int nPartitions) throws HyracksDataException {
RecordDescriptor intputRecDesc = recordDescProvider.getInputRecordDescriptor(getActivityId(), 0);
return new LSMSecondaryUpsertOperatorNodePushable(ctx, partition, indexHelperFactory, modCallbackFactory,
- tupleFilterFactory, fieldPermutation, intputRecDesc, operationFieldIndex, operationInspectorFactory,
- prevValuePermutation);
+ tupleFilterFactory, prevTupleFilterFactory, fieldPermutation, intputRecDesc, operationFieldIndex,
+ operationInspectorFactory, prevValuePermutation);
}
}
diff --git a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorNodePushable.java b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorNodePushable.java
index 482be06..68824b2 100644
--- a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorNodePushable.java
+++ b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertOperatorNodePushable.java
@@ -34,6 +34,7 @@
import org.apache.hyracks.dataflow.common.data.accessors.PermutingFrameTupleReference;
import org.apache.hyracks.dataflow.common.utils.TupleUtils;
import org.apache.hyracks.storage.am.common.api.IModificationOperationCallbackFactory;
+import org.apache.hyracks.storage.am.common.api.ITupleFilter;
import org.apache.hyracks.storage.am.common.api.ITupleFilterFactory;
import org.apache.hyracks.storage.am.common.dataflow.IIndexDataflowHelperFactory;
import org.apache.hyracks.storage.am.common.ophelpers.IndexOperation;
@@ -57,6 +58,8 @@
private final PermutingFrameTupleReference prevTuple = new PermutingFrameTupleReference();
private final int numberOfFields;
+ private final ITupleFilterFactory prevTupleFilterFactory;
+ private ITupleFilter prevTupleFilter;
protected final int operationFieldIndex;
protected final IBinaryIntegerInspector operationInspector;
@@ -64,15 +67,18 @@
public LSMSecondaryUpsertOperatorNodePushable(IHyracksTaskContext ctx, int partition,
IIndexDataflowHelperFactory indexHelperFactory, IModificationOperationCallbackFactory modCallbackFactory,
- ITupleFilterFactory tupleFilterFactory, int[] fieldPermutation, RecordDescriptor inputRecDesc,
- int operationFieldIndex, IBinaryIntegerInspectorFactory operationInspectorFactory,
- int[] prevTuplePermutation) throws HyracksDataException {
+ ITupleFilterFactory tupleFilterFactory, ITupleFilterFactory prevTupleFilterFactory, int[] fieldPermutation,
+ RecordDescriptor inputRecDesc, int operationFieldIndex,
+ IBinaryIntegerInspectorFactory operationInspectorFactory, int[] prevTuplePermutation)
+ throws HyracksDataException {
super(ctx, partition, indexHelperFactory, fieldPermutation, inputRecDesc, IndexOperation.UPSERT,
modCallbackFactory, tupleFilterFactory);
this.prevTuple.setFieldPermutation(prevTuplePermutation);
this.operationFieldIndex = operationFieldIndex;
this.operationInspector = operationInspectorFactory.createBinaryIntegerInspector(ctx);
this.numberOfFields = fieldPermutation.length;
+ this.prevTupleFilterFactory = prevTupleFilterFactory;
+
}
@Override
@@ -80,6 +86,9 @@
super.open();
frameTuple = new FrameTupleReference();
abstractModCallback = (AbstractIndexModificationOperationCallback) modCallback;
+ if (prevTupleFilterFactory != null) {
+ prevTupleFilter = prevTupleFilterFactory.createTupleFilter(ctx);
+ }
}
@Override
@@ -96,18 +105,30 @@
prevTuple.reset(accessor, i);
if (operation == UPSERT_NEW) {
- abstractModCallback.setOp(Operation.INSERT);
- lsmAccessor.forceInsert(tuple);
- } else if (operation == UPSERT_EXISTING) {
- if (!TupleUtils.equalTuples(tuple, prevTuple, numberOfFields)) {
- abstractModCallback.setOp(Operation.DELETE);
- lsmAccessor.forceDelete(prevTuple);
+ boolean processNewTuple = tupleFilter == null || tupleFilter.accept(frameTuple);
+ if (processNewTuple) {
abstractModCallback.setOp(Operation.INSERT);
lsmAccessor.forceInsert(tuple);
}
+ } else if (operation == UPSERT_EXISTING) {
+ if (!TupleUtils.equalTuples(tuple, prevTuple, numberOfFields)) {
+ boolean processOldTuple = prevTupleFilter == null || prevTupleFilter.accept(frameTuple);
+ if (processOldTuple) {
+ abstractModCallback.setOp(Operation.DELETE);
+ lsmAccessor.forceDelete(prevTuple);
+ }
+ boolean processNewTuple = tupleFilter == null || tupleFilter.accept(frameTuple);
+ if (processNewTuple) {
+ abstractModCallback.setOp(Operation.INSERT);
+ lsmAccessor.forceInsert(tuple);
+ }
+ }
} else if (operation == DELETE_EXISTING) {
- abstractModCallback.setOp(Operation.DELETE);
- lsmAccessor.forceDelete(prevTuple);
+ boolean processOldTuple = prevTupleFilter == null || prevTupleFilter.accept(frameTuple);
+ if (processOldTuple) {
+ abstractModCallback.setOp(Operation.DELETE);
+ lsmAccessor.forceDelete(prevTuple);
+ }
}
} catch (Exception e) {
throw HyracksDataException.create(e);
diff --git a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorDescriptor.java b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorDescriptor.java
index bbf0af1..41bd0fb 100644
--- a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorDescriptor.java
+++ b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorDescriptor.java
@@ -42,8 +42,8 @@
IModificationOperationCallbackFactory modCallbackFactory, int operationFieldIndex,
IBinaryIntegerInspectorFactory operationInspectorFactory, List<AlgebricksPipeline> secondaryKeysPipeline,
List<AlgebricksPipeline> prevSecondaryKeysPipeline) {
- super(spec, outRecDesc, fieldPermutation, indexHelperFactory, null, modCallbackFactory, operationFieldIndex,
- operationInspectorFactory, null);
+ super(spec, outRecDesc, fieldPermutation, indexHelperFactory, null, null, modCallbackFactory,
+ operationFieldIndex, operationInspectorFactory, null);
this.secondaryKeysPipeline = secondaryKeysPipeline;
this.prevSecondaryKeysPipeline = prevSecondaryKeysPipeline;
}
diff --git a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorNodePushable.java b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorNodePushable.java
index 3fe3e85..08fd566 100644
--- a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorNodePushable.java
+++ b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorNodePushable.java
@@ -51,7 +51,7 @@
int[] fieldPermutation, RecordDescriptor inputRecDesc, int operationFieldIndex,
IBinaryIntegerInspectorFactory operationInspectorFactory, List<AlgebricksPipeline> secondaryKeysPipeline,
List<AlgebricksPipeline> prevSecondaryKeysPipeline) throws HyracksDataException {
- super(ctx, partition, indexHelperFactory, modCallbackFactory, null, fieldPermutation, inputRecDesc,
+ super(ctx, partition, indexHelperFactory, modCallbackFactory, null, null, fieldPermutation, inputRecDesc,
operationFieldIndex, operationInspectorFactory, null);
this.numberOfPrimaryKeyAndFilterFields = fieldPermutation.length;
this.startOfNewKeyPipelines = buildStartOfPipelines(secondaryKeysPipeline, inputRecDesc, false);
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/metadata/IMetadataProvider.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/metadata/IMetadataProvider.java
index 9bafe3e..77fdd74 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/metadata/IMetadataProvider.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/metadata/IMetadataProvider.java
@@ -218,10 +218,10 @@
public Pair<IOperatorDescriptor, AlgebricksPartitionConstraint> getIndexUpsertRuntime(
IDataSourceIndex<I, S> dataSourceIndex, IOperatorSchema propagatedSchema, IOperatorSchema[] inputSchemas,
IVariableTypeEnvironment typeEnv, List<LogicalVariable> primaryKeys, List<LogicalVariable> secondaryKeys,
- List<LogicalVariable> additionalFilteringKeys, ILogicalExpression filterExpr, LogicalVariable operationVar,
- List<LogicalVariable> prevSecondaryKeys, LogicalVariable prevAdditionalFilteringKeys,
- RecordDescriptor inputDesc, JobGenContext context, JobSpecification spec,
- List<List<AlgebricksPipeline>> secondaryKeysPipelines) throws AlgebricksException;
+ List<LogicalVariable> additionalFilteringKeys, ILogicalExpression filterExpr,
+ ILogicalExpression prevFilterExpr, LogicalVariable operationVar, List<LogicalVariable> prevSecondaryKeys,
+ LogicalVariable prevAdditionalFilteringKeys, RecordDescriptor inputDesc, JobGenContext context,
+ JobSpecification spec, List<List<AlgebricksPipeline>> secondaryKeysPipelines) throws AlgebricksException;
public ITupleFilterFactory createTupleFilterFactory(IOperatorSchema[] inputSchemas,
IVariableTypeEnvironment typeEnv, ILogicalExpression filterExpr, JobGenContext context)
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/IndexInsertDeleteUpsertOperator.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/IndexInsertDeleteUpsertOperator.java
index f2ac41a..0098817 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/IndexInsertDeleteUpsertOperator.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/IndexInsertDeleteUpsertOperator.java
@@ -78,6 +78,7 @@
// Otherwise, it contains secondary key information.
private List<Mutable<ILogicalExpression>> secondaryKeyExprs;
private Mutable<ILogicalExpression> filterExpr;
+ private Mutable<ILogicalExpression> beforeOpFilterExpr;
private final Kind operation;
private final boolean bulkload;
private List<Mutable<ILogicalExpression>> additionalFilteringExpressions;
@@ -89,12 +90,13 @@
public IndexInsertDeleteUpsertOperator(IDataSourceIndex<?, ?> dataSourceIndex,
List<Mutable<ILogicalExpression>> primaryKeyExprs, List<Mutable<ILogicalExpression>> secondaryKeyExprs,
- Mutable<ILogicalExpression> filterExpr, Kind operation, boolean bulkload,
- int numberOfAdditionalNonFilteringFields) {
+ Mutable<ILogicalExpression> filterExpr, Mutable<ILogicalExpression> beforeOpFilterExpr, Kind operation,
+ boolean bulkload, int numberOfAdditionalNonFilteringFields) {
this.dataSourceIndex = dataSourceIndex;
this.primaryKeyExprs = primaryKeyExprs;
this.secondaryKeyExprs = secondaryKeyExprs;
this.filterExpr = filterExpr;
+ this.beforeOpFilterExpr = beforeOpFilterExpr;
this.operation = operation;
this.bulkload = bulkload;
this.numberOfAdditionalNonFilteringFields = numberOfAdditionalNonFilteringFields;
@@ -121,6 +123,12 @@
b = true;
}
}
+ // Old Filtering <For upsert>
+ if (beforeOpFilterExpr != null) {
+ if (visitor.transform(beforeOpFilterExpr)) {
+ b = true;
+ }
+ }
// Additional Filtering <For upsert>
if (additionalFilteringExpressions != null) {
for (int i = 0; i < additionalFilteringExpressions.size(); i++) {
@@ -141,7 +149,7 @@
}
}
}
- // Old Filtering <For upsert>
+ // Old Additional Filtering <For upsert>
if (prevAdditionalFilteringExpression != null) {
visitor.transform(prevAdditionalFilteringExpression);
}
@@ -164,6 +172,9 @@
if (getFilterExpression() != null) {
getFilterExpression().getValue().getUsedVariables(vars);
}
+ if (getBeforeOpFilterExpression() != null) {
+ getBeforeOpFilterExpression().getValue().getUsedVariables(vars);
+ }
if (getAdditionalFilteringExpressions() != null) {
for (Mutable<ILogicalExpression> e : getAdditionalFilteringExpressions()) {
e.getValue().getUsedVariables(vars);
@@ -232,10 +243,18 @@
return filterExpr;
}
+ public Mutable<ILogicalExpression> getBeforeOpFilterExpression() {
+ return beforeOpFilterExpr;
+ }
+
public void setFilterExpression(Mutable<ILogicalExpression> filterExpr) {
this.filterExpr = filterExpr;
}
+ public void setBeforeOpFilterExpression(Mutable<ILogicalExpression> beforeOpFilterExpr) {
+ this.beforeOpFilterExpr = beforeOpFilterExpr;
+ }
+
public Kind getOperation() {
return operation;
}
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java
index c4b4e47..aa5b5b4 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java
@@ -610,6 +610,7 @@
|| !Objects.equals(op.getPrimaryKeyExpressions(), insertOpArg.getPrimaryKeyExpressions())
|| !Objects.equals(op.getSecondaryKeyExpressions(), insertOpArg.getSecondaryKeyExpressions())
|| !Objects.equals(op.getFilterExpression(), insertOpArg.getFilterExpression())
+ || !Objects.equals(op.getBeforeOpFilterExpression(), insertOpArg.getBeforeOpFilterExpression())
|| !Objects.equals(op.getOperation(), insertOpArg.getOperation())
|| (op.isBulkload() != insertOpArg.isBulkload())
|| !Objects.equals(op.getAdditionalFilteringExpressions(),
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java
index c06ad5c..1c91d7b 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java
@@ -324,11 +324,14 @@
deepCopyExpressionRefs(newSecondaryKeyExpressions, op.getSecondaryKeyExpressions());
Mutable<ILogicalExpression> newFilterExpression =
new MutableObject<>(((AbstractLogicalExpression) op.getFilterExpression()).cloneExpression());
+ Mutable<ILogicalExpression> newBeforeOpFilterExpression =
+ new MutableObject<>(((AbstractLogicalExpression) op.getBeforeOpFilterExpression()).cloneExpression());
List<Mutable<ILogicalExpression>> newLSMComponentFilterExpressions = new ArrayList<>();
deepCopyExpressionRefs(newLSMComponentFilterExpressions, op.getAdditionalFilteringExpressions());
- IndexInsertDeleteUpsertOperator indexInsertDeleteOp = new IndexInsertDeleteUpsertOperator(
- op.getDataSourceIndex(), newPrimaryKeyExpressions, newSecondaryKeyExpressions, newFilterExpression,
- op.getOperation(), op.isBulkload(), op.getNumberOfAdditionalNonFilteringFields());
+ IndexInsertDeleteUpsertOperator indexInsertDeleteOp =
+ new IndexInsertDeleteUpsertOperator(op.getDataSourceIndex(), newPrimaryKeyExpressions,
+ newSecondaryKeyExpressions, newFilterExpression, newBeforeOpFilterExpression, op.getOperation(),
+ op.isBulkload(), op.getNumberOfAdditionalNonFilteringFields());
indexInsertDeleteOp.setAdditionalFilteringExpressions(newLSMComponentFilterExpressions);
for (ILogicalPlan plan : op.getNestedPlans()) {
indexInsertDeleteOp.getNestedPlans().add(OperatorManipulationUtil.deepCopy(plan, indexInsertDeleteOp));
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java
index 5aa63ae..439e493 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java
@@ -452,6 +452,7 @@
substUsedVariablesInExpr(op.getPrimaryKeyExpressions(), pair.first, pair.second);
substUsedVariablesInExpr(op.getSecondaryKeyExpressions(), pair.first, pair.second);
substUsedVariablesInExpr(op.getFilterExpression(), pair.first, pair.second);
+ substUsedVariablesInExpr(op.getBeforeOpFilterExpression(), pair.first, pair.second);
substUsedVariablesInExpr(op.getAdditionalFilteringExpressions(), pair.first, pair.second);
substUsedVariablesInExpr(op.getOperationExpr(), pair.first, pair.second);
substUsedVariablesInExpr(op.getPrevSecondaryKeyExprs(), pair.first, pair.second);
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java
index 4fb30b4..174b184 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java
@@ -416,6 +416,9 @@
if (op.getFilterExpression() != null) {
op.getFilterExpression().getValue().getUsedVariables(usedVariables);
}
+ if (op.getBeforeOpFilterExpression() != null) {
+ op.getBeforeOpFilterExpression().getValue().getUsedVariables(usedVariables);
+ }
if (op.getAdditionalFilteringExpressions() != null) {
for (Mutable<ILogicalExpression> e : op.getAdditionalFilteringExpressions()) {
e.getValue().getUsedVariables(usedVariables);
diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/IndexInsertDeleteUpsertPOperator.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/IndexInsertDeleteUpsertPOperator.java
index 26581b1..3416163 100644
--- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/IndexInsertDeleteUpsertPOperator.java
+++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/IndexInsertDeleteUpsertPOperator.java
@@ -55,6 +55,7 @@
private final List<LogicalVariable> primaryKeys;
private final List<LogicalVariable> secondaryKeys;
private final ILogicalExpression filterExpr;
+ private final ILogicalExpression prevFilterExpr;
private final IDataSourceIndex<?, ?> dataSourceIndex;
private final List<LogicalVariable> additionalFilteringKeys;
private final LogicalVariable operationVar;
@@ -64,9 +65,9 @@
public IndexInsertDeleteUpsertPOperator(List<LogicalVariable> primaryKeys, List<LogicalVariable> secondaryKeys,
List<LogicalVariable> additionalFilteringKeys, Mutable<ILogicalExpression> filterExpr,
- IDataSourceIndex<?, ?> dataSourceIndex, LogicalVariable operationVar,
- List<LogicalVariable> prevSecondaryKeys, LogicalVariable prevAdditionalFilteringKey,
- int numOfAdditionalNonFilteringFields) {
+ Mutable<ILogicalExpression> prevFilterExpr, IDataSourceIndex<?, ?> dataSourceIndex,
+ LogicalVariable operationVar, List<LogicalVariable> prevSecondaryKeys,
+ LogicalVariable prevAdditionalFilteringKey, int numOfAdditionalNonFilteringFields) {
this.primaryKeys = primaryKeys;
this.secondaryKeys = secondaryKeys;
if (filterExpr != null) {
@@ -74,6 +75,11 @@
} else {
this.filterExpr = null;
}
+ if (prevFilterExpr != null) {
+ this.prevFilterExpr = prevFilterExpr.getValue();
+ } else {
+ this.prevFilterExpr = null;
+ }
this.dataSourceIndex = dataSourceIndex;
this.additionalFilteringKeys = additionalFilteringKeys;
this.operationVar = operationVar;
@@ -157,8 +163,9 @@
break;
case UPSERT:
runtimeAndConstraints = mp.getIndexUpsertRuntime(dataSourceIndex, propagatedSchema, inputSchemas,
- typeEnv, primaryKeys, secondaryKeys, additionalFilteringKeys, filterExpr, operationVar,
- prevSecondaryKeys, prevAdditionalFilteringKey, inputDesc, context, spec, secondaryKeyPipelines);
+ typeEnv, primaryKeys, secondaryKeys, additionalFilteringKeys, filterExpr, prevFilterExpr,
+ operationVar, prevSecondaryKeys, prevAdditionalFilteringKey, inputDesc, context, spec,
+ secondaryKeyPipelines);
break;
default:
throw new AlgebricksException("Unsupported Operation " + operation);
diff --git a/hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/SetAlgebricksPhysicalOperatorsRule.java b/hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/SetAlgebricksPhysicalOperatorsRule.java
index cd0d996..a6fe495 100644
--- a/hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/SetAlgebricksPhysicalOperatorsRule.java
+++ b/hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/SetAlgebricksPhysicalOperatorsRule.java
@@ -442,8 +442,9 @@
}
}
return new IndexInsertDeleteUpsertPOperator(primaryKeys, secondaryKeys, additionalFilteringKeys,
- opInsDel.getFilterExpression(), opInsDel.getDataSourceIndex(), operationVar, prevSecondaryKeys,
- prevAdditionalFilteringKey, opInsDel.getNumberOfAdditionalNonFilteringFields());
+ opInsDel.getFilterExpression(), opInsDel.getBeforeOpFilterExpression(),
+ opInsDel.getDataSourceIndex(), operationVar, prevSecondaryKeys, prevAdditionalFilteringKey,
+ opInsDel.getNumberOfAdditionalNonFilteringFields());
}
}
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/12583
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I5a5111f9c8c4f1ae1c311609a64d7ebd413ca18f
Gerrit-Change-Number: 12583
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <al...@gmail.com>
Gerrit-MessageType: newchange
Change in asterixdb[master]: WIP: option to exclude unknowns in secondary indexes
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Jenkins <je...@fulliautomatix.ics.uci.edu>:
Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/12583 )
Change subject: WIP: option to exclude unknowns in secondary indexes
......................................................................
Patch Set 1: Integration-Tests+1
Integration Tests Successful
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/12300/ : SUCCESS
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/12583
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I5a5111f9c8c4f1ae1c311609a64d7ebd413ca18f
Gerrit-Change-Number: 12583
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <al...@gmail.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-CC: Anon. E. Moose #1000171
Gerrit-Comment-Date: Fri, 30 Jul 2021 05:18:14 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment
Change in asterixdb[master]: WIP: option to exclude unknowns in secondary indexes
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/12583 )
Change subject: WIP: option to exclude unknowns in secondary indexes
......................................................................
Patch Set 1: Contrib-2
Analytics Compatibility Tests Failed
https://cbjenkins.page.link/cz1pnQyWyPJTLpgJA : UNSTABLE
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/12583
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I5a5111f9c8c4f1ae1c311609a64d7ebd413ca18f
Gerrit-Change-Number: 12583
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <al...@gmail.com>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Comment-Date: Fri, 30 Jul 2021 06:40:53 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment