You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@asterixdb.apache.org by mb...@apache.org on 2021/01/13 19:36:39 UTC
[asterixdb] branch master updated (0b0d19a -> 6fe3096)
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git.
from 0b0d19a Merge branch 'gerrit/cheshire-cat'
new 49350fd [NO ISSUE] Refactoring for extension
new 0912956 [NO ISSUE][STO] Allocate Memory Component Index Lazily
new 3b2d625 [NO ISSUE][COMP] Remove accessed dataset tracking
new 1de53b1 [NO ISSUE][API] Ignore body of GET requests
new 0021184 Merge branch 'gerrit/mad-hatter'
new 6fe3096 Merge branch 'gerrit/cheshire-cat'
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../http/server/QueryServiceRequestParameters.java | 30 +++++++------
.../api/http/server/QueryServiceServlet.java | 4 +-
.../asterix/app/function/DatasetRewriter.java | 49 ++++++++++++----------
.../api/get-non-query/get-non-query.1.get.http | 4 +-
.../api/get-non-query/get-non-query.2.get.http | 5 ++-
.../api/get-non-query/get-non-query.3.get.http | 5 ++-
.../api/get-query/get-query.1.get.http | 4 +-
.../api/get-query/get-query.2.get.http | 5 ++-
.../api/get-query/get-query.3.get.http | 5 ++-
.../ignore-body-for-get.1.get.http} | 7 +++-
.../ignore-body-for-get.2.get.http} | 7 +++-
.../ignore-body-for-get.1.regexadm | 1 +
.../ignore-body-for-get.2.regexadm | 1 +
.../test/resources/runtimets/testsuite_sqlpp.xml | 5 +++
.../common/context/GlobalVirtualBufferCache.java | 6 ++-
.../visitor/VariableCheckAndRewriteVisitor.java | 1 -
.../metadata/declared/MetadataProvider.java | 14 -------
.../runtime/evaluators/EvaluatorContext.java | 6 +++
.../operators/std/StreamSelectRuntimeFactory.java | 7 +++-
.../apache/hyracks/http/server/utils/HttpUtil.java | 4 +-
.../common/impls/AbstractLSMMemoryComponent.java | 22 +++++++---
.../impls/AbstractLSMWithBuddyMemoryComponent.java | 18 ++++----
22 files changed, 128 insertions(+), 82 deletions(-)
copy asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/{request-param-validation-400-BAD/request-param-validation-400-BAD.01.post.http => ignore-body-for-get/ignore-body-for-get.1.get.http} (79%)
copy asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/{request-param-validation-400-BAD/request-param-validation-400-BAD.01.post.http => ignore-body-for-get/ignore-body-for-get.2.get.http} (77%)
create mode 100644 asterixdb/asterix-app/src/test/resources/runtimets/results/api/ignore-body-for-get/ignore-body-for-get.1.regexadm
create mode 100644 asterixdb/asterix-app/src/test/resources/runtimets/results/api/ignore-body-for-get/ignore-body-for-get.2.regexadm
[asterixdb] 03/06: [NO ISSUE][STO] Allocate Memory Component Index
Lazily
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit 09129566869073b84ba06dfbf147901d0eefbe38
Author: Murtadha Hubail <mh...@apache.org>
AuthorDate: Fri Jan 8 21:41:49 2021 +0300
[NO ISSUE][STO] Allocate Memory Component Index Lazily
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
- When a memory component is flushed, delay the allocation of
its index until the next activation.
- When the global virtual buffer cache is full, notify the
flush thread to check if any indexes can be flushed.
Change-Id: I2e07d4e857989fb742e1fc0c5e620a7293fcc3e4
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/9463
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <mh...@apache.org>
Reviewed-by: Ali Alsuliman <al...@gmail.com>
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
---
.../common/context/GlobalVirtualBufferCache.java | 6 +++++-
.../common/impls/AbstractLSMMemoryComponent.java | 22 ++++++++++++++++------
.../impls/AbstractLSMWithBuddyMemoryComponent.java | 18 ++++++++++--------
3 files changed, 31 insertions(+), 15 deletions(-)
diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/GlobalVirtualBufferCache.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/GlobalVirtualBufferCache.java
index f772038..a51e13c 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/GlobalVirtualBufferCache.java
+++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/GlobalVirtualBufferCache.java
@@ -199,7 +199,11 @@ public class GlobalVirtualBufferCache implements IVirtualBufferCache, ILifeCycle
@Override
public boolean isFull() {
- return vbc.isFull();
+ boolean full = vbc.isFull();
+ if (full) {
+ checkAndNotifyFlushThread();
+ }
+ return full;
}
@Override
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMMemoryComponent.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMMemoryComponent.java
index 3b6667e..8d37d97 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMMemoryComponent.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMMemoryComponent.java
@@ -35,6 +35,7 @@ import org.apache.logging.log4j.Logger;
public abstract class AbstractLSMMemoryComponent extends AbstractLSMComponent implements ILSMMemoryComponent {
private static final Logger LOGGER = LogManager.getLogger();
+ protected final AtomicBoolean allocated;
private final IVirtualBufferCache vbc;
private final AtomicBoolean isModified;
private int writerCount;
@@ -48,6 +49,7 @@ public abstract class AbstractLSMMemoryComponent extends AbstractLSMComponent im
writerCount = 0;
state = ComponentState.INACTIVE;
isModified = new AtomicBoolean();
+ allocated = new AtomicBoolean();
metadata = new MemoryComponentMetadata();
}
@@ -80,6 +82,9 @@ public abstract class AbstractLSMMemoryComponent extends AbstractLSMComponent im
private void activate() throws HyracksDataException {
if (state == ComponentState.INACTIVE) {
+ if (!allocated.get()) {
+ doAllocate();
+ }
state = ComponentState.READABLE_WRITABLE;
lsmIndex.getIOOperationCallback().recycled(this);
}
@@ -247,10 +252,11 @@ public abstract class AbstractLSMMemoryComponent extends AbstractLSMComponent im
@Override
public void cleanup() throws HyracksDataException {
- getIndex().deactivate();
- getIndex().destroy();
- getIndex().create();
- getIndex().activate();
+ if (allocated.get()) {
+ getIndex().deactivate();
+ getIndex().destroy();
+ allocated.set(false);
+ }
}
@Override
@@ -286,6 +292,7 @@ public abstract class AbstractLSMMemoryComponent extends AbstractLSMComponent im
created = true;
getIndex().activate();
activated = true;
+ allocated.set(true);
} finally {
if (created && !activated) {
getIndex().destroy();
@@ -305,8 +312,11 @@ public abstract class AbstractLSMMemoryComponent extends AbstractLSMComponent im
}
protected void doDeallocate() throws HyracksDataException {
- getIndex().deactivate();
- getIndex().destroy();
+ if (allocated.get()) {
+ getIndex().deactivate();
+ getIndex().destroy();
+ allocated.set(false);
+ }
componentId = null;
}
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMWithBuddyMemoryComponent.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMWithBuddyMemoryComponent.java
index 882b4bd..5ebaf26 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMWithBuddyMemoryComponent.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMWithBuddyMemoryComponent.java
@@ -34,11 +34,11 @@ public abstract class AbstractLSMWithBuddyMemoryComponent extends AbstractLSMMem
@Override
public void cleanup() throws HyracksDataException {
- super.cleanup();
- getBuddyIndex().deactivate();
- getBuddyIndex().destroy();
- getBuddyIndex().create();
- getBuddyIndex().activate();
+ if (allocated.get()) {
+ super.cleanup();
+ getBuddyIndex().deactivate();
+ getBuddyIndex().destroy();
+ }
}
@Override
@@ -50,9 +50,11 @@ public abstract class AbstractLSMWithBuddyMemoryComponent extends AbstractLSMMem
@Override
public void doDeallocate() throws HyracksDataException {
- super.doDeallocate();
- getBuddyIndex().deactivate();
- getBuddyIndex().destroy();
+ if (allocated.get()) {
+ super.doDeallocate();
+ getBuddyIndex().deactivate();
+ getBuddyIndex().destroy();
+ }
}
@Override
[asterixdb] 01/06: [NO ISSUE] Refactoring for extension
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit 49350fdec8202157336279d72e2a6428df169286
Author: Ali Alsuliman <al...@gmail.com>
AuthorDate: Wed Dec 23 10:54:36 2020 -0800
[NO ISSUE] Refactoring for extension
Change-Id: I49e4754b1c329c78b85925ca8af6837512cb15e8
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/9445
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: Michael Blow <mb...@apache.org>
---
.../hyracks/algebricks/runtime/evaluators/EvaluatorContext.java | 6 ++++++
.../runtime/operators/std/StreamSelectRuntimeFactory.java | 7 ++++++-
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/hyracks-fullstack/algebricks/algebricks-runtime/src/main/java/org/apache/hyracks/algebricks/runtime/evaluators/EvaluatorContext.java b/hyracks-fullstack/algebricks/algebricks-runtime/src/main/java/org/apache/hyracks/algebricks/runtime/evaluators/EvaluatorContext.java
index 8d31f6b..901e788 100644
--- a/hyracks-fullstack/algebricks/algebricks-runtime/src/main/java/org/apache/hyracks/algebricks/runtime/evaluators/EvaluatorContext.java
+++ b/hyracks-fullstack/algebricks/algebricks-runtime/src/main/java/org/apache/hyracks/algebricks/runtime/evaluators/EvaluatorContext.java
@@ -46,6 +46,12 @@ public final class EvaluatorContext implements IEvaluatorContext {
this.warningCollector = Objects.requireNonNull(warningCollector);
}
+ public EvaluatorContext(IHyracksTaskContext taskContext, IWarningCollector warningCollector) {
+ this.taskContext = Objects.requireNonNull(taskContext);
+ this.serviceContext = Objects.requireNonNull(taskContext.getJobletContext().getServiceContext());
+ this.warningCollector = Objects.requireNonNull(warningCollector);
+ }
+
@Override
public IServiceContext getServiceContext() {
return serviceContext;
diff --git a/hyracks-fullstack/algebricks/algebricks-runtime/src/main/java/org/apache/hyracks/algebricks/runtime/operators/std/StreamSelectRuntimeFactory.java b/hyracks-fullstack/algebricks/algebricks-runtime/src/main/java/org/apache/hyracks/algebricks/runtime/operators/std/StreamSelectRuntimeFactory.java
index acdc449..5cfebc0 100644
--- a/hyracks-fullstack/algebricks/algebricks-runtime/src/main/java/org/apache/hyracks/algebricks/runtime/operators/std/StreamSelectRuntimeFactory.java
+++ b/hyracks-fullstack/algebricks/algebricks-runtime/src/main/java/org/apache/hyracks/algebricks/runtime/operators/std/StreamSelectRuntimeFactory.java
@@ -34,6 +34,7 @@ import org.apache.hyracks.api.context.IHyracksTaskContext;
import org.apache.hyracks.api.dataflow.value.IMissingWriter;
import org.apache.hyracks.api.dataflow.value.IMissingWriterFactory;
import org.apache.hyracks.api.exceptions.HyracksDataException;
+import org.apache.hyracks.api.exceptions.IWarningCollector;
import org.apache.hyracks.data.std.api.IPointable;
import org.apache.hyracks.data.std.primitive.VoidPointable;
import org.apache.hyracks.dataflow.common.comm.io.ArrayTupleBuilder;
@@ -110,7 +111,7 @@ public class StreamSelectRuntimeFactory extends AbstractOneInputOneOutputRuntime
protected ArrayTupleBuilder missingTupleBuilder;
public StreamSelectRuntime(IHyracksTaskContext ctx, IBinaryBooleanInspector bbi) {
- this.ctx = new EvaluatorContext(ctx);
+ this.ctx = new EvaluatorContext(ctx, initWarningCollector(ctx));
this.bbi = bbi;
}
@@ -165,6 +166,10 @@ public class StreamSelectRuntimeFactory extends AbstractOneInputOneOutputRuntime
}
}
}
+
+ protected IWarningCollector initWarningCollector(IHyracksTaskContext ctx) {
+ return ctx.getWarningCollector();
+ }
}
}
[asterixdb] 05/06: Merge branch 'gerrit/mad-hatter'
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit 00211846ed300c0bda0de3ffb5a8c6f23929e186
Merge: 3b2d625 1de53b1
Author: Michael Blow <mb...@apache.org>
AuthorDate: Tue Jan 12 15:00:16 2021 -0500
Merge branch 'gerrit/mad-hatter'
Change-Id: Iaee9405168f188a913f3e22da70b2df813d4f37d
.../http/server/QueryServiceRequestParameters.java | 30 ++++++++++++----------
.../api/http/server/QueryServiceServlet.java | 4 +--
.../api/get-non-query/get-non-query.1.get.http | 4 +--
.../api/get-non-query/get-non-query.2.get.http | 5 ++--
.../api/get-non-query/get-non-query.3.get.http | 5 ++--
.../api/get-query/get-query.1.get.http | 4 +--
.../api/get-query/get-query.2.get.http | 5 ++--
.../api/get-query/get-query.3.get.http | 5 ++--
.../ignore-body-for-get.1.get.http} | 8 +++---
.../ignore-body-for-get.2.get.http} | 8 +++---
.../ignore-body-for-get.1.regexadm | 1 +
.../ignore-body-for-get.2.regexadm | 1 +
.../test/resources/runtimets/testsuite_sqlpp.xml | 5 ++++
.../apache/hyracks/http/server/utils/HttpUtil.java | 4 ++-
14 files changed, 55 insertions(+), 34 deletions(-)
[asterixdb] 04/06: [NO ISSUE][COMP] Remove accessed dataset tracking
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit 3b2d6255b5d61e232b853e66783f839868e3a8fe
Author: Dmitry Lychagin <dm...@couchbase.com>
AuthorDate: Fri Jan 8 11:45:53 2021 -0800
[NO ISSUE][COMP] Remove accessed dataset tracking
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
- Remove accessed dataset tracking from metadata provider
Change-Id: I0872330f4645d35a7db21c67cd22c255168c2ec9
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/9523
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <mh...@apache.org>
---
.../asterix/app/function/DatasetRewriter.java | 49 ++++++++++++----------
.../visitor/VariableCheckAndRewriteVisitor.java | 1 -
.../metadata/declared/MetadataProvider.java | 14 -------
3 files changed, 28 insertions(+), 36 deletions(-)
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/DatasetRewriter.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/DatasetRewriter.java
index c459547..974cd9e 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/DatasetRewriter.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/DatasetRewriter.java
@@ -71,14 +71,7 @@ public class DatasetRewriter implements IFunctionToDataSourceRewriter, IResultTy
}
MetadataProvider metadataProvider = (MetadataProvider) context.getMetadataProvider();
- Pair<DataverseName, String> datasetReference = FunctionUtil.parseDatasetFunctionArguments(f);
- DataverseName dataverseName = datasetReference.first;
- String datasetName = datasetReference.second;
- Dataset dataset = metadataProvider.findDataset(dataverseName, datasetName);
- if (dataset == null) {
- throw new CompilationException(ErrorCode.UNKNOWN_DATASET_IN_DATAVERSE, unnest.getSourceLocation(),
- datasetName, dataverseName);
- }
+ Dataset dataset = fetchDataset(metadataProvider, f);
DataSourceId dsid = new DataSourceId(dataset.getDataverseName(), dataset.getDatasetName());
List<LogicalVariable> variables = new ArrayList<>();
if (dataset.getDatasetType() == DatasetType.INTERNAL) {
@@ -123,21 +116,35 @@ public class DatasetRewriter implements IFunctionToDataSourceRewriter, IResultTy
@Override
public IAType computeType(ILogicalExpression expression, IVariableTypeEnvironment env, IMetadataProvider<?, ?> mp)
throws AlgebricksException {
- AbstractFunctionCallExpression f = (AbstractFunctionCallExpression) expression;
+ AbstractFunctionCallExpression datasetFnCall = (AbstractFunctionCallExpression) expression;
MetadataProvider metadata = (MetadataProvider) mp;
- Pair<DataverseName, String> datasetInfo = FunctionUtil.parseDatasetFunctionArguments(f);
- DataverseName dataverseName = datasetInfo.first;
- String datasetName = datasetInfo.second;
- Dataset dataset = metadata.findDataset(dataverseName, datasetName);
- if (dataset == null) {
- throw new CompilationException(ErrorCode.UNKNOWN_DATASET_IN_DATAVERSE, f.getSourceLocation(), datasetName,
- dataverseName);
+ Dataset dataset = fetchDataset(metadata, datasetFnCall);
+ IAType type = metadata.findType(dataset.getItemTypeDataverseName(), dataset.getItemTypeName());
+ if (type == null) {
+ throw new CompilationException(ErrorCode.COMPILATION_ERROR, datasetFnCall.getSourceLocation(),
+ "No type for dataset " + dataset.getDatasetName());
}
- String tn = dataset.getItemTypeName();
- IAType t2 = metadata.findType(dataset.getItemTypeDataverseName(), tn);
- if (t2 == null) {
- throw new AlgebricksException("No type for dataset " + datasetName);
+ return type;
+ }
+
+ public static Dataset fetchDataset(MetadataProvider metadataProvider, AbstractFunctionCallExpression datasetFnCall)
+ throws CompilationException {
+ Pair<DataverseName, String> datasetReference = FunctionUtil.parseDatasetFunctionArguments(datasetFnCall);
+ DataverseName dataverseName = datasetReference.first;
+ String datasetName = datasetReference.second;
+ Dataset dataset;
+ try {
+ dataset = metadataProvider.findDataset(dataverseName, datasetName);
+ } catch (CompilationException e) {
+ throw e;
+ } catch (AlgebricksException e) {
+ throw new CompilationException(ErrorCode.COMPILATION_ERROR, e, datasetFnCall.getSourceLocation(),
+ e.getMessage());
+ }
+ if (dataset == null) {
+ throw new CompilationException(ErrorCode.UNKNOWN_DATASET_IN_DATAVERSE, datasetFnCall.getSourceLocation(),
+ datasetName, dataverseName);
}
- return t2;
+ return dataset;
}
}
diff --git a/asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/rewrites/visitor/VariableCheckAndRewriteVisitor.java b/asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/rewrites/visitor/VariableCheckAndRewriteVisitor.java
index cf8469f..291bcd0 100644
--- a/asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/rewrites/visitor/VariableCheckAndRewriteVisitor.java
+++ b/asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/rewrites/visitor/VariableCheckAndRewriteVisitor.java
@@ -159,7 +159,6 @@ public class VariableCheckAndRewriteVisitor extends AbstractSqlppExpressionScopi
if (dataset == null) {
throw createUnresolvableError(dataverseName, datasetName, sourceLoc);
}
- metadataProvider.addAccessedDataset(dataset);
List<Expression> argList = new ArrayList<>(2);
argList.add(new LiteralExpr(new StringLiteral(dataset.getDataverseName().getCanonicalForm())));
argList.add(new LiteralExpr(new StringLiteral(dataset.getDatasetName())));
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
index 075c2a8..e2505b7 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
@@ -22,14 +22,11 @@ import java.io.File;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
-import java.util.Collections;
import java.util.HashMap;
-import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Optional;
-import java.util.Set;
import org.apache.asterix.common.cluster.IClusterStateManager;
import org.apache.asterix.common.config.DatasetConfig.DatasetType;
@@ -171,7 +168,6 @@ public class MetadataProvider implements IMetadataProvider<DataSourceId, String>
private final IFunctionManager functionManager;
private final LockList locks;
private final Map<String, Object> config;
- private final Set<Dataset> txnAccessedDatasets;
private Dataverse defaultDataverse;
private MetadataTransactionContext mdTxnCtx;
@@ -201,7 +197,6 @@ public class MetadataProvider implements IMetadataProvider<DataSourceId, String>
functionManager = ((IFunctionExtensionManager) appCtx.getExtensionManager()).getFunctionManager();
locks = new LockList();
config = new HashMap<>();
- txnAccessedDatasets = new HashSet<>();
}
@SuppressWarnings("unchecked")
@@ -262,7 +257,6 @@ public class MetadataProvider implements IMetadataProvider<DataSourceId, String>
public void setMetadataTxnContext(MetadataTransactionContext mdTxnCtx) {
this.mdTxnCtx = mdTxnCtx;
- txnAccessedDatasets.clear();
}
public MetadataTransactionContext getMetadataTxnContext() {
@@ -1750,14 +1744,6 @@ public class MetadataProvider implements IMetadataProvider<DataSourceId, String>
return appCtx.getCompressionManager();
}
- public void addAccessedDataset(Dataset dataset) {
- txnAccessedDatasets.add(dataset);
- }
-
- public Set<Dataset> getAccessedDatasets() {
- return Collections.unmodifiableSet(txnAccessedDatasets);
- }
-
public void validateDataverseName(DataverseName dataverseName, SourceLocation sourceLoc)
throws AlgebricksException {
int totalLengthUTF8 = 0;
[asterixdb] 02/06: [NO ISSUE][API] Ignore body of GET requests
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit 1de53b1444a7fcb4871d2549c8625e467c7885be
Author: Ali Alsuliman <al...@gmail.com>
AuthorDate: Fri Jan 8 09:14:11 2021 -0800
[NO ISSUE][API] Ignore body of GET requests
- user model changes: yes
- storage format changes: no
- interface changes: no
Change-Id: Ia2db86bab403ef881880388775c00b497955dde8
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/9483
Reviewed-by: Michael Blow <mb...@apache.org>
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
---
.../http/server/QueryServiceRequestParameters.java | 30 ++++++++++++----------
.../api/http/server/QueryServiceServlet.java | 4 +--
.../api/get-non-query/get-non-query.1.get.http | 4 +--
.../api/get-non-query/get-non-query.2.get.http | 5 ++--
.../api/get-non-query/get-non-query.3.get.http | 5 ++--
.../api/get-query/get-query.1.get.http | 4 +--
.../api/get-query/get-query.2.get.http | 5 ++--
.../api/get-query/get-query.3.get.http | 5 ++--
.../ignore-body-for-get.1.get.http} | 8 +++---
.../ignore-body-for-get.2.get.http} | 8 +++---
.../ignore-body-for-get.1.regexadm | 1 +
.../ignore-body-for-get.2.regexadm | 1 +
.../test/resources/runtimets/testsuite_sqlpp.xml | 5 ++++
.../apache/hyracks/http/server/utils/HttpUtil.java | 4 ++-
14 files changed, 55 insertions(+), 34 deletions(-)
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryServiceRequestParameters.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryServiceRequestParameters.java
index 2379e69..06f8325 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryServiceRequestParameters.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryServiceRequestParameters.java
@@ -51,6 +51,8 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.google.common.collect.ImmutableMap;
+import io.netty.handler.codec.http.HttpMethod;
+
public class QueryServiceRequestParameters {
public enum Parameter {
@@ -375,32 +377,36 @@ public class QueryServiceRequestParameters {
throws IOException {
setHost(servlet.host(request));
setPath(servlet.servletPath(request));
- String contentType = HttpUtil.getContentTypeOnly(request);
setOptionalParams(optionalParams);
try {
- if (HttpUtil.ContentType.APPLICATION_JSON.equals(contentType)) {
- setParamFromJSON(request, optionalParams);
+ if (useRequestParameters(request)) {
+ setFromRequestParameters(request);
} else {
- setParamFromRequest(request, optionalParams);
+ setFromRequestBody(request);
}
} catch (JsonParseException | JsonMappingException e) {
throw new RuntimeDataException(ErrorCode.INVALID_REQ_JSON_VAL);
}
}
- private void setParamFromJSON(IServletRequest request, Map<String, String> optionalParameters) throws IOException {
+ private boolean useRequestParameters(IServletRequest request) {
+ String contentType = HttpUtil.getContentTypeOnly(request);
+ HttpMethod method = request.getHttpRequest().method();
+ return HttpMethod.GET.equals(method) || !HttpUtil.ContentType.APPLICATION_JSON.equals(contentType);
+ }
+
+ private void setFromRequestBody(IServletRequest request) throws IOException {
JsonNode jsonRequest = OBJECT_MAPPER.readTree(HttpUtil.getRequestBody(request));
setParams(jsonRequest, request.getHeader(HttpHeaders.ACCEPT), QueryServiceRequestParameters::getParameter);
setStatementParams(getOptStatementParameters(jsonRequest, jsonRequest.fieldNames(), JsonNode::get, v -> v));
- setJsonOptionalParameters(jsonRequest, optionalParameters);
+ setExtraParams(jsonRequest);
}
- private void setParamFromRequest(IServletRequest request, Map<String, String> optionalParameters)
- throws IOException {
+ private void setFromRequestParameters(IServletRequest request) throws IOException {
setParams(request, request.getHeader(HttpHeaders.ACCEPT), IServletRequest::getParameter);
setStatementParams(getOptStatementParameters(request, request.getParameterNames().iterator(),
IServletRequest::getParameter, OBJECT_MAPPER::readTree));
- setOptionalParameters(request, optionalParameters);
+ setExtraParams(request);
}
private <Req> void setParams(Req req, String acceptHeader, BiFunction<Req, String, String> valGetter)
@@ -431,13 +437,11 @@ public class QueryServiceRequestParameters {
setSignature(parseBoolean(req, Parameter.SIGNATURE.str(), valGetter, isSignature()));
}
- protected void setJsonOptionalParameters(JsonNode jsonRequest, Map<String, String> optionalParameters)
- throws HyracksDataException {
+ protected void setExtraParams(JsonNode jsonRequest) throws HyracksDataException {
// allows extensions to set extra parameters
}
- protected void setOptionalParameters(IServletRequest request, Map<String, String> optionalParameters)
- throws HyracksDataException {
+ protected void setExtraParams(IServletRequest request) throws HyracksDataException {
// allows extensions to set extra parameters
}
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryServiceServlet.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryServiceServlet.java
index 440b351..254bdae 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryServiceServlet.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryServiceServlet.java
@@ -135,12 +135,12 @@ public class QueryServiceServlet extends AbstractQueryApiServlet {
}
@Override
- protected void get(IServletRequest request, IServletResponse response) throws IOException {
+ protected final void get(IServletRequest request, IServletResponse response) throws IOException {
handleRequest(request, response, true);
}
@Override
- protected void post(IServletRequest request, IServletResponse response) throws IOException {
+ protected final void post(IServletRequest request, IServletResponse response) throws IOException {
handleRequest(request, response, false);
}
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.1.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.1.get.http
index e6aff85..bbacf58 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.1.get.http
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.1.get.http
@@ -20,5 +20,5 @@
* Description: testing passing a non-query statement using GET
* Result: failure
*/
-/query/service
---body={"statement": "CREATE DATAVERSE dv1;"}
\ No newline at end of file
+# param statement=CREATE DATAVERSE dv1
+/query/service
\ No newline at end of file
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.2.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.2.get.http
index 5dc6ec7..71153c9 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.2.get.http
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.2.get.http
@@ -20,5 +20,6 @@
* Description: testing passing a non-query statement using GET
* Result: failure
*/
-/query/service
---body={"statement": "CREATE TYPE t1 AS {id: int};", "readonly": true}
\ No newline at end of file
+# param statement=CREATE TYPE t1 AS {id: int}
+# param readonly=true
+/query/service
\ No newline at end of file
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.3.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.3.get.http
index 4f08b95..ba3c68e 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.3.get.http
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.3.get.http
@@ -20,5 +20,6 @@
* Description: testing passing a non-query statement using GET
* Result: failure
*/
-/query/service
---body={"statement": "CREATE FUNCTION foo(){1};", "readonly": false}
\ No newline at end of file
+# param statement=CREATE FUNCTION foo(){1}
+# param readonly=false
+/query/service
\ No newline at end of file
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-query/get-query.1.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-query/get-query.1.get.http
index 4743d9f..f17053a 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-query/get-query.1.get.http
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-query/get-query.1.get.http
@@ -21,5 +21,5 @@
* Result: success
*/
-- extractresult=true
-/query/service
---body={"statement": "FROM Metadata.`Dataverse` v WHERE v.DataverseName = 'Metadata' SELECT VALUE v.DataverseName;"}
\ No newline at end of file
+# param statement=FROM Metadata.`Dataverse` v WHERE v.DataverseName = 'Metadata' SELECT VALUE v.DataverseName
+/query/service
\ No newline at end of file
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-query/get-query.2.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-query/get-query.2.get.http
index 8995d4f..57dc1ca 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-query/get-query.2.get.http
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-query/get-query.2.get.http
@@ -21,5 +21,6 @@
* Result: success
*/
-- extractresult=true
-/query/service
---body={"statement": "FROM Metadata.`Dataverse` v WHERE v.DataverseName = 'Metadata' SELECT VALUE v.DataverseName;", "readonly": false}
\ No newline at end of file
+# param statement=FROM Metadata.`Dataverse` v WHERE v.DataverseName = 'Metadata' SELECT VALUE v.DataverseName
+# param readonly=false
+/query/service
\ No newline at end of file
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-query/get-query.3.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-query/get-query.3.get.http
index f821428..14c90a0 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-query/get-query.3.get.http
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-query/get-query.3.get.http
@@ -21,5 +21,6 @@
* Result: success
*/
-- extractresult=true
-/query/service
---body={"statement": "FROM Metadata.`Dataverse` v WHERE v.DataverseName = 'Metadata' SELECT VALUE v.DataverseName;", "readonly": true}
\ No newline at end of file
+# param statement=FROM Metadata.`Dataverse` v WHERE v.DataverseName = 'Metadata' SELECT VALUE v.DataverseName
+# param readonly=true
+/query/service
\ No newline at end of file
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.3.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/ignore-body-for-get/ignore-body-for-get.1.get.http
similarity index 78%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.3.get.http
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/ignore-body-for-get/ignore-body-for-get.1.get.http
index 4f08b95..ff8b376 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.3.get.http
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/ignore-body-for-get/ignore-body-for-get.1.get.http
@@ -17,8 +17,10 @@
* under the License.
*/
/*
- * Description: testing passing a non-query statement using GET
- * Result: failure
+ * Description: testing that body of GET requests are ignored
+ * Result: failure (missing the required "statement" parameter since it will be ignored)
*/
+// statuscode 400
+-- requesttype=application/json
/query/service
---body={"statement": "CREATE FUNCTION foo(){1};", "readonly": false}
\ No newline at end of file
+--body={"statement": "SELECT 1;"}
\ No newline at end of file
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.3.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/ignore-body-for-get/ignore-body-for-get.2.get.http
similarity index 76%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.3.get.http
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/ignore-body-for-get/ignore-body-for-get.2.get.http
index 4f08b95..79771fb 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/get-non-query/get-non-query.3.get.http
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/api/ignore-body-for-get/ignore-body-for-get.2.get.http
@@ -17,8 +17,10 @@
* under the License.
*/
/*
- * Description: testing passing a non-query statement using GET
- * Result: failure
+ * Description: testing that body of GET requests are ignored
+ * Result: failure (missing the required "statement" parameter since it will be ignored)
*/
+// statuscode 400
+-- requesttype=application/x-www-form-urlencoded
/query/service
---body={"statement": "CREATE FUNCTION foo(){1};", "readonly": false}
\ No newline at end of file
+--body={"statement": "SELECT 1;"}
\ No newline at end of file
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/api/ignore-body-for-get/ignore-body-for-get.1.regexadm b/asterixdb/asterix-app/src/test/resources/runtimets/results/api/ignore-body-for-get/ignore-body-for-get.1.regexadm
new file mode 100644
index 0000000..73efe45
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/api/ignore-body-for-get/ignore-body-for-get.1.regexadm
@@ -0,0 +1 @@
+.*No statement provided.*
\ No newline at end of file
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/api/ignore-body-for-get/ignore-body-for-get.2.regexadm b/asterixdb/asterix-app/src/test/resources/runtimets/results/api/ignore-body-for-get/ignore-body-for-get.2.regexadm
new file mode 100644
index 0000000..73efe45
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/api/ignore-body-for-get/ignore-body-for-get.2.regexadm
@@ -0,0 +1 @@
+.*No statement provided.*
\ No newline at end of file
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml
index ff37ac5..c8e5bfa 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml
@@ -127,6 +127,11 @@
<expected-error>TYPE_DECL statement is not supported in read-only mode</expected-error>
</compilation-unit>
</test-case>
+ <test-case FilePath="api">
+ <compilation-unit name="ignore-body-for-get">
+ <output-dir compare="Text">ignore-body-for-get</output-dir>
+ </compilation-unit>
+ </test-case>
</test-group>
<test-group name="flwor">
<test-case FilePath="flwor">
diff --git a/hyracks-fullstack/hyracks/hyracks-http/src/main/java/org/apache/hyracks/http/server/utils/HttpUtil.java b/hyracks-fullstack/hyracks/hyracks-http/src/main/java/org/apache/hyracks/http/server/utils/HttpUtil.java
index f9ada92..1f0d3bc 100644
--- a/hyracks-fullstack/hyracks/hyracks-http/src/main/java/org/apache/hyracks/http/server/utils/HttpUtil.java
+++ b/hyracks-fullstack/hyracks/hyracks-http/src/main/java/org/apache/hyracks/http/server/utils/HttpUtil.java
@@ -53,6 +53,7 @@ import io.netty.handler.codec.http.DefaultHttpResponse;
import io.netty.handler.codec.http.FullHttpRequest;
import io.netty.handler.codec.http.HttpHeaderNames;
import io.netty.handler.codec.http.HttpHeaderValues;
+import io.netty.handler.codec.http.HttpMethod;
import io.netty.handler.codec.http.HttpRequest;
import io.netty.handler.codec.http.HttpScheme;
import io.netty.util.AsciiString;
@@ -76,7 +77,8 @@ public class HttpUtil {
public static IServletRequest toServletRequest(ChannelHandlerContext ctx, FullHttpRequest request,
HttpScheme scheme) {
return ContentType.APPLICATION_X_WWW_FORM_URLENCODED.equals(getContentTypeOnly(request))
- ? FormUrlEncodedRequest.create(ctx, request, scheme) : BaseRequest.create(ctx, request, scheme);
+ && !HttpMethod.GET.equals(request.method()) ? FormUrlEncodedRequest.create(ctx, request, scheme)
+ : BaseRequest.create(ctx, request, scheme);
}
public static String getContentTypeOnly(IServletRequest request) {
[asterixdb] 06/06: Merge branch 'gerrit/cheshire-cat'
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mblow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit 6fe3096d0459d1ed39a075962e9498f0f6b5c2c5
Merge: 0b0d19a 0021184
Author: Michael Blow <mb...@apache.org>
AuthorDate: Wed Jan 13 12:17:20 2021 -0500
Merge branch 'gerrit/cheshire-cat'
Change-Id: I988669d7ba812e637511296531a5b6c08babb40d
.../http/server/QueryServiceRequestParameters.java | 30 +++++++------
.../api/http/server/QueryServiceServlet.java | 4 +-
.../asterix/app/function/DatasetRewriter.java | 49 ++++++++++++----------
.../api/get-non-query/get-non-query.1.get.http | 4 +-
.../api/get-non-query/get-non-query.2.get.http | 5 ++-
.../api/get-non-query/get-non-query.3.get.http | 5 ++-
.../api/get-query/get-query.1.get.http | 4 +-
.../api/get-query/get-query.2.get.http | 5 ++-
.../api/get-query/get-query.3.get.http | 5 ++-
.../ignore-body-for-get.1.get.http} | 8 ++--
.../ignore-body-for-get.2.get.http} | 8 ++--
.../ignore-body-for-get.1.regexadm | 1 +
.../ignore-body-for-get.2.regexadm | 1 +
.../test/resources/runtimets/testsuite_sqlpp.xml | 5 +++
.../common/context/GlobalVirtualBufferCache.java | 6 ++-
.../visitor/VariableCheckAndRewriteVisitor.java | 1 -
.../metadata/declared/MetadataProvider.java | 14 -------
.../runtime/evaluators/EvaluatorContext.java | 6 +++
.../operators/std/StreamSelectRuntimeFactory.java | 7 +++-
.../apache/hyracks/http/server/utils/HttpUtil.java | 4 +-
.../common/impls/AbstractLSMMemoryComponent.java | 22 +++++++---
.../impls/AbstractLSMWithBuddyMemoryComponent.java | 18 ++++----
22 files changed, 126 insertions(+), 86 deletions(-)