You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@asterixdb.apache.org by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu> on 2021/04/30 15:57:14 UTC

Change in asterixdb[cheshire-cat]: [ASTERIXDB-2896] Increase UDF argument buffer size

From Ian Maxon <im...@uci.edu>:

Ian Maxon has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11243 )


Change subject: [ASTERIXDB-2896] Increase UDF argument buffer size
......................................................................

[ASTERIXDB-2896] Increase UDF argument buffer size

-user model changes: no
-storage format changes: no
-interface changes: yes

Details:

Bump buffer sizes in Python IPC to 1MB for individual
arguments, and in the case of batching, match the
buffer size of the Hyracks IPC layer for deserialization.

Change-Id: If847ac3b09406d1e9e6a976a7e0193b6e81bcc8b
---
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalScalarPythonFunctionEvaluator.java
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/operators/ExternalAssignBatchRuntimeFactory.java
2 files changed, 15 insertions(+), 3 deletions(-)



  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/43/11243/1

diff --git a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalScalarPythonFunctionEvaluator.java b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalScalarPythonFunctionEvaluator.java
index e664f47..fc91bdc 100644
--- a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalScalarPythonFunctionEvaluator.java
+++ b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalScalarPythonFunctionEvaluator.java
@@ -41,6 +41,7 @@
 import org.apache.hyracks.data.std.primitive.VoidPointable;
 import org.apache.hyracks.data.std.util.ArrayBackedValueStorage;
 import org.apache.hyracks.dataflow.common.data.accessors.IFrameTupleReference;
+import org.apache.hyracks.util.StorageUtil;
 import org.msgpack.core.MessagePack;
 import org.msgpack.core.MessageUnpacker;
 import org.msgpack.core.buffer.ArrayBufferInput;
@@ -62,6 +63,9 @@
 
     private long fnId;
 
+    private static final int MAX_ARGUMENT_SZ = 1024 * 1024;
+    private static final int HEADER_FUDGE = 64;
+
     ExternalScalarPythonFunctionEvaluator(IExternalFunctionInfo finfo, IScalarEvaluatorFactory[] args,
             IAType[] argTypes, IEvaluatorContext ctx, SourceLocation sourceLoc) throws HyracksDataException {
         super(finfo, args, argTypes, ctx);
@@ -77,8 +81,8 @@
             argValues[i] = VoidPointable.FACTORY.createPointable();
         }
         //TODO: these should be dynamic
-        this.argHolder = ByteBuffer.wrap(new byte[Short.MAX_VALUE * 2]);
-        this.outputWrapper = ByteBuffer.wrap(new byte[Short.MAX_VALUE * 2]);
+        this.argHolder = ByteBuffer.wrap(new byte[MAX_ARGUMENT_SZ + HEADER_FUDGE]);
+        this.outputWrapper = ByteBuffer.wrap(new byte[MAX_ARGUMENT_SZ + HEADER_FUDGE]);
         this.evaluatorContext = ctx;
         this.sourceLocation = sourceLoc;
         this.unpackerInput = new ArrayBufferInput(new byte[0]);
diff --git a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/operators/ExternalAssignBatchRuntimeFactory.java b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/operators/ExternalAssignBatchRuntimeFactory.java
index 39e480a..20d19f1 100644
--- a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/operators/ExternalAssignBatchRuntimeFactory.java
+++ b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/operators/ExternalAssignBatchRuntimeFactory.java
@@ -61,6 +61,9 @@
     private final IExternalFunctionDescriptor[] fnDescs;
     private final int[][] fnArgColumns;
 
+    private static final int MAX_ARGUMENT_SZ = 1024 * 1024;
+    private static final int HEADER_FUDGE = 64;
+
     public ExternalAssignBatchRuntimeFactory(int[] outColumns, IExternalFunctionDescriptor[] fnDescs,
             int[][] fnArgColumns, int[] projectionList) {
         super(projectionList);
@@ -110,7 +113,7 @@
                 }
                 argHolders = new ArrayList<>(fnArgColumns.length);
                 for (int i = 0; i < fnArgColumns.length; i++) {
-                    argHolders.add(ctx.allocateFrame());
+                    argHolders.add(ctx.allocateFrame(MAX_ARGUMENT_SZ + HEADER_FUDGE));
                 }
                 outputWrapper = ctx.allocateFrame();
                 nullCalls = new ATypeTag[argHolders.size()][0];
@@ -262,6 +265,11 @@
                                 outputWrapper.clear();
                                 outputWrapper.position(0);
                                 Pair<ByteBuffer, Counter> result = batchResults.get(k);
+                                if(result.getFirst() != null){
+                                    if(result.getFirst().capacity() > outputWrapper.capacity()){
+                                        outputWrapper = ctx.allocateFrame(result.getFirst().capacity());
+                                    }
+                                }
                                 int start = outputWrapper.arrayOffset();
                                 ATypeTag functionCalled = nullCalls[k][i];
                                 if (functionCalled == ATypeTag.TYPE) {

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11243
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-Project: asterixdb
Gerrit-Branch: cheshire-cat
Gerrit-Change-Id: If847ac3b09406d1e9e6a976a7e0193b6e81bcc8b
Gerrit-Change-Number: 11243
Gerrit-PatchSet: 1
Gerrit-Owner: Ian Maxon <im...@uci.edu>
Gerrit-MessageType: newchange