You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "vibhatha (via GitHub)" <gi...@apache.org> on 2024/06/04 23:45:57 UTC
[PR] GH-40942: [Java] Implement C Data Interface for StringView [WIP] [arrow]
vibhatha opened a new pull request, #41967:
URL: https://github.com/apache/arrow/pull/41967
## DO NOT REVIEW WIP
### Rationale for this change
Recent inclusion of `Utf8View` and `BinaryView` support to Java also requires adding C Data interface for integrating it with other systems.
### What changes are included in this PR?
- [X] Adding core functionality for C Data interface for `Utf8View` and `BinaryView`
- [X] Adding `RoundtripTest`
- [X] Adding `StreamingTest`
### Are these changes tested?
Yes, with new tests.
### Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1635865070
##########
java/vector/src/main/java/org/apache/arrow/vector/FieldVector.java:
##########
@@ -56,6 +56,13 @@ public interface FieldVector extends ValueVector {
*/
List<ArrowBuf> getFieldBuffers();
+ /**
+ * Retrieves the export buffer count for the C Data Interface.
+ *
+ * @return the number of variadic buffers
+ */
+ int getExportedCDataBuffers();
Review Comment:
nit: let's name this with Count so it's clear
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
+ // 0th buffer is validity buffer
+ // 1st buffer is view buffer
+ // 2nd buffer onwards are variadic buffer
+ // N-1 (this.buffers.length - 1) buffer is variadic size buffer
+ final int variadicBufferReadOffset = 2;
+ variadicSizeBuffer.getReferenceManager().retain();
Review Comment:
Why do we need to retain this buffer?
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
Review Comment:
Why does the view buffer need a refcount but not the validity buffer?
##########
java/c/src/main/java/org/apache/arrow/c/ArrayExporter.java:
##########
@@ -91,7 +91,8 @@ void export(ArrowArray array, FieldVector vector, DictionaryProvider dictionaryP
if (buffers != null) {
data.buffers = new ArrayList<>(buffers.size());
Review Comment:
Shouldn't this also use the new buffer count? And we shouldn't call getFieldBuffers above at all anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1635891976
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
+ // 0th buffer is validity buffer
+ // 1st buffer is view buffer
+ // 2nd buffer onwards are variadic buffer
+ // N-1 (this.buffers.length - 1) buffer is variadic size buffer
+ final int variadicBufferReadOffset = 2;
+ variadicSizeBuffer.getReferenceManager().retain();
Review Comment:
It's only used during import. We should discard it as soon as we're done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1634876338
##########
java/c/src/test/python/integration_tests.py:
##########
@@ -183,6 +190,55 @@ def round_trip_reader(self, schema, batches):
def test_string_array(self):
self.round_trip_array(lambda: pa.array([None, "a", "bb", "ccc"]))
+ def test_stringview_array(self):
+ # with nulls short strings
+ self.round_trip_array(lambda: pa.array([None, "a", "bb", "c"], type=pa.string_view()))
+ # with nulls long and strings
+ self.round_trip_array(lambda: pa.array([None, "a", "bb"*10, "c"*13], type=pa.string_view()))
+ # without nulls short strings
+ self.round_trip_array(lambda: pa.array(["a", "bb", "c"], type=pa.string_view()))
+ # without nulls long and strings
+ self.round_trip_array(lambda: pa.array(["a", "bb"*10, "c"*13], type=pa.string_view()))
+ # with multiple data buffers
+ data = []
+ for i in range(1, 501):
+ s = ''.join(str(j) for j in range(i))
+ data.append(s)
+ self.round_trip_array(lambda: pa.array(data, type=pa.string_view()))
Review Comment:
Ah that's more neater.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1632921154
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +210,38 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ try (ArrowBuf view = importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ List<ArrowBuf> buffers = new ArrayList<>();
+ view.getReferenceManager().retain();
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ // 0th buffer is validity buffer
+ // 1st buffer is view buffer
+ // 2nd buffer onwards are variadic buffer
+ // N-1 (this.buffers.length - 1) buffer is variadic size buffer
+ final int variadicBufferReadOffset = 2;
+ try (ArrowBuf variadicSizeBufferPrime = importBuffer(type, variadicSizeBufferIndex,
+ variadicSizeBufferCapacity)) {
+ variadicSizeBufferPrime.getReferenceManager().retain();
Review Comment:
Mmm... I thought the try-with-resource would be enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1632505649
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +210,38 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ try (ArrowBuf view = importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ List<ArrowBuf> buffers = new ArrayList<>();
+ view.getReferenceManager().retain();
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ // 0th buffer is validity buffer
+ // 1st buffer is view buffer
+ // 2nd buffer onwards are variadic buffer
+ // N-1 (this.buffers.length - 1) buffer is variadic size buffer
+ final int variadicBufferReadOffset = 2;
+ try (ArrowBuf variadicSizeBufferPrime = importBuffer(type, variadicSizeBufferIndex,
+ variadicSizeBufferCapacity)) {
+ variadicSizeBufferPrime.getReferenceManager().retain();
Review Comment:
Hmm, probably yes, let me check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1630595193
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -117,10 +121,19 @@ private ArrowBuf importOffsets(ArrowType type, long bytesPerSlot) {
return importBuffer(type, 1, capacity);
}
+ private ArrowBuf importView(ArrowType type) {
+ final long capacity = (long) fieldNode.getLength() * BaseVariableWidthViewVector.ELEMENT_SIZE;
+ return importBuffer(type, 1, capacity);
+ }
Review Comment:
Why do we need this vs just using importFixedBytes?
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -117,10 +121,19 @@ private ArrowBuf importOffsets(ArrowType type, long bytesPerSlot) {
return importBuffer(type, 1, capacity);
}
+ private ArrowBuf importView(ArrowType type) {
+ final long capacity = (long) fieldNode.getLength() * BaseVariableWidthViewVector.ELEMENT_SIZE;
+ return importBuffer(type, 1, capacity);
+ }
+
private ArrowBuf importData(ArrowType type, long capacity) {
return importBuffer(type, 2, capacity);
}
+ private ArrowBuf importViewData(ArrowType type, int index, long capacity) {
+ return importBuffer(type, index, capacity);
+ }
Review Comment:
I don't think a pure alias helps readability here
##########
java/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java:
##########
@@ -1667,4 +1667,23 @@ protected void getData(int index, ReusableBuffer<?> buffer) {
public <OUT, IN> OUT accept(VectorVisitor<OUT, IN> visitor, IN value) {
return visitor.visit(this, value);
}
+
+ @Override
+ public void exportCDataBuffers(List<ArrowBuf> buffers, ArrowBuf buffersPtr, long nullValue) {
+ exportBuffer(validityBuffer, buffers, buffersPtr, nullValue, true);
+
+ if (viewBuffer.capacity() == 0) {
+ // Empty view buffer is allowed here.
+ // We set `retain = false` to explicitly not increase the ref count for the exported buffer.
+ // The ref count of the newly created buffer (i.e., 1) already represents the usage
+ // of the imported side.
+ exportBuffer(allocator.buffer(INITIAL_BYTE_COUNT), buffers, buffersPtr, nullValue, false);
Review Comment:
Why can't we export an empty buffer? That is allowed by the spec
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +223,48 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ try (ArrowBuf view = importView(type)) {
+ List<ArrowBuf> buffers = new ArrayList<>();
+ view.getReferenceManager().retain();
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ final int elementSize = BaseVariableWidthViewVector.ELEMENT_SIZE;
+ final int lengthWidth = BaseVariableWidthViewVector.LENGTH_WIDTH;
+ final int prefixWidth = BaseVariableWidthViewVector.PREFIX_WIDTH;
+ final int lengthPrefixWidth = lengthWidth + prefixWidth;
+ // Map to store the data buffer index and the total length of data in that buffer
+ Map<Integer, Long> dataBufferInfo = new HashMap<>();
+ for (int i = 0; i < fieldNode.getLength(); i++) {
+ final int length = view.getInt((long) i * elementSize);
+ if (length > BaseVariableWidthViewVector.INLINE_SIZE) {
+ checkState(maybeValidityBuffer != null,
+ "Validity buffer is required for data of type " + type);
+ if (BitVectorHelper.get(maybeValidityBuffer, i) == 1) {
+ final int bufferIndex =
+ view.getInt(((long) i * elementSize) + lengthPrefixWidth);
+ if (dataBufferInfo.containsKey(bufferIndex)) {
+ dataBufferInfo.compute(bufferIndex, (key, value) -> value != null ? value + (long) length : 0);
+ } else {
+ dataBufferInfo.put(bufferIndex, (long) length);
+ }
+ }
+ }
+ }
Review Comment:
I'm not sure you could do much better? Other than just allocating a static long[] array of lengths since you should know the number of buffers already
##########
java/c/src/test/java/org/apache/arrow/c/StreamTest.java:
##########
@@ -132,6 +134,80 @@ public void roundtripStrings() throws Exception {
}
}
+ @Test
+ public void roundtripStringViews() throws Exception {
+ final Schema schema = new Schema(Arrays.asList(Field.nullable("ints", new ArrowType.Int(32, true)),
+ Field.nullable("string_views", new ArrowType.Utf8View())));
+ final List<ArrowRecordBatch> batches = new ArrayList<>();
+ try (final VectorSchemaRoot root = VectorSchemaRoot.create(schema, allocator)) {
+ final IntVector ints = (IntVector) root.getVector(0);
+ final ViewVarCharVector strs = (ViewVarCharVector) root.getVector(1);
+ VectorUnloader unloader = new VectorUnloader(root);
+
+ root.allocateNew();
+ ints.setSafe(0, 1);
+ ints.setSafe(1, 2);
+ ints.setSafe(2, 4);
+ ints.setSafe(3, 8);
+ strs.setSafe(0, "".getBytes(StandardCharsets.UTF_8));
+ strs.setSafe(1, "a".getBytes(StandardCharsets.UTF_8));
+ strs.setSafe(2, "bc1234567890bc".getBytes(StandardCharsets.UTF_8));
+ strs.setSafe(3, "defg1234567890defg".getBytes(StandardCharsets.UTF_8));
+ root.setRowCount(4);
+ batches.add(unloader.getRecordBatch());
+
+ root.allocateNew();
+ ints.setSafe(0, 1);
+ ints.setNull(1);
+ ints.setSafe(2, 4);
+ ints.setNull(3);
+ strs.setSafe(0, "".getBytes(StandardCharsets.UTF_8));
+ strs.setNull(1);
+ strs.setSafe(2, "bc1234567890bc".getBytes(StandardCharsets.UTF_8));
+ strs.setNull(3);
+ root.setRowCount(4);
+ batches.add(unloader.getRecordBatch());
+ roundtrip(schema, batches);
+ }
+ }
+
+ @Test
+ public void roundtripBinaryViews() throws Exception {
+ final Schema schema = new Schema(Arrays.asList(Field.nullable("ints", new ArrowType.Int(32, true)),
+ Field.nullable("binary_views", new ArrowType.BinaryView())));
+ final List<ArrowRecordBatch> batches = new ArrayList<>();
+ try (final VectorSchemaRoot root = VectorSchemaRoot.create(schema, allocator)) {
+ final IntVector ints = (IntVector) root.getVector(0);
+ final ViewVarBinaryVector strs = (ViewVarBinaryVector) root.getVector(1);
+ VectorUnloader unloader = new VectorUnloader(root);
+
+ root.allocateNew();
+ ints.setSafe(0, 1);
+ ints.setSafe(1, 2);
+ ints.setSafe(2, 4);
+ ints.setSafe(3, 8);
+ strs.setSafe(0, "".getBytes(StandardCharsets.UTF_8));
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1632441720
##########
java/c/src/test/python/integration_tests.py:
##########
Review Comment:
CC @jorisvandenbossche perhaps?
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +210,38 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ try (ArrowBuf view = importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
Review Comment:
Why is view being treated specially here? Shouldn't it be the size buffer?
##########
java/c/src/main/java/org/apache/arrow/c/ArrayExporter.java:
##########
@@ -96,8 +97,34 @@ void export(ArrowArray array, FieldVector vector, DictionaryProvider dictionaryP
}
if (buffers != null) {
+ final long bufferSize = buffers.size();
+ /*
+ * For Variadic types, an additional buffer is kept to store
+ * the size of each variadic buffer since that information
+ * cannot be retrieved in the import component.
+ * Here, the dataBufferReqCount is calculated to determine
+ * the additional number of buffers required.
+ * Also note that if the bufferSize is greater than 2, it means
+ * there is one or more data buffers.
+ * Thus, the dataBufferReqCount is set to 1 to get additional buffer
+ * for to store variadic size buffer.
+ * If it is not the case, the dataBuffer is not present.
+ * According to the spec and C Data interface in C++, there must be
+ * at least 3 data buffers present at the import component.
+ * Thus, the dataBufferReqCount is set to 2 to get additional buffer
+ * for empty dataBuffer and the variadic size buffer.
+ */
+ int dataBufferReqCount = 0;
+ if (vector instanceof BaseVariableWidthViewVector) {
Review Comment:
Carving out special cases all over the place means that the implementation is probably wrong. In this case, it looks like getFieldBuffers is too specialized to IPC/the previous assumption that IPC and C Data used the same buffers is now wrong.
I think for C Data, it would be better to have one new method that simply gives the buffer count for ArrayExporter to allocate the buffers pointers, and do any type specific logic during the export.
##########
java/c/src/test/python/integration_tests.py:
##########
@@ -183,6 +190,38 @@ def round_trip_reader(self, schema, batches):
def test_string_array(self):
self.round_trip_array(lambda: pa.array([None, "a", "bb", "ccc"]))
+ def test_stringview_array(self):
Review Comment:
I'd like to see empty arrays, empty strings, nulls in different positions, and actual binary values tested
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +210,38 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ try (ArrowBuf view = importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ List<ArrowBuf> buffers = new ArrayList<>();
+ view.getReferenceManager().retain();
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ // 0th buffer is validity buffer
+ // 1st buffer is view buffer
+ // 2nd buffer onwards are variadic buffer
+ // N-1 (this.buffers.length - 1) buffer is variadic size buffer
+ final int variadicBufferReadOffset = 2;
+ try (ArrowBuf variadicSizeBufferPrime = importBuffer(type, variadicSizeBufferIndex,
+ variadicSizeBufferCapacity)) {
+ variadicSizeBufferPrime.getReferenceManager().retain();
Review Comment:
And shouldn't we discard the variadic size buffer at the end, since we don't need it anymore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1632512141
##########
java/c/src/main/java/org/apache/arrow/c/ArrayExporter.java:
##########
@@ -96,8 +97,34 @@ void export(ArrowArray array, FieldVector vector, DictionaryProvider dictionaryP
}
if (buffers != null) {
+ final long bufferSize = buffers.size();
+ /*
+ * For Variadic types, an additional buffer is kept to store
+ * the size of each variadic buffer since that information
+ * cannot be retrieved in the import component.
+ * Here, the dataBufferReqCount is calculated to determine
+ * the additional number of buffers required.
+ * Also note that if the bufferSize is greater than 2, it means
+ * there is one or more data buffers.
+ * Thus, the dataBufferReqCount is set to 1 to get additional buffer
+ * for to store variadic size buffer.
+ * If it is not the case, the dataBuffer is not present.
+ * According to the spec and C Data interface in C++, there must be
+ * at least 3 data buffers present at the import component.
+ * Thus, the dataBufferReqCount is set to 2 to get additional buffer
+ * for empty dataBuffer and the variadic size buffer.
+ */
+ int dataBufferReqCount = 0;
+ if (vector instanceof BaseVariableWidthViewVector) {
Review Comment:
I couldn't agree more. I was hesitant to change the C Data interface as it is a public API.
But some thoughts...
In `Data.java` we call the `exportVector(...)` which ultimately calls `ArrowExporter.exprt(...)`. And since we recently updated `ArrowRecordBatch` with `variadicBufferCounts` would it be okay/feasible if we just pass it here by introducing a new function
```
public static void exportVector(BufferAllocator allocator, FieldVector vector, DictionaryProvider provider,
ArrowArray out, ArrowSchema outSchema, List<Integer> variadicBufferCounts) {
....
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1635937178
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
+ // 0th buffer is validity buffer
+ // 1st buffer is view buffer
+ // 2nd buffer onwards are variadic buffer
+ // N-1 (this.buffers.length - 1) buffer is variadic size buffer
+ final int variadicBufferReadOffset = 2;
+ variadicSizeBuffer.getReferenceManager().retain();
Review Comment:
There's no need to use try-with-resources
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1644368306
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
Review Comment:
I removed the try-with-resource
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1633040619
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +210,38 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ try (ArrowBuf view = importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
Review Comment:
I updated, is it okay?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1630797265
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -117,10 +121,19 @@ private ArrowBuf importOffsets(ArrowType type, long bytesPerSlot) {
return importBuffer(type, 1, capacity);
}
+ private ArrowBuf importView(ArrowType type) {
+ final long capacity = (long) fieldNode.getLength() * BaseVariableWidthViewVector.ELEMENT_SIZE;
+ return importBuffer(type, 1, capacity);
+ }
+
private ArrowBuf importData(ArrowType type, long capacity) {
return importBuffer(type, 2, capacity);
}
+ private ArrowBuf importViewData(ArrowType type, int index, long capacity) {
+ return importBuffer(type, index, capacity);
+ }
Review Comment:
Yes, you're right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1644252391
##########
java/c/src/main/java/org/apache/arrow/c/StructVectorUnloader.java:
##########
@@ -108,7 +108,7 @@ private void appendNodes(
int expectedBufferCount =
(int) (TypeLayout.getTypeBufferCount(vector.getField().getType()) + variadicBufferCount);
// only update variadicBufferCounts for vectors that have variadic buffers
- if (variadicBufferCount > 0) {
Review Comment:
For all in-line view data, we should note that the variadic buffer count is 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [WIP] [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1628995808
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +223,48 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ try (ArrowBuf view = importView(type)) {
+ List<ArrowBuf> buffers = new ArrayList<>();
+ view.getReferenceManager().retain();
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ final int elementSize = BaseVariableWidthViewVector.ELEMENT_SIZE;
+ final int lengthWidth = BaseVariableWidthViewVector.LENGTH_WIDTH;
+ final int prefixWidth = BaseVariableWidthViewVector.PREFIX_WIDTH;
+ final int lengthPrefixWidth = lengthWidth + prefixWidth;
+ // Map to store the data buffer index and the total length of data in that buffer
+ Map<Integer, Long> dataBufferInfo = new HashMap<>();
+ for (int i = 0; i < fieldNode.getLength(); i++) {
+ final int length = view.getInt((long) i * elementSize);
+ if (length > BaseVariableWidthViewVector.INLINE_SIZE) {
+ checkState(maybeValidityBuffer != null,
+ "Validity buffer is required for data of type " + type);
+ if (BitVectorHelper.get(maybeValidityBuffer, i) == 1) {
+ final int bufferIndex =
+ view.getInt(((long) i * elementSize) + lengthPrefixWidth);
+ if (dataBufferInfo.containsKey(bufferIndex)) {
+ dataBufferInfo.compute(bufferIndex, (key, value) -> value != null ? value + (long) length : 0);
+ } else {
+ dataBufferInfo.put(bufferIndex, (long) length);
+ }
+ }
+ }
+ }
Review Comment:
should we go for a more efficient approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1630644483
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +223,48 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ try (ArrowBuf view = importView(type)) {
+ List<ArrowBuf> buffers = new ArrayList<>();
+ view.getReferenceManager().retain();
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ final int elementSize = BaseVariableWidthViewVector.ELEMENT_SIZE;
+ final int lengthWidth = BaseVariableWidthViewVector.LENGTH_WIDTH;
+ final int prefixWidth = BaseVariableWidthViewVector.PREFIX_WIDTH;
+ final int lengthPrefixWidth = lengthWidth + prefixWidth;
+ // Map to store the data buffer index and the total length of data in that buffer
+ Map<Integer, Long> dataBufferInfo = new HashMap<>();
+ for (int i = 0; i < fieldNode.getLength(); i++) {
+ final int length = view.getInt((long) i * elementSize);
+ if (length > BaseVariableWidthViewVector.INLINE_SIZE) {
+ checkState(maybeValidityBuffer != null,
+ "Validity buffer is required for data of type " + type);
+ if (BitVectorHelper.get(maybeValidityBuffer, i) == 1) {
+ final int bufferIndex =
+ view.getInt(((long) i * elementSize) + lengthPrefixWidth);
+ if (dataBufferInfo.containsKey(bufferIndex)) {
+ dataBufferInfo.compute(bufferIndex, (key, value) -> value != null ? value + (long) length : 0);
+ } else {
+ dataBufferInfo.put(bufferIndex, (long) length);
+ }
+ }
+ }
+ }
Review Comment:
Yeah I was referring to using two arrays, but it's less readable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [WIP] [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1628981128
##########
java/c/src/test/java/org/apache/arrow/c/StreamTest.java:
##########
@@ -132,6 +134,80 @@ public void roundtripStrings() throws Exception {
}
}
+ @Test
+ public void roundtripStringViews() throws Exception {
+ final Schema schema = new Schema(Arrays.asList(Field.nullable("ints", new ArrowType.Int(32, true)),
+ Field.nullable("string_views", new ArrowType.Utf8View())));
+ final List<ArrowRecordBatch> batches = new ArrayList<>();
+ try (final VectorSchemaRoot root = VectorSchemaRoot.create(schema, allocator)) {
+ final IntVector ints = (IntVector) root.getVector(0);
+ final ViewVarCharVector strs = (ViewVarCharVector) root.getVector(1);
+ VectorUnloader unloader = new VectorUnloader(root);
+
+ root.allocateNew();
+ ints.setSafe(0, 1);
+ ints.setSafe(1, 2);
+ ints.setSafe(2, 4);
+ ints.setSafe(3, 8);
+ strs.setSafe(0, "".getBytes(StandardCharsets.UTF_8));
+ strs.setSafe(1, "a".getBytes(StandardCharsets.UTF_8));
+ strs.setSafe(2, "bc1234567890bc".getBytes(StandardCharsets.UTF_8));
+ strs.setSafe(3, "defg1234567890defg".getBytes(StandardCharsets.UTF_8));
+ root.setRowCount(4);
+ batches.add(unloader.getRecordBatch());
+
+ root.allocateNew();
+ ints.setSafe(0, 1);
+ ints.setNull(1);
+ ints.setSafe(2, 4);
+ ints.setNull(3);
+ strs.setSafe(0, "".getBytes(StandardCharsets.UTF_8));
+ strs.setNull(1);
+ strs.setSafe(2, "bc1234567890bc".getBytes(StandardCharsets.UTF_8));
+ strs.setNull(3);
+ root.setRowCount(4);
+ batches.add(unloader.getRecordBatch());
+ roundtrip(schema, batches);
+ }
+ }
+
+ @Test
+ public void roundtripBinaryViews() throws Exception {
+ final Schema schema = new Schema(Arrays.asList(Field.nullable("ints", new ArrowType.Int(32, true)),
+ Field.nullable("binary_views", new ArrowType.BinaryView())));
+ final List<ArrowRecordBatch> batches = new ArrayList<>();
+ try (final VectorSchemaRoot root = VectorSchemaRoot.create(schema, allocator)) {
+ final IntVector ints = (IntVector) root.getVector(0);
+ final ViewVarBinaryVector strs = (ViewVarBinaryVector) root.getVector(1);
+ VectorUnloader unloader = new VectorUnloader(root);
+
+ root.allocateNew();
+ ints.setSafe(0, 1);
+ ints.setSafe(1, 2);
+ ints.setSafe(2, 4);
+ ints.setSafe(3, 8);
+ strs.setSafe(0, "".getBytes(StandardCharsets.UTF_8));
Review Comment:
@lidavidm prefer real binary data here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1632525671
##########
java/c/src/main/java/org/apache/arrow/c/ArrayExporter.java:
##########
@@ -96,8 +97,34 @@ void export(ArrowArray array, FieldVector vector, DictionaryProvider dictionaryP
}
if (buffers != null) {
+ final long bufferSize = buffers.size();
+ /*
+ * For Variadic types, an additional buffer is kept to store
+ * the size of each variadic buffer since that information
+ * cannot be retrieved in the import component.
+ * Here, the dataBufferReqCount is calculated to determine
+ * the additional number of buffers required.
+ * Also note that if the bufferSize is greater than 2, it means
+ * there is one or more data buffers.
+ * Thus, the dataBufferReqCount is set to 1 to get additional buffer
+ * for to store variadic size buffer.
+ * If it is not the case, the dataBuffer is not present.
+ * According to the spec and C Data interface in C++, there must be
+ * at least 3 data buffers present at the import component.
+ * Thus, the dataBufferReqCount is set to 2 to get additional buffer
+ * for empty dataBuffer and the variadic size buffer.
+ */
+ int dataBufferReqCount = 0;
+ if (vector instanceof BaseVariableWidthViewVector) {
Review Comment:
We don't have that here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1632576374
##########
java/c/src/main/java/org/apache/arrow/c/ArrayExporter.java:
##########
@@ -96,8 +97,34 @@ void export(ArrowArray array, FieldVector vector, DictionaryProvider dictionaryP
}
if (buffers != null) {
+ final long bufferSize = buffers.size();
+ /*
+ * For Variadic types, an additional buffer is kept to store
+ * the size of each variadic buffer since that information
+ * cannot be retrieved in the import component.
+ * Here, the dataBufferReqCount is calculated to determine
+ * the additional number of buffers required.
+ * Also note that if the bufferSize is greater than 2, it means
+ * there is one or more data buffers.
+ * Thus, the dataBufferReqCount is set to 1 to get additional buffer
+ * for to store variadic size buffer.
+ * If it is not the case, the dataBuffer is not present.
+ * According to the spec and C Data interface in C++, there must be
+ * at least 3 data buffers present at the import component.
+ * Thus, the dataBufferReqCount is set to 2 to get additional buffer
+ * for empty dataBuffer and the variadic size buffer.
+ */
+ int dataBufferReqCount = 0;
+ if (vector instanceof BaseVariableWidthViewVector) {
Review Comment:
Yes I mean we can pass that to the export as a variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1635890904
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
Review Comment:
Oh well, I followed the existing pattern for other vectors, honestly didn't change much. Instead of offset, view is retaining, and I used the validity util. Thoughts?
##########
java/c/src/main/java/org/apache/arrow/c/ArrayExporter.java:
##########
@@ -91,7 +91,8 @@ void export(ArrowArray array, FieldVector vector, DictionaryProvider dictionaryP
if (buffers != null) {
data.buffers = new ArrayList<>(buffers.size());
Review Comment:
Yes, that's right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1630796040
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -117,10 +121,19 @@ private ArrowBuf importOffsets(ArrowType type, long bytesPerSlot) {
return importBuffer(type, 1, capacity);
}
+ private ArrowBuf importView(ArrowType type) {
+ final long capacity = (long) fieldNode.getLength() * BaseVariableWidthViewVector.ELEMENT_SIZE;
+ return importBuffer(type, 1, capacity);
+ }
Review Comment:
Yes, we should use that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1632525569
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +210,38 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ try (ArrowBuf view = importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
Review Comment:
Why is the view buffer the first one we import? The one we care about is the sizes buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1632144335
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -331,7 +360,7 @@ public List<ArrowBuf> visit(ArrowType.Duration type) {
}
@Override
- public List<ArrowBuf> visit(ListView type) {
+ public List<ArrowBuf> visit(ArrowType.ListView type) {
Review Comment:
minor change to keep the consistency which was a typo made in a previous PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1632609769
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +210,38 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ try (ArrowBuf view = importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
Review Comment:
In the sense that...you need the sizes buffer to do anything, so import that first, then discard it at the end? Why are we treating the view buffer specially when it's not special here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1632602109
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +210,38 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ try (ArrowBuf view = importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
Review Comment:
Ah nothing special there, I can change the order. Validity buffer, view buffer, sizes buffer then variadic buffer.
Any issue doing it this way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1632143149
##########
java/c/src/main/java/org/apache/arrow/c/Format.java:
##########
@@ -212,6 +212,10 @@ static String asString(ArrowType arrowType) {
}
case Utf8:
return "u";
+ case Utf8View:
+ return "vu";
+ case BinaryView:
+ return "VZ";
Review Comment:
You're right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1632526488
##########
java/c/src/main/java/org/apache/arrow/c/ArrayExporter.java:
##########
@@ -96,8 +97,34 @@ void export(ArrowArray array, FieldVector vector, DictionaryProvider dictionaryP
}
if (buffers != null) {
+ final long bufferSize = buffers.size();
+ /*
+ * For Variadic types, an additional buffer is kept to store
+ * the size of each variadic buffer since that information
+ * cannot be retrieved in the import component.
+ * Here, the dataBufferReqCount is calculated to determine
+ * the additional number of buffers required.
+ * Also note that if the bufferSize is greater than 2, it means
+ * there is one or more data buffers.
+ * Thus, the dataBufferReqCount is set to 1 to get additional buffer
+ * for to store variadic size buffer.
+ * If it is not the case, the dataBuffer is not present.
+ * According to the spec and C Data interface in C++, there must be
+ * at least 3 data buffers present at the import component.
+ * Thus, the dataBufferReqCount is set to 2 to get additional buffer
+ * for empty dataBuffer and the variadic size buffer.
+ */
+ int dataBufferReqCount = 0;
+ if (vector instanceof BaseVariableWidthViewVector) {
Review Comment:
Or rather, who would populate that?
I don't think the C Data Interface should be called directly by consumers. In any case they were going to break at some point because of this anyways so I'd rather fix it properly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "urvishdesai (via GitHub)" <gi...@apache.org>.
urvishdesai commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1631736490
##########
java/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java:
##########
@@ -1667,4 +1667,23 @@ protected void getData(int index, ReusableBuffer<?> buffer) {
public <OUT, IN> OUT accept(VectorVisitor<OUT, IN> visitor, IN value) {
return visitor.visit(this, value);
}
+
+ @Override
+ public void exportCDataBuffers(List<ArrowBuf> buffers, ArrowBuf buffersPtr, long nullValue) {
+ exportBuffer(validityBuffer, buffers, buffersPtr, nullValue, true);
+
+ if (viewBuffer.capacity() == 0) {
+ // Empty view buffer is allowed here.
+ // We set `retain = false` to explicitly not increase the ref count for the exported buffer.
+ // The ref count of the newly created buffer (i.e., 1) already represents the usage
+ // of the imported side.
+ exportBuffer(allocator.getEmpty(), buffers, buffersPtr, nullValue, false);
+ } else {
+ exportBuffer(viewBuffer, buffers, buffersPtr, nullValue, true);
+ }
+
+ for (int i = 0; i < dataBuffers.size(); i++) {
+ exportBuffer(dataBuffers.get(i), buffers, buffersPtr, nullValue, true);
+ }
Review Comment:
We need another buffer to be appended at the end that stores the sizes of each of the data buffer. This buffer from the spec is missing here. This is probably why you are seeing "Expected at least 3 buffers for imported type" error.
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +223,48 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ try (ArrowBuf view = importView(type)) {
+ List<ArrowBuf> buffers = new ArrayList<>();
+ view.getReferenceManager().retain();
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ final int elementSize = BaseVariableWidthViewVector.ELEMENT_SIZE;
+ final int lengthWidth = BaseVariableWidthViewVector.LENGTH_WIDTH;
+ final int prefixWidth = BaseVariableWidthViewVector.PREFIX_WIDTH;
+ final int lengthPrefixWidth = lengthWidth + prefixWidth;
+ // Map to store the data buffer index and the total length of data in that buffer
+ Map<Integer, Long> dataBufferInfo = new HashMap<>();
+ for (int i = 0; i < fieldNode.getLength(); i++) {
+ final int length = view.getInt((long) i * elementSize);
+ if (length > BaseVariableWidthViewVector.INLINE_SIZE) {
+ checkState(maybeValidityBuffer != null,
+ "Validity buffer is required for data of type " + type);
+ if (BitVectorHelper.get(maybeValidityBuffer, i) == 1) {
+ final int bufferIndex =
+ view.getInt(((long) i * elementSize) + lengthPrefixWidth);
+ if (dataBufferInfo.containsKey(bufferIndex)) {
+ dataBufferInfo.compute(bufferIndex, (key, value) -> value != null ? value + (long) length : 0);
+ } else {
+ dataBufferInfo.put(bufferIndex, (long) length);
+ }
+ }
+ }
+ }
Review Comment:
The last buffer was added to the arrow spec to store the length of each of the data buffers. You can use the values of this buffer as array of lengths to import the variably sized data buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1635893773
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
Review Comment:
Presumably what it's doing is trying to be safe if an exception happens, but in this case it's not doing that fully, so let's make it fully safe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1631879205
##########
java/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java:
##########
@@ -1667,4 +1667,23 @@ protected void getData(int index, ReusableBuffer<?> buffer) {
public <OUT, IN> OUT accept(VectorVisitor<OUT, IN> visitor, IN value) {
return visitor.visit(this, value);
}
+
+ @Override
+ public void exportCDataBuffers(List<ArrowBuf> buffers, ArrowBuf buffersPtr, long nullValue) {
+ exportBuffer(validityBuffer, buffers, buffersPtr, nullValue, true);
+
+ if (viewBuffer.capacity() == 0) {
+ // Empty view buffer is allowed here.
+ // We set `retain = false` to explicitly not increase the ref count for the exported buffer.
+ // The ref count of the newly created buffer (i.e., 1) already represents the usage
+ // of the imported side.
+ exportBuffer(allocator.getEmpty(), buffers, buffersPtr, nullValue, false);
+ } else {
+ exportBuffer(viewBuffer, buffers, buffersPtr, nullValue, true);
+ }
+
+ for (int i = 0; i < dataBuffers.size(); i++) {
+ exportBuffer(dataBuffers.get(i), buffers, buffersPtr, nullValue, true);
+ }
Review Comment:
Yes, you're right. I realized it a bit late, once seeing the integration test failure. I am working on that.
Thanks for the thoughts. :slightly_smiling_face:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1635916677
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
+ // 0th buffer is validity buffer
+ // 1st buffer is view buffer
+ // 2nd buffer onwards are variadic buffer
+ // N-1 (this.buffers.length - 1) buffer is variadic size buffer
+ final int variadicBufferReadOffset = 2;
+ variadicSizeBuffer.getReferenceManager().retain();
Review Comment:
I mean I got the negative refCount when variadicSIzeBuffer wasn't retained.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "urvishdesai (via GitHub)" <gi...@apache.org>.
urvishdesai commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1631711855
##########
java/c/src/main/java/org/apache/arrow/c/Format.java:
##########
@@ -212,6 +212,10 @@ static String asString(ArrowType arrowType) {
}
case Utf8:
return "u";
+ case Utf8View:
+ return "vu";
+ case BinaryView:
+ return "VZ";
Review Comment:
AFAIK the BinaryView format string is lowercase "vz"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1632504377
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +210,38 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ try (ArrowBuf view = importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
Review Comment:
Treated specially means? Sorry I didn't follow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1644369815
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
+ // 0th buffer is validity buffer
+ // 1st buffer is view buffer
+ // 2nd buffer onwards are variadic buffer
+ // N-1 (this.buffers.length - 1) buffer is variadic size buffer
+ final int variadicBufferReadOffset = 2;
+ variadicSizeBuffer.getReferenceManager().retain();
Review Comment:
Yes, I think in this case, the `BufferImportTypeVisitor` is used with try-with-resource already, thus, the imported buffers are always released, so we don't need to explicitly release the `variadicSizeBuffer`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [WIP] [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1628977945
##########
java/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java:
##########
@@ -1667,4 +1667,23 @@ protected void getData(int index, ReusableBuffer<?> buffer) {
public <OUT, IN> OUT accept(VectorVisitor<OUT, IN> visitor, IN value) {
return visitor.visit(this, value);
}
+
+ @Override
+ public void exportCDataBuffers(List<ArrowBuf> buffers, ArrowBuf buffersPtr, long nullValue) {
+ exportBuffer(validityBuffer, buffers, buffersPtr, nullValue, true);
+
+ if (viewBuffer.capacity() == 0) {
+ // Empty view buffer is allowed here.
+ // We set `retain = false` to explicitly not increase the ref count for the exported buffer.
+ // The ref count of the newly created buffer (i.e., 1) already represents the usage
+ // of the imported side.
+ exportBuffer(allocator.buffer(INITIAL_BYTE_COUNT), buffers, buffersPtr, nullValue, false);
Review Comment:
@lidavidm should we do this in a better way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #41967:
URL: https://github.com/apache/arrow/pull/41967#issuecomment-2156287755
> Added some comments regarding the missing buffer at the end that stores variadic buffer lengths.
@urvishdesai I updated the PR. Could you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1634861855
##########
java/c/src/test/python/integration_tests.py:
##########
@@ -183,6 +190,55 @@ def round_trip_reader(self, schema, batches):
def test_string_array(self):
self.round_trip_array(lambda: pa.array([None, "a", "bb", "ccc"]))
+ def test_stringview_array(self):
+ # with nulls short strings
+ self.round_trip_array(lambda: pa.array([None, "a", "bb", "c"], type=pa.string_view()))
+ # with nulls long and strings
+ self.round_trip_array(lambda: pa.array([None, "a", "bb"*10, "c"*13], type=pa.string_view()))
+ # without nulls short strings
+ self.round_trip_array(lambda: pa.array(["a", "bb", "c"], type=pa.string_view()))
+ # without nulls long and strings
+ self.round_trip_array(lambda: pa.array(["a", "bb"*10, "c"*13], type=pa.string_view()))
+ # with multiple data buffers
+ data = []
+ for i in range(1, 501):
+ s = ''.join(str(j) for j in range(i))
+ data.append(s)
+ self.round_trip_array(lambda: pa.array(data, type=pa.string_view()))
Review Comment:
This will indeed create multiple data buffers, although that relies a bit on the Builder's default size used for a data buffer. Another potentially slightly easier way to create am array with multiple data buffers is by concatenating;
```
In [30]: arr1 = pa.array([None, "a", "bb", "c"], type=pa.string_view())
In [31]: arr2 = pa.array(["a", "bb"*10, "c"*13], type=pa.string_view())
In [32]: pa.concat_arrays([arr1, arr2]).buffers()
Out[32]:
[<pyarrow.Buffer address=0x7faeab209180 size=1 is_cpu=True is_mutable=True>,
<pyarrow.Buffer address=0x7faeab208100 size=112 is_cpu=True is_mutable=True>,
<pyarrow.Buffer address=0x7faead33e5c0 size=0 is_cpu=True is_mutable=True>,
<pyarrow.Buffer address=0x7faeab2092c0 size=33 is_cpu=True is_mutable=True>]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1635893152
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
Review Comment:
Either: try-with-resources + retain _all_ the buffers consistently, or do it for none of the buffers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1635892470
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
Review Comment:
Unless we have to retain it, wouldn't this be a memory leak?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1635915830
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
+ // 0th buffer is validity buffer
+ // 1st buffer is view buffer
+ // 2nd buffer onwards are variadic buffer
+ // N-1 (this.buffers.length - 1) buffer is variadic size buffer
+ final int variadicBufferReadOffset = 2;
+ variadicSizeBuffer.getReferenceManager().retain();
Review Comment:
So just load and close?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1630805165
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -117,10 +121,19 @@ private ArrowBuf importOffsets(ArrowType type, long bytesPerSlot) {
return importBuffer(type, 1, capacity);
}
+ private ArrowBuf importView(ArrowType type) {
+ final long capacity = (long) fieldNode.getLength() * BaseVariableWidthViewVector.ELEMENT_SIZE;
+ return importBuffer(type, 1, capacity);
+ }
Review Comment:
I am looking into this...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #41967:
URL: https://github.com/apache/arrow/pull/41967#issuecomment-2154302361
@lidavidm I updated the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #41967:
URL: https://github.com/apache/arrow/pull/41967#issuecomment-2154124212
> Can we enable the integration tests?
Let me take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #41967:
URL: https://github.com/apache/arrow/pull/41967#issuecomment-2159588699
@lidavidm ready for another round of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "urvishdesai (via GitHub)" <gi...@apache.org>.
urvishdesai commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1631712425
##########
java/c/src/main/java/org/apache/arrow/c/Format.java:
##########
@@ -291,6 +295,10 @@ static ArrowType asType(String format, long flags)
case "+m":
boolean keysSorted = (flags & Flags.ARROW_FLAG_MAP_KEYS_SORTED) != 0;
return new ArrowType.Map(keysSorted);
+ case "vu":
+ return new ArrowType.Utf8View();
+ case "VZ":
Review Comment:
Same here. BinaryView format string should be lowercase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #41967:
URL: https://github.com/apache/arrow/pull/41967#issuecomment-2156287576
@lidavidm I enabled CIs please take a look. Also logic was updated. The initial version didn't have the variadic size buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #41967:
URL: https://github.com/apache/arrow/pull/41967#issuecomment-2154436239
@lidavidm seems like we have to append an empty data buffer too?[1]: https://github.com/apache/arrow/actions/runs/9414397121/job/25933218677?pr=41967#step:6:8512
[1]. https://github.com/apache/arrow/blob/a045770b94972bb4063bde13cb95f1c5b5c8bbe8/cpp/src/arrow/c/bridge.cc#L1742
Is this correct according to the spec?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1635914950
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
+ // 0th buffer is validity buffer
+ // 1st buffer is view buffer
+ // 2nd buffer onwards are variadic buffer
+ // N-1 (this.buffers.length - 1) buffer is variadic size buffer
+ final int variadicBufferReadOffset = 2;
+ variadicSizeBuffer.getReferenceManager().retain();
Review Comment:
I think the other methods are just being too careful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1635914325
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
+ // 0th buffer is validity buffer
+ // 1st buffer is view buffer
+ // 2nd buffer onwards are variadic buffer
+ // N-1 (this.buffers.length - 1) buffer is variadic size buffer
+ final int variadicBufferReadOffset = 2;
+ variadicSizeBuffer.getReferenceManager().retain();
Review Comment:
I think you don't need the try-with-resources
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1635897143
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
+ // 0th buffer is validity buffer
+ // 1st buffer is view buffer
+ // 2nd buffer onwards are variadic buffer
+ // N-1 (this.buffers.length - 1) buffer is variadic size buffer
+ final int variadicBufferReadOffset = 2;
+ variadicSizeBuffer.getReferenceManager().retain();
Review Comment:
Related to the previous one, I got a ref count negative issue without the retain.
Doing anything wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40942: [Java] Implement C Data Interface for StringView [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1635889964
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -227,10 +227,39 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ final int viewBufferIndex = 1;
+ final int variadicSizeBufferIndex = this.buffers.length - 1;
+ final long numOfVariadicBuffers = this.buffers.length - 3;
+ final long variadicSizeBufferCapacity = numOfVariadicBuffers * Long.BYTES;
+ List<ArrowBuf> buffers = new ArrayList<>();
+
+ try (ArrowBuf variadicSizeBuffer =
+ importBuffer(type, variadicSizeBufferIndex, variadicSizeBufferCapacity)) {
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ try (ArrowBuf view =
+ importFixedBytes(type, viewBufferIndex, BaseVariableWidthViewVector.ELEMENT_SIZE)) {
+ view.getReferenceManager().retain();
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ }
+ // 0th buffer is validity buffer
+ // 1st buffer is view buffer
+ // 2nd buffer onwards are variadic buffer
+ // N-1 (this.buffers.length - 1) buffer is variadic size buffer
+ final int variadicBufferReadOffset = 2;
+ variadicSizeBuffer.getReferenceManager().retain();
Review Comment:
should we discard here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org