You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "ViggoC (via GitHub)" <gi...@apache.org> on 2024/08/29 17:38:29 UTC
[PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
ViggoC opened a new pull request, #43888:
URL: https://github.com/apache/arrow/pull/43888
# WIP
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
### What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
### Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
### Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "ViggoC (via GitHub)" <gi...@apache.org>.
ViggoC commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1757126169
##########
java/vector/src/main/java/org/apache/arrow/vector/validate/ValidateVectorBufferVisitor.java:
##########
@@ -287,4 +289,37 @@ public Void visit(ExtensionTypeVector<?> vector, Void value) {
vector.getUnderlyingVector().accept(this, value);
return null;
}
+
+ @Override
+ public Void visit(RunEndEncodedVector vector, Void value) {
+ validateVectorCommon(vector);
+ int valueCount = vector.getValueCount();
+ FieldVector runEndsVector = vector.getRunEndsVector();
+
+ if (runEndsVector != null) {
Review Comment:
According to comment here, O(1) time is committed. However the worst time of monotonicity check is O(n). Is it acceptable?
https://github.com/apache/arrow/blob/2f99cf85835c05e0d3d23d06592e6d2f6322aede/java/vector/src/main/java/org/apache/arrow/vector/util/ValueVectorUtility.java#L99-L108
##########
java/vector/src/main/java/org/apache/arrow/vector/compare/RangeEqualsVisitor.java:
##########
@@ -255,6 +266,46 @@ public Boolean visit(LargeListViewVector left, Range range) {
return compareLargeListViewVectors(range);
}
+ protected boolean compareRunEndEncodedVectors(Range range) {
+ RunEndEncodedVector leftVector = (RunEndEncodedVector) left;
+ RunEndEncodedVector rightVector = (RunEndEncodedVector) right;
+
+ final int leftRangeEnd = range.getLeftStart() + range.getLength();
+ final int rightRangeEnd = range.getRightStart() + range.getLength();
+
+ FieldVector leftValuesVector = leftVector.getValuesVector();
+ FieldVector rightValuesVector = rightVector.getValuesVector();
+
+ RangeEqualsVisitor innerVisitor = createInnerVisitor(leftValuesVector, rightValuesVector, null);
+
+ int leftLogicalIndex = range.getLeftStart();
+ int rightLogicalIndex = range.getRightStart();
+
+ while (leftLogicalIndex < leftRangeEnd) {
+ int leftPhysicalIndex = leftVector.getPhysicalIndex(leftLogicalIndex);
Review Comment:
Thank you for your suggestion, I'll try to do it.
##########
java/vector/src/test/java/org/apache/arrow/vector/TestRunEndEncodedVector.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.List;
+import java.util.function.Function;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.vector.compare.Range;
+import org.apache.arrow.vector.compare.RangeEqualsVisitor;
+import org.apache.arrow.vector.complex.RunEndEncodedVector;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.ArrowType.RunEndEncoded;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+public class TestRunEndEncodedVector {
+
+ private BufferAllocator allocator;
+
+ @BeforeEach
+ public void init() {
+ allocator = new DirtyRootAllocator(Long.MAX_VALUE, (byte) 100);
Review Comment:
I have no idea which allocator is better, I just copy it from TestListVector. So you prefer to just use a RootAllocator?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm merged PR #43888:
URL: https://github.com/apache/arrow/pull/43888
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1755951156
##########
java/vector/src/main/java/org/apache/arrow/vector/compare/VectorVisitor.java:
##########
@@ -71,4 +72,6 @@ default OUT visit(LargeListViewVector left, IN value) {
throw new UnsupportedOperationException(
"VectorVisitor for LargeListViewVector is not supported.");
}
+
+ OUT visit(RunEndEncodedVector left, IN value);
Review Comment:
Let's make this a default method
##########
java/vector/src/main/java/org/apache/arrow/vector/types/Types.java:
##########
@@ -786,6 +788,18 @@ public FieldWriter getNewFieldWriter(ValueVector vector) {
.getNewFieldWriter(vector);
}
},
+ RUNENDENCODED(RunEndEncoded.INSTANCE) {
+ @Override
+ public FieldVector getNewVector(
+ Field field, BufferAllocator allocator, CallBack schemaChangeCallback) {
+ return new RunEndEncodedVector(field, allocator, schemaChangeCallback);
+ }
+
+ @Override
+ public FieldWriter getNewFieldWriter(ValueVector vector) {
+ return null; // TODO
Review Comment:
Throw an exception instead
##########
java/vector/src/test/java/org/apache/arrow/vector/TestRunEndEncodedVector.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.List;
+import java.util.function.Function;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.vector.compare.Range;
+import org.apache.arrow.vector.compare.RangeEqualsVisitor;
+import org.apache.arrow.vector.complex.RunEndEncodedVector;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.ArrowType.RunEndEncoded;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+public class TestRunEndEncodedVector {
+
+ private BufferAllocator allocator;
+
+ @BeforeEach
+ public void init() {
+ allocator = new DirtyRootAllocator(Long.MAX_VALUE, (byte) 100);
Review Comment:
Do we specifically need this allocator here?
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,670 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ return null;
+ }
+
+ /**
+ * Set the initial record capacity.
+ *
+ * @param numRecords the initial record capacity.
+ */
+ @Override
+ public void setInitialCapacity(int numRecords) {}
+
+ /**
+ * Returns the maximum number of values that can be stored in this vector instance.
+ *
+ * @return the maximum number of values that can be stored in this vector instance.
+ */
+ @Override
+ public int getValueCapacity() {
+ return getChildrenFromFields().stream()
+ .mapToInt(item -> item != null ? item.getValueCapacity() : 0)
+ .min()
+ .orElseThrow(NoSuchElementException::new);
+ }
+
+ /** Alternative to clear(). Allows use as an AutoCloseable in try-with-resources. */
+ @Override
+ public void close() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.close();
+ }
+ }
+
+ /**
+ * Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has
+ * any child vectors, they will also be cleared.
+ */
+ @Override
+ public void clear() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.clear();
+ }
+ }
+
+ /**
+ * Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer
+ * capacities will remain unchanged and any previous data will be zeroed out. This includes
+ * buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be
+ * reset.
+ */
+ @Override
+ public void reset() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reset();
+ }
+ valueCount = 0;
+ }
+
+ /**
+ * Get information about how this field is materialized.
+ *
+ * @return the field corresponding to this vector
+ */
+ @Override
+ public Field getField() {
+ return field;
+ }
+
+ @Override
+ public MinorType getMinorType() {
+ return MinorType.RUNENDENCODED;
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(BufferAllocator allocator) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(BufferAllocator)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator) {
+ return getTransferPair(ref, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator) {
+ return getTransferPair(field, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(String, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(Field, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * Makes a new transfer pair used to transfer underlying buffers.
+ *
+ * @param target the target for the transfer
+ * @return a new {@link org.apache.arrow.vector.util.TransferPair transfer pair} that is used to
+ * transfer underlying buffers into the target vector.
+ */
+ @Override
+ public TransferPair makeTransferPair(ValueVector target) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support makeTransferPair(ValueVector)");
+ }
+
+ /**
+ * Get a reader for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.reader.FieldReader field reader} that supports
+ * reading values from this vector.
+ */
+ @Override
+ public FieldReader getReader() {
+ return null; // TODO
+ }
+
+ /**
+ * Get a writer for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.writer.FieldWriter field writer} that supports
+ * writing values to this vector.
+ */
+ public FieldWriter getWriter() {
+ return null; // TODO
+ }
+
+ /**
+ * Get the number of bytes used by this vector.
+ *
+ * @return the number of bytes that is used by this vector instance.
+ */
+ @Override
+ public int getBufferSize() {
+ int bufferSize = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ bufferSize += v.getBufferSize();
+ }
+ return bufferSize;
+ }
+
+ /**
+ * Returns the number of bytes that is used by this vector if it holds the given number of values.
+ * The result will be the same as if setValueCount() were called, followed by calling
+ * getBufferSize(), but without any of the closing side-effects that setValueCount() implies wrt
+ * finishing off the population of a vector. Some operations might wish to use this to determine
+ * how much memory has been used by a vector so far, even though it is not finished being
+ * populated.
+ *
+ * @param valueCount the number of values to assume this vector contains
+ * @return the buffer size if this vector is holding valueCount values
+ */
+ @Override
+ public int getBufferSizeFor(int valueCount) {
+ return 0;
+ }
+
+ /**
+ * Return the underlying buffers associated with this vector. Note that this doesn't impact the
+ * reference counts for this buffer so it only should be used for in-context access. Also note
+ * that this buffer changes regularly thus external classes shouldn't hold a reference to it
+ * (unless they change it).
+ *
+ * @param clear Whether to clear vector before returning; the buffers will still be refcounted;
+ * but the returned array will be the only reference to them
+ * @return The underlying {@link ArrowBuf buffers} that is used by this vector instance.
+ */
+ @Override
+ public ArrowBuf[] getBuffers(boolean clear) {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with validity vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getValidityBuffer() {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with data vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getDataBuffer() {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with offset vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getOffsetBuffer() {
+ return null;
+ }
+
+ /**
+ * Gets the number of values.
+ *
+ * @return number of values in the vector
+ */
+ @Override
+ public int getValueCount() {
+ return valueCount;
+ }
+
+ /** Set number of values in the vector. */
+ @Override
+ public void setValueCount(int valueCount) {
+ this.valueCount = valueCount;
+ }
+
+ /**
+ * Get friendly type object from the vector.
+ *
+ * @param index index of object to get
+ * @return friendly type object
+ */
+ @Override
+ public Object getObject(int index) {
+ checkIndex(index);
+ int physicalIndex = getPhysicalIndex(index);
+ return valuesVector.getObject(physicalIndex);
+ }
+
+ /**
+ * Get the run end of giving index.
+ *
+ * @param index index of the run end to get
+ * @return the run end of giving index
+ */
+ public int getRunEnd(int index) {
+ checkIndex(index);
+ int physicalIndex = getPhysicalIndex(index);
+ return (int) ((BaseIntVector) runEndsVector).getValueAsLong(physicalIndex);
+ }
+
+ /**
+ * Returns number of null elements in the vector.
+ *
+ * @return number of null elements
+ */
+ @Override
+ public int getNullCount() {
+ // Null count is always 0 for run-end encoded array
+ return 0;
+ }
+
+ /**
+ * Check whether an element in the vector is null.
+ *
+ * @param index index to check for null
+ * @return true if element is null
+ */
+ @Override
+ public boolean isNull(int index) {
+ int physicalIndex = getPhysicalIndex(runEndsVector, index);
+ return valuesVector.isNull(physicalIndex);
+ }
+
+ /** Returns hashCode of element in index with the default hasher. */
+ @Override
+ public int hashCode(int index) {
+ return hashCode(index, null);
+ }
+
+ /** Returns hashCode of element in index with the given hasher. */
+ @Override
+ public int hashCode(int index, ArrowBufHasher hasher) {
+ int hash = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ if (index < v.getValueCount()) {
+ hash = ByteFunctionHelpers.combineHash(hash, v.hashCode(index, hasher));
+ }
+ }
+ return hash;
+ }
+
+ /**
+ * Accept a generic {@link VectorVisitor} and return the result.
+ *
+ * @param <OUT> the output result type.
+ * @param <IN> the input data together with visitor.
+ */
+ @Override
+ public <OUT, IN> OUT accept(VectorVisitor<OUT, IN> visitor, IN value) {
+ return visitor.visit(this, value);
+ }
+
+ /**
+ * Gets the name of the vector.
+ *
+ * @return the name of the vector.
+ */
+ @Override
+ public String getName() {
+ return this.field.getName();
+ }
+
+ @Override
+ public Iterator<ValueVector> iterator() {
+ return Collections.<ValueVector>unmodifiableCollection(getChildrenFromFields()).iterator();
+ }
+
+ /** FieldVector interface */
+
+ /**
+ * Initializes the child vectors to be later loaded with loadBuffers.
+ *
+ * @param children the schema containing the run_ends column first and the values column second
+ */
+ @Override
+ public void initializeChildrenFromFields(List<Field> children) {
+ checkArgument(
+ children.size() == 2,
+ "Run-end encoded vectors must have two child Fields. Found: %s",
+ children.isEmpty() ? "none" : children);
+ checkArgument(
+ Arrays.asList(
+ MinorType.SMALLINT.getType(), MinorType.INT.getType(), MinorType.BIGINT.getType())
+ .contains(children.get(0).getType()),
+ "The first field represents the run-end vector and must be of type int "
+ + "with size 16, 32, or 64 bits. Found: %s",
+ children.get(0).getType());
+ runEndsVector = (BaseIntVector) children.get(0).createVector(allocator);
+ valuesVector = children.get(1).createVector(allocator);
+ field = new Field(field.getName(), field.getFieldType(), children);
+ }
+
+ /**
+ * The returned list is the same size as the list passed to initializeChildrenFromFields.
+ *
+ * @return the children according to schema (empty for primitive types)
+ */
+ @Override
+ public List<FieldVector> getChildrenFromFields() {
+ return Arrays.asList(runEndsVector, valuesVector);
+ }
+
+ /**
+ * Loads data in the vectors. (ownBuffers must be the same size as getFieldVectors())
+ *
+ * @param fieldNode the fieldNode
+ * @param ownBuffers the buffers for this Field (own buffers only, children not included)
+ */
+ @Override
+ public void loadFieldBuffers(ArrowFieldNode fieldNode, List<ArrowBuf> ownBuffers) {
+ throw new UnsupportedOperationException(
+ "Run-end encoded vectors do not have any associated buffers.");
+ }
+
+ /**
+ * Get the buffers of the fields, (same size as getFieldVectors() since it is their content).
+ *
+ * @return the buffers containing the data for this vector (ready for reading)
+ */
+ @Override
+ public List<ArrowBuf> getFieldBuffers() {
+ return List.of();
+ }
+
+ /**
+ * Get the inner vectors.
+ *
+ * @return the inner vectors for this field as defined by the TypeLayout
+ * @deprecated This API will be removed as the current implementations no longer support inner
+ * vectors.
+ */
+ @Deprecated
+ @Override
+ public List<BufferBacked> getFieldInnerVectors() {
+ throw new UnsupportedOperationException("There are no inner vectors. Use getFieldBuffers().");
+ }
+
+ /**
+ * Gets the starting address of the underlying buffer associated with validity vector.
+ *
+ * @return buffer address
+ */
+ @Override
+ public long getValidityBufferAddress() {
+ throw new UnsupportedOperationException(
+ "Run-end encoded vectors do not have a validity buffer.");
+ }
+
+ /**
+ * Gets the starting address of the underlying buffer associated with data vector.
+ *
+ * @return buffer address
+ */
+ @Override
+ public long getDataBufferAddress() {
+ throw new UnsupportedOperationException("Run-end encoded vectors do not have a data buffer.");
+ }
+
+ /**
+ * Gets the starting address of the underlying buffer associated with offset vector.
+ *
+ * @return buffer address
+ */
+ @Override
+ public long getOffsetBufferAddress() {
+ throw new UnsupportedOperationException(
+ "Run-end encoded vectors do not have an offset buffer.");
+ }
+
+ /**
+ * Set the element at the given index to null.
+ *
+ * @param index the value to change
+ */
+ @Override
+ public void setNull(int index) {
+ throw new UnsupportedOperationException(
Review Comment:
Hmm, we could splice the child, but let's ignore that for now
##########
java/vector/src/main/java/org/apache/arrow/vector/compare/RangeEqualsVisitor.java:
##########
@@ -255,6 +266,46 @@ public Boolean visit(LargeListViewVector left, Range range) {
return compareLargeListViewVectors(range);
}
+ protected boolean compareRunEndEncodedVectors(Range range) {
+ RunEndEncodedVector leftVector = (RunEndEncodedVector) left;
+ RunEndEncodedVector rightVector = (RunEndEncodedVector) right;
+
+ final int leftRangeEnd = range.getLeftStart() + range.getLength();
+ final int rightRangeEnd = range.getRightStart() + range.getLength();
+
+ FieldVector leftValuesVector = leftVector.getValuesVector();
+ FieldVector rightValuesVector = rightVector.getValuesVector();
+
+ RangeEqualsVisitor innerVisitor = createInnerVisitor(leftValuesVector, rightValuesVector, null);
+
+ int leftLogicalIndex = range.getLeftStart();
+ int rightLogicalIndex = range.getRightStart();
+
+ while (leftLogicalIndex < leftRangeEnd) {
+ int leftPhysicalIndex = leftVector.getPhysicalIndex(leftLogicalIndex);
Review Comment:
This is going to do a binary search on every step? One thing we could consider for the future is an iterator that can avoid that
##########
java/vector/src/main/java/org/apache/arrow/vector/validate/ValidateVectorDataVisitor.java:
##########
@@ -206,4 +207,9 @@ public Void visit(ExtensionTypeVector<?> vector, Void value) {
vector.getUnderlyingVector().accept(this, value);
return null;
}
+
+ @Override
+ public Void visit(RunEndEncodedVector vector, Void value) {
+ return null; // TODO
Review Comment:
Throw an exception instead
##########
java/vector/src/test/java/org/apache/arrow/vector/validate/TestValidateVector.java:
##########
@@ -265,6 +268,41 @@ public void testBaseFixedWidthVectorInstanceMethod() {
}
}
+ @Test
+ public void testRunEndEncodedVector() {
+ final FieldType valueType = FieldType.notNullable(Types.MinorType.BIGINT.getType());
+ final FieldType runEndType = FieldType.notNullable(Types.MinorType.INT.getType());
+
+ final Field valueField = new Field("value", valueType, null);
+ final Field runEndField = new Field("ree", runEndType, null);
+
+ try (RunEndEncodedVector vector =
+ new RunEndEncodedVector(
+ new Field(
+ "ree",
+ FieldType.notNullable(RunEndEncoded.INSTANCE),
+ List.of(runEndField, valueField)),
+ allocator,
+ null)) {
+ vector.validate();
+
+ int runCount = 1;
+ vector.allocateNew();
+ ((BigIntVector) vector.getValuesVector()).set(0, 1);
+ ((IntVector) vector.getRunEndsVector()).set(0, 10);
+ vector.getValuesVector().setValueCount(runCount);
+ vector.getRunEndsVector().setValueCount(runCount);
+ vector.setValueCount(10);
+
+ vector.validate();
+
+ vector.getRunEndsVector().setValueCount(0);
+ ValidateUtil.ValidateException e =
+ assertThrows(ValidateUtil.ValidateException.class, () -> vector.validate());
+ assertTrue(e.getMessage().contains("do not have does not contain enough elements."));
Review Comment:
The message seems off
##########
java/vector/src/main/java/org/apache/arrow/vector/validate/ValidateVectorVisitor.java:
##########
@@ -318,4 +319,9 @@ public Void visit(ExtensionTypeVector<?> vector, Void value) {
vector.getUnderlyingVector().accept(this, value);
return null;
}
+
+ @Override
+ public Void visit(RunEndEncodedVector vector, Void value) {
+ return null; // TODO
Review Comment:
Throw an exception?
##########
java/vector/src/main/java/org/apache/arrow/vector/validate/ValidateVectorBufferVisitor.java:
##########
@@ -287,4 +289,37 @@ public Void visit(ExtensionTypeVector<?> vector, Void value) {
vector.getUnderlyingVector().accept(this, value);
return null;
}
+
+ @Override
+ public Void visit(RunEndEncodedVector vector, Void value) {
+ validateVectorCommon(vector);
+ int valueCount = vector.getValueCount();
+ FieldVector runEndsVector = vector.getRunEndsVector();
+
+ if (runEndsVector != null) {
+ validateOrThrow(
+ runEndsVector.getNullCount() == 0, "Run ends vector cannot contain null values");
+ runEndsVector.accept(this, null);
+
+ int runCount = runEndsVector.getValueCount();
+ if (runCount == 0) {
+ validateOrThrow(
+ valueCount == 0, "Run end vector do not have does not contain enough elements.");
Review Comment:
```suggestion
valueCount == 0, "Run end vector does not contain enough elements");
```
##########
java/vector/src/main/java/org/apache/arrow/vector/validate/ValidateVectorBufferVisitor.java:
##########
@@ -287,4 +289,37 @@ public Void visit(ExtensionTypeVector<?> vector, Void value) {
vector.getUnderlyingVector().accept(this, value);
return null;
}
+
+ @Override
+ public Void visit(RunEndEncodedVector vector, Void value) {
+ validateVectorCommon(vector);
+ int valueCount = vector.getValueCount();
+ FieldVector runEndsVector = vector.getRunEndsVector();
+
+ if (runEndsVector != null) {
Review Comment:
Should we also check monotonicity of the run ends?
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,670 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ return null;
+ }
+
+ /**
+ * Set the initial record capacity.
+ *
+ * @param numRecords the initial record capacity.
+ */
+ @Override
+ public void setInitialCapacity(int numRecords) {}
+
+ /**
+ * Returns the maximum number of values that can be stored in this vector instance.
+ *
+ * @return the maximum number of values that can be stored in this vector instance.
+ */
+ @Override
+ public int getValueCapacity() {
+ return getChildrenFromFields().stream()
+ .mapToInt(item -> item != null ? item.getValueCapacity() : 0)
+ .min()
+ .orElseThrow(NoSuchElementException::new);
+ }
+
+ /** Alternative to clear(). Allows use as an AutoCloseable in try-with-resources. */
+ @Override
+ public void close() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.close();
+ }
+ }
+
+ /**
+ * Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has
+ * any child vectors, they will also be cleared.
+ */
+ @Override
+ public void clear() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.clear();
+ }
+ }
+
+ /**
+ * Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer
+ * capacities will remain unchanged and any previous data will be zeroed out. This includes
+ * buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be
+ * reset.
+ */
+ @Override
+ public void reset() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reset();
+ }
+ valueCount = 0;
+ }
+
+ /**
+ * Get information about how this field is materialized.
+ *
+ * @return the field corresponding to this vector
+ */
+ @Override
+ public Field getField() {
+ return field;
+ }
+
+ @Override
+ public MinorType getMinorType() {
+ return MinorType.RUNENDENCODED;
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(BufferAllocator allocator) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(BufferAllocator)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator) {
+ return getTransferPair(ref, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator) {
+ return getTransferPair(field, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(String, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(Field, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * Makes a new transfer pair used to transfer underlying buffers.
+ *
+ * @param target the target for the transfer
+ * @return a new {@link org.apache.arrow.vector.util.TransferPair transfer pair} that is used to
+ * transfer underlying buffers into the target vector.
+ */
+ @Override
+ public TransferPair makeTransferPair(ValueVector target) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support makeTransferPair(ValueVector)");
+ }
+
+ /**
+ * Get a reader for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.reader.FieldReader field reader} that supports
+ * reading values from this vector.
+ */
+ @Override
+ public FieldReader getReader() {
+ return null; // TODO
+ }
+
+ /**
+ * Get a writer for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.writer.FieldWriter field writer} that supports
+ * writing values to this vector.
+ */
+ public FieldWriter getWriter() {
+ return null; // TODO
+ }
+
+ /**
+ * Get the number of bytes used by this vector.
+ *
+ * @return the number of bytes that is used by this vector instance.
+ */
+ @Override
+ public int getBufferSize() {
+ int bufferSize = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ bufferSize += v.getBufferSize();
+ }
+ return bufferSize;
+ }
+
+ /**
+ * Returns the number of bytes that is used by this vector if it holds the given number of values.
+ * The result will be the same as if setValueCount() were called, followed by calling
+ * getBufferSize(), but without any of the closing side-effects that setValueCount() implies wrt
+ * finishing off the population of a vector. Some operations might wish to use this to determine
+ * how much memory has been used by a vector so far, even though it is not finished being
+ * populated.
+ *
+ * @param valueCount the number of values to assume this vector contains
+ * @return the buffer size if this vector is holding valueCount values
+ */
+ @Override
+ public int getBufferSizeFor(int valueCount) {
+ return 0;
+ }
+
+ /**
+ * Return the underlying buffers associated with this vector. Note that this doesn't impact the
+ * reference counts for this buffer so it only should be used for in-context access. Also note
+ * that this buffer changes regularly thus external classes shouldn't hold a reference to it
+ * (unless they change it).
+ *
+ * @param clear Whether to clear vector before returning; the buffers will still be refcounted;
+ * but the returned array will be the only reference to them
+ * @return The underlying {@link ArrowBuf buffers} that is used by this vector instance.
+ */
+ @Override
+ public ArrowBuf[] getBuffers(boolean clear) {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with validity vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getValidityBuffer() {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with data vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getDataBuffer() {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with offset vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getOffsetBuffer() {
+ return null;
+ }
+
+ /**
+ * Gets the number of values.
+ *
+ * @return number of values in the vector
+ */
+ @Override
+ public int getValueCount() {
+ return valueCount;
+ }
+
+ /** Set number of values in the vector. */
+ @Override
+ public void setValueCount(int valueCount) {
+ this.valueCount = valueCount;
+ }
+
+ /**
+ * Get friendly type object from the vector.
+ *
+ * @param index index of object to get
+ * @return friendly type object
+ */
+ @Override
+ public Object getObject(int index) {
+ checkIndex(index);
+ int physicalIndex = getPhysicalIndex(index);
+ return valuesVector.getObject(physicalIndex);
+ }
+
+ /**
+ * Get the run end of giving index.
+ *
+ * @param index index of the run end to get
+ * @return the run end of giving index
+ */
+ public int getRunEnd(int index) {
+ checkIndex(index);
+ int physicalIndex = getPhysicalIndex(index);
+ return (int) ((BaseIntVector) runEndsVector).getValueAsLong(physicalIndex);
+ }
+
+ /**
+ * Returns number of null elements in the vector.
+ *
+ * @return number of null elements
+ */
+ @Override
+ public int getNullCount() {
+ // Null count is always 0 for run-end encoded array
+ return 0;
+ }
+
+ /**
+ * Check whether an element in the vector is null.
+ *
+ * @param index index to check for null
+ * @return true if element is null
+ */
+ @Override
+ public boolean isNull(int index) {
+ int physicalIndex = getPhysicalIndex(runEndsVector, index);
+ return valuesVector.isNull(physicalIndex);
+ }
+
+ /** Returns hashCode of element in index with the default hasher. */
+ @Override
+ public int hashCode(int index) {
+ return hashCode(index, null);
+ }
+
+ /** Returns hashCode of element in index with the given hasher. */
+ @Override
+ public int hashCode(int index, ArrowBufHasher hasher) {
+ int hash = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ if (index < v.getValueCount()) {
+ hash = ByteFunctionHelpers.combineHash(hash, v.hashCode(index, hasher));
+ }
+ }
+ return hash;
+ }
+
+ /**
+ * Accept a generic {@link VectorVisitor} and return the result.
+ *
+ * @param <OUT> the output result type.
+ * @param <IN> the input data together with visitor.
+ */
+ @Override
+ public <OUT, IN> OUT accept(VectorVisitor<OUT, IN> visitor, IN value) {
+ return visitor.visit(this, value);
+ }
+
+ /**
+ * Gets the name of the vector.
+ *
+ * @return the name of the vector.
+ */
+ @Override
+ public String getName() {
+ return this.field.getName();
+ }
+
+ @Override
+ public Iterator<ValueVector> iterator() {
+ return Collections.<ValueVector>unmodifiableCollection(getChildrenFromFields()).iterator();
+ }
+
+ /** FieldVector interface */
+
+ /**
+ * Initializes the child vectors to be later loaded with loadBuffers.
+ *
+ * @param children the schema containing the run_ends column first and the values column second
+ */
+ @Override
+ public void initializeChildrenFromFields(List<Field> children) {
+ checkArgument(
+ children.size() == 2,
+ "Run-end encoded vectors must have two child Fields. Found: %s",
+ children.isEmpty() ? "none" : children);
+ checkArgument(
+ Arrays.asList(
+ MinorType.SMALLINT.getType(), MinorType.INT.getType(), MinorType.BIGINT.getType())
+ .contains(children.get(0).getType()),
+ "The first field represents the run-end vector and must be of type int "
+ + "with size 16, 32, or 64 bits. Found: %s",
+ children.get(0).getType());
+ runEndsVector = (BaseIntVector) children.get(0).createVector(allocator);
+ valuesVector = children.get(1).createVector(allocator);
+ field = new Field(field.getName(), field.getFieldType(), children);
+ }
+
+ /**
+ * The returned list is the same size as the list passed to initializeChildrenFromFields.
+ *
+ * @return the children according to schema (empty for primitive types)
+ */
+ @Override
+ public List<FieldVector> getChildrenFromFields() {
+ return Arrays.asList(runEndsVector, valuesVector);
+ }
+
+ /**
+ * Loads data in the vectors. (ownBuffers must be the same size as getFieldVectors())
+ *
+ * @param fieldNode the fieldNode
+ * @param ownBuffers the buffers for this Field (own buffers only, children not included)
+ */
+ @Override
+ public void loadFieldBuffers(ArrowFieldNode fieldNode, List<ArrowBuf> ownBuffers) {
+ throw new UnsupportedOperationException(
+ "Run-end encoded vectors do not have any associated buffers.");
+ }
+
+ /**
+ * Get the buffers of the fields, (same size as getFieldVectors() since it is their content).
+ *
+ * @return the buffers containing the data for this vector (ready for reading)
+ */
+ @Override
+ public List<ArrowBuf> getFieldBuffers() {
+ return List.of();
+ }
+
+ /**
+ * Get the inner vectors.
+ *
+ * @return the inner vectors for this field as defined by the TypeLayout
+ * @deprecated This API will be removed as the current implementations no longer support inner
+ * vectors.
+ */
+ @Deprecated
+ @Override
+ public List<BufferBacked> getFieldInnerVectors() {
+ throw new UnsupportedOperationException("There are no inner vectors. Use getFieldBuffers().");
+ }
+
+ /**
+ * Gets the starting address of the underlying buffer associated with validity vector.
+ *
+ * @return buffer address
+ */
+ @Override
+ public long getValidityBufferAddress() {
+ throw new UnsupportedOperationException(
+ "Run-end encoded vectors do not have a validity buffer.");
+ }
+
+ /**
+ * Gets the starting address of the underlying buffer associated with data vector.
+ *
+ * @return buffer address
+ */
+ @Override
+ public long getDataBufferAddress() {
+ throw new UnsupportedOperationException("Run-end encoded vectors do not have a data buffer.");
+ }
+
+ /**
+ * Gets the starting address of the underlying buffer associated with offset vector.
+ *
+ * @return buffer address
+ */
+ @Override
+ public long getOffsetBufferAddress() {
+ throw new UnsupportedOperationException(
+ "Run-end encoded vectors do not have an offset buffer.");
+ }
+
+ /**
+ * Set the element at the given index to null.
+ *
+ * @param index the value to change
+ */
+ @Override
+ public void setNull(int index) {
+ throw new UnsupportedOperationException(
+ "Run-end encoded vectors do not have a validity buffer.");
+ }
+
+ public FieldVector getRunEndsVector() {
+ return runEndsVector;
+ }
+
+ public FieldVector getValuesVector() {
+ return valuesVector;
+ }
+
+ private void checkIndex(int logicalIndex) {
+ if (logicalIndex < 0 || logicalIndex >= valueCount) {
+ throw new IndexOutOfBoundsException();
Review Comment:
Include the index in the exception message
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,670 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ return null;
+ }
+
+ /**
+ * Set the initial record capacity.
+ *
+ * @param numRecords the initial record capacity.
+ */
+ @Override
+ public void setInitialCapacity(int numRecords) {}
+
+ /**
+ * Returns the maximum number of values that can be stored in this vector instance.
+ *
+ * @return the maximum number of values that can be stored in this vector instance.
+ */
+ @Override
+ public int getValueCapacity() {
+ return getChildrenFromFields().stream()
+ .mapToInt(item -> item != null ? item.getValueCapacity() : 0)
+ .min()
+ .orElseThrow(NoSuchElementException::new);
+ }
+
+ /** Alternative to clear(). Allows use as an AutoCloseable in try-with-resources. */
+ @Override
+ public void close() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.close();
+ }
+ }
+
+ /**
+ * Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has
+ * any child vectors, they will also be cleared.
+ */
+ @Override
+ public void clear() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.clear();
+ }
+ }
+
+ /**
+ * Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer
+ * capacities will remain unchanged and any previous data will be zeroed out. This includes
+ * buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be
+ * reset.
+ */
+ @Override
+ public void reset() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reset();
+ }
+ valueCount = 0;
+ }
+
+ /**
+ * Get information about how this field is materialized.
+ *
+ * @return the field corresponding to this vector
+ */
+ @Override
+ public Field getField() {
+ return field;
+ }
+
+ @Override
+ public MinorType getMinorType() {
+ return MinorType.RUNENDENCODED;
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(BufferAllocator allocator) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(BufferAllocator)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator) {
+ return getTransferPair(ref, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator) {
+ return getTransferPair(field, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(String, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(Field, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * Makes a new transfer pair used to transfer underlying buffers.
+ *
+ * @param target the target for the transfer
+ * @return a new {@link org.apache.arrow.vector.util.TransferPair transfer pair} that is used to
+ * transfer underlying buffers into the target vector.
+ */
+ @Override
+ public TransferPair makeTransferPair(ValueVector target) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support makeTransferPair(ValueVector)");
+ }
+
+ /**
+ * Get a reader for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.reader.FieldReader field reader} that supports
+ * reading values from this vector.
+ */
+ @Override
+ public FieldReader getReader() {
+ return null; // TODO
+ }
+
+ /**
+ * Get a writer for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.writer.FieldWriter field writer} that supports
+ * writing values to this vector.
+ */
+ public FieldWriter getWriter() {
+ return null; // TODO
+ }
+
+ /**
+ * Get the number of bytes used by this vector.
+ *
+ * @return the number of bytes that is used by this vector instance.
+ */
+ @Override
+ public int getBufferSize() {
+ int bufferSize = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ bufferSize += v.getBufferSize();
+ }
+ return bufferSize;
+ }
+
+ /**
+ * Returns the number of bytes that is used by this vector if it holds the given number of values.
+ * The result will be the same as if setValueCount() were called, followed by calling
+ * getBufferSize(), but without any of the closing side-effects that setValueCount() implies wrt
+ * finishing off the population of a vector. Some operations might wish to use this to determine
+ * how much memory has been used by a vector so far, even though it is not finished being
+ * populated.
+ *
+ * @param valueCount the number of values to assume this vector contains
+ * @return the buffer size if this vector is holding valueCount values
+ */
+ @Override
+ public int getBufferSizeFor(int valueCount) {
+ return 0;
+ }
+
+ /**
+ * Return the underlying buffers associated with this vector. Note that this doesn't impact the
+ * reference counts for this buffer so it only should be used for in-context access. Also note
+ * that this buffer changes regularly thus external classes shouldn't hold a reference to it
+ * (unless they change it).
+ *
+ * @param clear Whether to clear vector before returning; the buffers will still be refcounted;
+ * but the returned array will be the only reference to them
+ * @return The underlying {@link ArrowBuf buffers} that is used by this vector instance.
+ */
+ @Override
+ public ArrowBuf[] getBuffers(boolean clear) {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with validity vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getValidityBuffer() {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with data vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getDataBuffer() {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with offset vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getOffsetBuffer() {
+ return null;
+ }
+
+ /**
+ * Gets the number of values.
+ *
+ * @return number of values in the vector
+ */
+ @Override
+ public int getValueCount() {
+ return valueCount;
+ }
+
+ /** Set number of values in the vector. */
+ @Override
+ public void setValueCount(int valueCount) {
+ this.valueCount = valueCount;
+ }
+
+ /**
+ * Get friendly type object from the vector.
+ *
+ * @param index index of object to get
+ * @return friendly type object
+ */
+ @Override
+ public Object getObject(int index) {
+ checkIndex(index);
+ int physicalIndex = getPhysicalIndex(index);
+ return valuesVector.getObject(physicalIndex);
+ }
+
+ /**
+ * Get the run end of giving index.
+ *
+ * @param index index of the run end to get
+ * @return the run end of giving index
+ */
+ public int getRunEnd(int index) {
+ checkIndex(index);
+ int physicalIndex = getPhysicalIndex(index);
+ return (int) ((BaseIntVector) runEndsVector).getValueAsLong(physicalIndex);
+ }
+
+ /**
+ * Returns number of null elements in the vector.
+ *
+ * @return number of null elements
+ */
+ @Override
+ public int getNullCount() {
+ // Null count is always 0 for run-end encoded array
+ return 0;
+ }
+
+ /**
+ * Check whether an element in the vector is null.
+ *
+ * @param index index to check for null
+ * @return true if element is null
+ */
+ @Override
+ public boolean isNull(int index) {
+ int physicalIndex = getPhysicalIndex(runEndsVector, index);
+ return valuesVector.isNull(physicalIndex);
+ }
+
+ /** Returns hashCode of element in index with the default hasher. */
+ @Override
+ public int hashCode(int index) {
+ return hashCode(index, null);
+ }
+
+ /** Returns hashCode of element in index with the given hasher. */
+ @Override
+ public int hashCode(int index, ArrowBufHasher hasher) {
+ int hash = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ if (index < v.getValueCount()) {
+ hash = ByteFunctionHelpers.combineHash(hash, v.hashCode(index, hasher));
+ }
+ }
+ return hash;
+ }
+
+ /**
+ * Accept a generic {@link VectorVisitor} and return the result.
+ *
+ * @param <OUT> the output result type.
+ * @param <IN> the input data together with visitor.
+ */
+ @Override
+ public <OUT, IN> OUT accept(VectorVisitor<OUT, IN> visitor, IN value) {
+ return visitor.visit(this, value);
+ }
+
+ /**
+ * Gets the name of the vector.
+ *
+ * @return the name of the vector.
+ */
+ @Override
+ public String getName() {
+ return this.field.getName();
+ }
+
+ @Override
+ public Iterator<ValueVector> iterator() {
+ return Collections.<ValueVector>unmodifiableCollection(getChildrenFromFields()).iterator();
+ }
+
+ /** FieldVector interface */
+
+ /**
+ * Initializes the child vectors to be later loaded with loadBuffers.
+ *
+ * @param children the schema containing the run_ends column first and the values column second
+ */
+ @Override
+ public void initializeChildrenFromFields(List<Field> children) {
+ checkArgument(
+ children.size() == 2,
+ "Run-end encoded vectors must have two child Fields. Found: %s",
+ children.isEmpty() ? "none" : children);
+ checkArgument(
+ Arrays.asList(
+ MinorType.SMALLINT.getType(), MinorType.INT.getType(), MinorType.BIGINT.getType())
+ .contains(children.get(0).getType()),
+ "The first field represents the run-end vector and must be of type int "
+ + "with size 16, 32, or 64 bits. Found: %s",
+ children.get(0).getType());
+ runEndsVector = (BaseIntVector) children.get(0).createVector(allocator);
+ valuesVector = children.get(1).createVector(allocator);
+ field = new Field(field.getName(), field.getFieldType(), children);
+ }
+
+ /**
+ * The returned list is the same size as the list passed to initializeChildrenFromFields.
+ *
+ * @return the children according to schema (empty for primitive types)
+ */
+ @Override
+ public List<FieldVector> getChildrenFromFields() {
+ return Arrays.asList(runEndsVector, valuesVector);
+ }
+
+ /**
+ * Loads data in the vectors. (ownBuffers must be the same size as getFieldVectors())
+ *
+ * @param fieldNode the fieldNode
+ * @param ownBuffers the buffers for this Field (own buffers only, children not included)
+ */
+ @Override
+ public void loadFieldBuffers(ArrowFieldNode fieldNode, List<ArrowBuf> ownBuffers) {
+ throw new UnsupportedOperationException(
Review Comment:
Maybe only throw if ownBuffers is not empty
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,670 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ return null;
+ }
+
+ /**
+ * Set the initial record capacity.
+ *
+ * @param numRecords the initial record capacity.
+ */
+ @Override
+ public void setInitialCapacity(int numRecords) {}
+
+ /**
+ * Returns the maximum number of values that can be stored in this vector instance.
+ *
+ * @return the maximum number of values that can be stored in this vector instance.
+ */
+ @Override
+ public int getValueCapacity() {
+ return getChildrenFromFields().stream()
+ .mapToInt(item -> item != null ? item.getValueCapacity() : 0)
+ .min()
+ .orElseThrow(NoSuchElementException::new);
+ }
+
+ /** Alternative to clear(). Allows use as an AutoCloseable in try-with-resources. */
+ @Override
+ public void close() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.close();
+ }
+ }
+
+ /**
+ * Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has
+ * any child vectors, they will also be cleared.
+ */
+ @Override
+ public void clear() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.clear();
+ }
+ }
+
+ /**
+ * Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer
+ * capacities will remain unchanged and any previous data will be zeroed out. This includes
+ * buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be
+ * reset.
+ */
+ @Override
+ public void reset() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reset();
+ }
+ valueCount = 0;
+ }
+
+ /**
+ * Get information about how this field is materialized.
+ *
+ * @return the field corresponding to this vector
+ */
+ @Override
+ public Field getField() {
+ return field;
+ }
+
+ @Override
+ public MinorType getMinorType() {
+ return MinorType.RUNENDENCODED;
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(BufferAllocator allocator) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(BufferAllocator)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator) {
+ return getTransferPair(ref, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator) {
+ return getTransferPair(field, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(String, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(Field, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * Makes a new transfer pair used to transfer underlying buffers.
+ *
+ * @param target the target for the transfer
+ * @return a new {@link org.apache.arrow.vector.util.TransferPair transfer pair} that is used to
+ * transfer underlying buffers into the target vector.
+ */
+ @Override
+ public TransferPair makeTransferPair(ValueVector target) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support makeTransferPair(ValueVector)");
+ }
+
+ /**
+ * Get a reader for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.reader.FieldReader field reader} that supports
+ * reading values from this vector.
+ */
+ @Override
+ public FieldReader getReader() {
+ return null; // TODO
+ }
+
+ /**
+ * Get a writer for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.writer.FieldWriter field writer} that supports
+ * writing values to this vector.
+ */
+ public FieldWriter getWriter() {
+ return null; // TODO
Review Comment:
Same here and elsewhere, throw instead of returning null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "ViggoC (via GitHub)" <gi...@apache.org>.
ViggoC commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1765129961
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,683 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param runEndsVector The vector represents run ends. Only Zero vector or type int vector with
+ * size 16, 32 is allowed
+ * @param valuesVector The vector represents values
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ return null;
Review Comment:
fixed in https://github.com/apache/arrow/pull/43888/commits/2b24d39257bbdbee8697532d766631ba570224dc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "ViggoC (via GitHub)" <gi...@apache.org>.
ViggoC commented on PR #43888:
URL: https://github.com/apache/arrow/pull/43888#issuecomment-2362882089
@lidavidm @vibhatha Thank you for your helps on my first PR of arrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1767921613
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,686 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param runEndsVector The vector represents run ends. Only Zero vector or type int vector with
+ * size 16, 32 is allowed
+ * @param valuesVector The vector represents values
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Set the initial record capacity.
+ *
+ * @param numRecords the initial record capacity.
+ */
+ @Override
+ public void setInitialCapacity(int numRecords) {}
+
+ /**
+ * Returns the maximum number of values that can be stored in this vector instance.
+ *
+ * @return the maximum number of values that can be stored in this vector instance.
+ */
+ @Override
+ public int getValueCapacity() {
+ return getChildrenFromFields().stream()
+ .mapToInt(item -> item != null ? item.getValueCapacity() : 0)
+ .min()
+ .orElseThrow(NoSuchElementException::new);
+ }
+
+ /** Alternative to clear(). Allows use as an AutoCloseable in try-with-resources. */
+ @Override
+ public void close() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.close();
+ }
+ }
+
+ /**
+ * Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has
+ * any child vectors, they will also be cleared.
+ */
+ @Override
+ public void clear() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.clear();
+ }
+ }
+
+ /**
+ * Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer
+ * capacities will remain unchanged and any previous data will be zeroed out. This includes
+ * buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be
+ * reset.
+ */
+ @Override
+ public void reset() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reset();
+ }
+ valueCount = 0;
+ }
+
+ /**
+ * Get information about how this field is materialized.
+ *
+ * @return the field corresponding to this vector
+ */
+ @Override
+ public Field getField() {
+ return field;
+ }
+
+ @Override
+ public MinorType getMinorType() {
+ return MinorType.RUNENDENCODED;
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(BufferAllocator allocator) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(BufferAllocator)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator) {
+ return getTransferPair(ref, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator) {
+ return getTransferPair(field, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(String, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(Field, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * Makes a new transfer pair used to transfer underlying buffers.
+ *
+ * @param target the target for the transfer
+ * @return a new {@link org.apache.arrow.vector.util.TransferPair transfer pair} that is used to
+ * transfer underlying buffers into the target vector.
+ */
+ @Override
+ public TransferPair makeTransferPair(ValueVector target) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support makeTransferPair(ValueVector)");
+ }
+
+ /**
+ * Get a reader for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.reader.FieldReader field reader} that supports
+ * reading values from this vector.
+ */
+ @Override
+ public FieldReader getReader() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get a writer for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.writer.FieldWriter field writer} that supports
+ * writing values to this vector.
+ */
+ public FieldWriter getWriter() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get the number of bytes used by this vector.
+ *
+ * @return the number of bytes that is used by this vector instance.
+ */
+ @Override
+ public int getBufferSize() {
+ int bufferSize = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ bufferSize += v.getBufferSize();
+ }
+ return bufferSize;
+ }
+
+ /**
+ * Returns the number of bytes that is used by this vector if it holds the given number of values.
+ * The result will be the same as if setValueCount() were called, followed by calling
+ * getBufferSize(), but without any of the closing side-effects that setValueCount() implies wrt
+ * finishing off the population of a vector. Some operations might wish to use this to determine
+ * how much memory has been used by a vector so far, even though it is not finished being
+ * populated.
+ *
+ * @param valueCount the number of values to assume this vector contains
+ * @return the buffer size if this vector is holding valueCount values
+ */
+ @Override
+ public int getBufferSizeFor(int valueCount) {
+ return 0;
+ }
+
+ /**
+ * Return the underlying buffers associated with this vector. Note that this doesn't impact the
+ * reference counts for this buffer so it only should be used for in-context access. Also note
+ * that this buffer changes regularly thus external classes shouldn't hold a reference to it
+ * (unless they change it).
+ *
+ * @param clear Whether to clear vector before returning; the buffers will still be refcounted;
+ * but the returned array will be the only reference to them
+ * @return The underlying {@link ArrowBuf buffers} that is used by this vector instance.
+ */
+ @Override
+ public ArrowBuf[] getBuffers(boolean clear) {
+ return new ArrowBuf[0];
+ }
+
+ /**
+ * Gets the underlying buffer associated with validity vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getValidityBuffer() {
+ throw new UnsupportedOperationException(
+ "Run-end encoded vectors do not have a validity buffer.");
+ }
+
+ /**
+ * Gets the underlying buffer associated with data vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getDataBuffer() {
+ throw new UnsupportedOperationException(
+ "Run-end encoded vectors do not have a data buffer.");
+ }
+
+ /**
+ * Gets the underlying buffer associated with offset vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getOffsetBuffer() {
+ throw new UnsupportedOperationException(
+ "Run-end encoded vectors do not have a offset buffer.");
+ }
Review Comment:
```suggestion
@Override
public ArrowBuf getDataBuffer() {
throw new UnsupportedOperationException("Run-end encoded vectors do not have a data buffer.");
}
/**
* Gets the underlying buffer associated with offset vector.
*
* @return buffer
*/
@Override
public ArrowBuf getOffsetBuffer() {
throw new UnsupportedOperationException("Run-end encoded vectors do not have a offset buffer.");
}
```
Make spotless happy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "ViggoC (via GitHub)" <gi...@apache.org>.
ViggoC commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1765149914
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,683 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param runEndsVector The vector represents run ends. Only Zero vector or type int vector with
+ * size 16, 32 is allowed
+ * @param valuesVector The vector represents values
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ return null;
+ }
+
+ /**
+ * Set the initial record capacity.
+ *
+ * @param numRecords the initial record capacity.
+ */
+ @Override
+ public void setInitialCapacity(int numRecords) {}
+
+ /**
+ * Returns the maximum number of values that can be stored in this vector instance.
+ *
+ * @return the maximum number of values that can be stored in this vector instance.
+ */
+ @Override
+ public int getValueCapacity() {
+ return getChildrenFromFields().stream()
+ .mapToInt(item -> item != null ? item.getValueCapacity() : 0)
+ .min()
+ .orElseThrow(NoSuchElementException::new);
+ }
+
+ /** Alternative to clear(). Allows use as an AutoCloseable in try-with-resources. */
+ @Override
+ public void close() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.close();
+ }
+ }
+
+ /**
+ * Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has
+ * any child vectors, they will also be cleared.
+ */
+ @Override
+ public void clear() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.clear();
+ }
+ }
+
+ /**
+ * Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer
+ * capacities will remain unchanged and any previous data will be zeroed out. This includes
+ * buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be
+ * reset.
+ */
+ @Override
+ public void reset() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reset();
+ }
+ valueCount = 0;
+ }
+
+ /**
+ * Get information about how this field is materialized.
+ *
+ * @return the field corresponding to this vector
+ */
+ @Override
+ public Field getField() {
+ return field;
+ }
+
+ @Override
+ public MinorType getMinorType() {
+ return MinorType.RUNENDENCODED;
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(BufferAllocator allocator) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(BufferAllocator)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator) {
+ return getTransferPair(ref, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator) {
+ return getTransferPair(field, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(String, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(Field, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * Makes a new transfer pair used to transfer underlying buffers.
+ *
+ * @param target the target for the transfer
+ * @return a new {@link org.apache.arrow.vector.util.TransferPair transfer pair} that is used to
+ * transfer underlying buffers into the target vector.
+ */
+ @Override
+ public TransferPair makeTransferPair(ValueVector target) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support makeTransferPair(ValueVector)");
+ }
+
+ /**
+ * Get a reader for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.reader.FieldReader field reader} that supports
+ * reading values from this vector.
+ */
+ @Override
+ public FieldReader getReader() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get a writer for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.writer.FieldWriter field writer} that supports
+ * writing values to this vector.
+ */
+ public FieldWriter getWriter() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get the number of bytes used by this vector.
+ *
+ * @return the number of bytes that is used by this vector instance.
+ */
+ @Override
+ public int getBufferSize() {
+ int bufferSize = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ bufferSize += v.getBufferSize();
+ }
+ return bufferSize;
+ }
+
+ /**
+ * Returns the number of bytes that is used by this vector if it holds the given number of values.
+ * The result will be the same as if setValueCount() were called, followed by calling
+ * getBufferSize(), but without any of the closing side-effects that setValueCount() implies wrt
+ * finishing off the population of a vector. Some operations might wish to use this to determine
+ * how much memory has been used by a vector so far, even though it is not finished being
+ * populated.
+ *
+ * @param valueCount the number of values to assume this vector contains
+ * @return the buffer size if this vector is holding valueCount values
+ */
+ @Override
+ public int getBufferSizeFor(int valueCount) {
+ return 0;
+ }
+
+ /**
+ * Return the underlying buffers associated with this vector. Note that this doesn't impact the
+ * reference counts for this buffer so it only should be used for in-context access. Also note
+ * that this buffer changes regularly thus external classes shouldn't hold a reference to it
+ * (unless they change it).
+ *
+ * @param clear Whether to clear vector before returning; the buffers will still be refcounted;
+ * but the returned array will be the only reference to them
+ * @return The underlying {@link ArrowBuf buffers} that is used by this vector instance.
+ */
+ @Override
+ public ArrowBuf[] getBuffers(boolean clear) {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with validity vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getValidityBuffer() {
+ return null;
Review Comment:
Oh, i see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1764984338
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,683 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param runEndsVector The vector represents run ends. Only Zero vector or type int vector with
+ * size 16, 32 is allowed
+ * @param valuesVector The vector represents values
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ return null;
Review Comment:
nit: shall we throw unsupported here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1765135743
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,683 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param runEndsVector The vector represents run ends. Only Zero vector or type int vector with
+ * size 16, 32 is allowed
+ * @param valuesVector The vector represents values
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ return null;
+ }
+
+ /**
+ * Set the initial record capacity.
+ *
+ * @param numRecords the initial record capacity.
+ */
+ @Override
+ public void setInitialCapacity(int numRecords) {}
+
+ /**
+ * Returns the maximum number of values that can be stored in this vector instance.
+ *
+ * @return the maximum number of values that can be stored in this vector instance.
+ */
+ @Override
+ public int getValueCapacity() {
+ return getChildrenFromFields().stream()
+ .mapToInt(item -> item != null ? item.getValueCapacity() : 0)
+ .min()
+ .orElseThrow(NoSuchElementException::new);
+ }
+
+ /** Alternative to clear(). Allows use as an AutoCloseable in try-with-resources. */
+ @Override
+ public void close() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.close();
+ }
+ }
+
+ /**
+ * Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has
+ * any child vectors, they will also be cleared.
+ */
+ @Override
+ public void clear() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.clear();
+ }
+ }
+
+ /**
+ * Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer
+ * capacities will remain unchanged and any previous data will be zeroed out. This includes
+ * buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be
+ * reset.
+ */
+ @Override
+ public void reset() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reset();
+ }
+ valueCount = 0;
+ }
+
+ /**
+ * Get information about how this field is materialized.
+ *
+ * @return the field corresponding to this vector
+ */
+ @Override
+ public Field getField() {
+ return field;
+ }
+
+ @Override
+ public MinorType getMinorType() {
+ return MinorType.RUNENDENCODED;
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(BufferAllocator allocator) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(BufferAllocator)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator) {
+ return getTransferPair(ref, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator) {
+ return getTransferPair(field, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(String, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(Field, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * Makes a new transfer pair used to transfer underlying buffers.
+ *
+ * @param target the target for the transfer
+ * @return a new {@link org.apache.arrow.vector.util.TransferPair transfer pair} that is used to
+ * transfer underlying buffers into the target vector.
+ */
+ @Override
+ public TransferPair makeTransferPair(ValueVector target) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support makeTransferPair(ValueVector)");
+ }
+
+ /**
+ * Get a reader for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.reader.FieldReader field reader} that supports
+ * reading values from this vector.
+ */
+ @Override
+ public FieldReader getReader() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get a writer for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.writer.FieldWriter field writer} that supports
+ * writing values to this vector.
+ */
+ public FieldWriter getWriter() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get the number of bytes used by this vector.
+ *
+ * @return the number of bytes that is used by this vector instance.
+ */
+ @Override
+ public int getBufferSize() {
+ int bufferSize = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ bufferSize += v.getBufferSize();
+ }
+ return bufferSize;
+ }
+
+ /**
+ * Returns the number of bytes that is used by this vector if it holds the given number of values.
+ * The result will be the same as if setValueCount() were called, followed by calling
+ * getBufferSize(), but without any of the closing side-effects that setValueCount() implies wrt
+ * finishing off the population of a vector. Some operations might wish to use this to determine
+ * how much memory has been used by a vector so far, even though it is not finished being
+ * populated.
+ *
+ * @param valueCount the number of values to assume this vector contains
+ * @return the buffer size if this vector is holding valueCount values
+ */
+ @Override
+ public int getBufferSizeFor(int valueCount) {
+ return 0;
+ }
+
+ /**
+ * Return the underlying buffers associated with this vector. Note that this doesn't impact the
+ * reference counts for this buffer so it only should be used for in-context access. Also note
+ * that this buffer changes regularly thus external classes shouldn't hold a reference to it
+ * (unless they change it).
+ *
+ * @param clear Whether to clear vector before returning; the buffers will still be refcounted;
+ * but the returned array will be the only reference to them
+ * @return The underlying {@link ArrowBuf buffers} that is used by this vector instance.
+ */
+ @Override
+ public ArrowBuf[] getBuffers(boolean clear) {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with validity vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getValidityBuffer() {
+ return null;
Review Comment:
Ah I think I wasn't clear. Sorry about that. For instance, we don't have a `offsetBuffer` for StringView. And there rather than returning null, we have added `throw new UnsupportedOperationException(...)`. I was referring to that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1767906883
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,687 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param runEndsVector The vector represents run ends. Only Zero vector or type int vector with
+ * size 16, 32 is allowed
+ * @param valuesVector The vector represents values
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ ;
Review Comment:
```suggestion
```
The compiler is complaining about this line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1764990373
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,683 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param runEndsVector The vector represents run ends. Only Zero vector or type int vector with
+ * size 16, 32 is allowed
+ * @param valuesVector The vector represents values
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ return null;
+ }
+
+ /**
+ * Set the initial record capacity.
+ *
+ * @param numRecords the initial record capacity.
+ */
+ @Override
+ public void setInitialCapacity(int numRecords) {}
+
+ /**
+ * Returns the maximum number of values that can be stored in this vector instance.
+ *
+ * @return the maximum number of values that can be stored in this vector instance.
+ */
+ @Override
+ public int getValueCapacity() {
+ return getChildrenFromFields().stream()
+ .mapToInt(item -> item != null ? item.getValueCapacity() : 0)
+ .min()
+ .orElseThrow(NoSuchElementException::new);
+ }
+
+ /** Alternative to clear(). Allows use as an AutoCloseable in try-with-resources. */
+ @Override
+ public void close() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.close();
+ }
+ }
+
+ /**
+ * Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has
+ * any child vectors, they will also be cleared.
+ */
+ @Override
+ public void clear() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.clear();
+ }
+ }
+
+ /**
+ * Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer
+ * capacities will remain unchanged and any previous data will be zeroed out. This includes
+ * buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be
+ * reset.
+ */
+ @Override
+ public void reset() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reset();
+ }
+ valueCount = 0;
+ }
+
+ /**
+ * Get information about how this field is materialized.
+ *
+ * @return the field corresponding to this vector
+ */
+ @Override
+ public Field getField() {
+ return field;
+ }
+
+ @Override
+ public MinorType getMinorType() {
+ return MinorType.RUNENDENCODED;
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(BufferAllocator allocator) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(BufferAllocator)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator) {
+ return getTransferPair(ref, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator) {
+ return getTransferPair(field, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(String, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(Field, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * Makes a new transfer pair used to transfer underlying buffers.
+ *
+ * @param target the target for the transfer
+ * @return a new {@link org.apache.arrow.vector.util.TransferPair transfer pair} that is used to
+ * transfer underlying buffers into the target vector.
+ */
+ @Override
+ public TransferPair makeTransferPair(ValueVector target) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support makeTransferPair(ValueVector)");
+ }
+
+ /**
+ * Get a reader for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.reader.FieldReader field reader} that supports
+ * reading values from this vector.
+ */
+ @Override
+ public FieldReader getReader() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get a writer for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.writer.FieldWriter field writer} that supports
+ * writing values to this vector.
+ */
+ public FieldWriter getWriter() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get the number of bytes used by this vector.
+ *
+ * @return the number of bytes that is used by this vector instance.
+ */
+ @Override
+ public int getBufferSize() {
+ int bufferSize = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ bufferSize += v.getBufferSize();
+ }
+ return bufferSize;
+ }
+
+ /**
+ * Returns the number of bytes that is used by this vector if it holds the given number of values.
+ * The result will be the same as if setValueCount() were called, followed by calling
+ * getBufferSize(), but without any of the closing side-effects that setValueCount() implies wrt
+ * finishing off the population of a vector. Some operations might wish to use this to determine
+ * how much memory has been used by a vector so far, even though it is not finished being
+ * populated.
+ *
+ * @param valueCount the number of values to assume this vector contains
+ * @return the buffer size if this vector is holding valueCount values
+ */
+ @Override
+ public int getBufferSizeFor(int valueCount) {
+ return 0;
+ }
+
+ /**
+ * Return the underlying buffers associated with this vector. Note that this doesn't impact the
+ * reference counts for this buffer so it only should be used for in-context access. Also note
+ * that this buffer changes regularly thus external classes shouldn't hold a reference to it
+ * (unless they change it).
+ *
+ * @param clear Whether to clear vector before returning; the buffers will still be refcounted;
+ * but the returned array will be the only reference to them
+ * @return The underlying {@link ArrowBuf buffers} that is used by this vector instance.
+ */
+ @Override
+ public ArrowBuf[] getBuffers(boolean clear) {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with validity vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getValidityBuffer() {
+ return null;
Review Comment:
@lidavidm should this be `null` or `unsupported`? I know in definition there is no validity buffer. And at the same time, it is not part of this API though we have to extend upon the existing interfaces we have to override. Maybe they should have a default implementation. I know it is kind of out of scope here. But WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1767784995
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,683 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param runEndsVector The vector represents run ends. Only Zero vector or type int vector with
+ * size 16, 32 is allowed
+ * @param valuesVector The vector represents values
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ return null;
+ }
+
+ /**
+ * Set the initial record capacity.
+ *
+ * @param numRecords the initial record capacity.
+ */
+ @Override
+ public void setInitialCapacity(int numRecords) {}
+
+ /**
+ * Returns the maximum number of values that can be stored in this vector instance.
+ *
+ * @return the maximum number of values that can be stored in this vector instance.
+ */
+ @Override
+ public int getValueCapacity() {
+ return getChildrenFromFields().stream()
+ .mapToInt(item -> item != null ? item.getValueCapacity() : 0)
+ .min()
+ .orElseThrow(NoSuchElementException::new);
+ }
+
+ /** Alternative to clear(). Allows use as an AutoCloseable in try-with-resources. */
+ @Override
+ public void close() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.close();
+ }
+ }
+
+ /**
+ * Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has
+ * any child vectors, they will also be cleared.
+ */
+ @Override
+ public void clear() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.clear();
+ }
+ }
+
+ /**
+ * Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer
+ * capacities will remain unchanged and any previous data will be zeroed out. This includes
+ * buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be
+ * reset.
+ */
+ @Override
+ public void reset() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reset();
+ }
+ valueCount = 0;
+ }
+
+ /**
+ * Get information about how this field is materialized.
+ *
+ * @return the field corresponding to this vector
+ */
+ @Override
+ public Field getField() {
+ return field;
+ }
+
+ @Override
+ public MinorType getMinorType() {
+ return MinorType.RUNENDENCODED;
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(BufferAllocator allocator) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(BufferAllocator)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator) {
+ return getTransferPair(ref, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator) {
+ return getTransferPair(field, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(String, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(Field, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * Makes a new transfer pair used to transfer underlying buffers.
+ *
+ * @param target the target for the transfer
+ * @return a new {@link org.apache.arrow.vector.util.TransferPair transfer pair} that is used to
+ * transfer underlying buffers into the target vector.
+ */
+ @Override
+ public TransferPair makeTransferPair(ValueVector target) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support makeTransferPair(ValueVector)");
+ }
+
+ /**
+ * Get a reader for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.reader.FieldReader field reader} that supports
+ * reading values from this vector.
+ */
+ @Override
+ public FieldReader getReader() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get a writer for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.writer.FieldWriter field writer} that supports
+ * writing values to this vector.
+ */
+ public FieldWriter getWriter() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get the number of bytes used by this vector.
+ *
+ * @return the number of bytes that is used by this vector instance.
+ */
+ @Override
+ public int getBufferSize() {
+ int bufferSize = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ bufferSize += v.getBufferSize();
+ }
+ return bufferSize;
+ }
+
+ /**
+ * Returns the number of bytes that is used by this vector if it holds the given number of values.
+ * The result will be the same as if setValueCount() were called, followed by calling
+ * getBufferSize(), but without any of the closing side-effects that setValueCount() implies wrt
+ * finishing off the population of a vector. Some operations might wish to use this to determine
+ * how much memory has been used by a vector so far, even though it is not finished being
+ * populated.
+ *
+ * @param valueCount the number of values to assume this vector contains
+ * @return the buffer size if this vector is holding valueCount values
+ */
+ @Override
+ public int getBufferSizeFor(int valueCount) {
+ return 0;
+ }
+
+ /**
+ * Return the underlying buffers associated with this vector. Note that this doesn't impact the
+ * reference counts for this buffer so it only should be used for in-context access. Also note
+ * that this buffer changes regularly thus external classes shouldn't hold a reference to it
+ * (unless they change it).
+ *
+ * @param clear Whether to clear vector before returning; the buffers will still be refcounted;
+ * but the returned array will be the only reference to them
+ * @return The underlying {@link ArrowBuf buffers} that is used by this vector instance.
+ */
+ @Override
+ public ArrowBuf[] getBuffers(boolean clear) {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with validity vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getValidityBuffer() {
+ return null;
Review Comment:
Oops. Yes, please make all of these (except for getBuffers) throw instead of returning null. (And I think `getBuffers` should maybe return an empty array instead of null.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1764992275
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,683 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param runEndsVector The vector represents run ends. Only Zero vector or type int vector with
+ * size 16, 32 is allowed
+ * @param valuesVector The vector represents values
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ return null;
+ }
+
+ /**
+ * Set the initial record capacity.
+ *
+ * @param numRecords the initial record capacity.
+ */
+ @Override
+ public void setInitialCapacity(int numRecords) {}
+
+ /**
+ * Returns the maximum number of values that can be stored in this vector instance.
+ *
+ * @return the maximum number of values that can be stored in this vector instance.
+ */
+ @Override
+ public int getValueCapacity() {
+ return getChildrenFromFields().stream()
+ .mapToInt(item -> item != null ? item.getValueCapacity() : 0)
+ .min()
+ .orElseThrow(NoSuchElementException::new);
+ }
+
+ /** Alternative to clear(). Allows use as an AutoCloseable in try-with-resources. */
+ @Override
+ public void close() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.close();
+ }
+ }
+
+ /**
+ * Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has
+ * any child vectors, they will also be cleared.
+ */
+ @Override
+ public void clear() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.clear();
+ }
+ }
+
+ /**
+ * Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer
+ * capacities will remain unchanged and any previous data will be zeroed out. This includes
+ * buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be
+ * reset.
+ */
+ @Override
+ public void reset() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reset();
+ }
+ valueCount = 0;
+ }
+
+ /**
+ * Get information about how this field is materialized.
+ *
+ * @return the field corresponding to this vector
+ */
+ @Override
+ public Field getField() {
+ return field;
+ }
+
+ @Override
+ public MinorType getMinorType() {
+ return MinorType.RUNENDENCODED;
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(BufferAllocator allocator) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(BufferAllocator)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator) {
+ return getTransferPair(ref, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator) {
+ return getTransferPair(field, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(String, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(Field, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * Makes a new transfer pair used to transfer underlying buffers.
+ *
+ * @param target the target for the transfer
+ * @return a new {@link org.apache.arrow.vector.util.TransferPair transfer pair} that is used to
+ * transfer underlying buffers into the target vector.
+ */
+ @Override
+ public TransferPair makeTransferPair(ValueVector target) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support makeTransferPair(ValueVector)");
+ }
+
+ /**
+ * Get a reader for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.reader.FieldReader field reader} that supports
+ * reading values from this vector.
+ */
+ @Override
+ public FieldReader getReader() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get a writer for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.writer.FieldWriter field writer} that supports
+ * writing values to this vector.
+ */
+ public FieldWriter getWriter() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get the number of bytes used by this vector.
+ *
+ * @return the number of bytes that is used by this vector instance.
+ */
+ @Override
+ public int getBufferSize() {
+ int bufferSize = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ bufferSize += v.getBufferSize();
+ }
+ return bufferSize;
+ }
+
+ /**
+ * Returns the number of bytes that is used by this vector if it holds the given number of values.
+ * The result will be the same as if setValueCount() were called, followed by calling
+ * getBufferSize(), but without any of the closing side-effects that setValueCount() implies wrt
+ * finishing off the population of a vector. Some operations might wish to use this to determine
+ * how much memory has been used by a vector so far, even though it is not finished being
+ * populated.
+ *
+ * @param valueCount the number of values to assume this vector contains
+ * @return the buffer size if this vector is holding valueCount values
+ */
+ @Override
+ public int getBufferSizeFor(int valueCount) {
+ return 0;
+ }
+
+ /**
+ * Return the underlying buffers associated with this vector. Note that this doesn't impact the
+ * reference counts for this buffer so it only should be used for in-context access. Also note
+ * that this buffer changes regularly thus external classes shouldn't hold a reference to it
+ * (unless they change it).
+ *
+ * @param clear Whether to clear vector before returning; the buffers will still be refcounted;
+ * but the returned array will be the only reference to them
+ * @return The underlying {@link ArrowBuf buffers} that is used by this vector instance.
+ */
+ @Override
+ public ArrowBuf[] getBuffers(boolean clear) {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with validity vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getValidityBuffer() {
+ return null;
Review Comment:
Just as a note, for StringView, we used unsupported. Maybe we could keep things consistent? Or file an issue for refactor these into a common pattern?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #43888:
URL: https://github.com/apache/arrow/pull/43888#issuecomment-2358584933
@ViggoC I added a few comments, I am still going through the content in the implementation. I will finish that review by the end of the day tomorrow. Sorry to keep you waiting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1764315027
##########
java/vector/src/main/java/org/apache/arrow/vector/compare/RangeEqualsVisitor.java:
##########
@@ -255,6 +266,46 @@ public Boolean visit(LargeListViewVector left, Range range) {
return compareLargeListViewVectors(range);
}
+ protected boolean compareRunEndEncodedVectors(Range range) {
+ RunEndEncodedVector leftVector = (RunEndEncodedVector) left;
+ RunEndEncodedVector rightVector = (RunEndEncodedVector) right;
+
+ final int leftRangeEnd = range.getLeftStart() + range.getLength();
+ final int rightRangeEnd = range.getRightStart() + range.getLength();
+
+ FieldVector leftValuesVector = leftVector.getValuesVector();
+ FieldVector rightValuesVector = rightVector.getValuesVector();
+
+ RangeEqualsVisitor innerVisitor = createInnerVisitor(leftValuesVector, rightValuesVector, null);
+
+ int leftLogicalIndex = range.getLeftStart();
+ int rightLogicalIndex = range.getRightStart();
+
+ while (leftLogicalIndex < leftRangeEnd) {
+ int leftPhysicalIndex = leftVector.getPhysicalIndex(leftLogicalIndex);
Review Comment:
We can file a new issue for it and leave a TODO here.
##########
java/vector/src/main/java/org/apache/arrow/vector/validate/ValidateVectorBufferVisitor.java:
##########
@@ -287,4 +289,37 @@ public Void visit(ExtensionTypeVector<?> vector, Void value) {
vector.getUnderlyingVector().accept(this, value);
return null;
}
+
+ @Override
+ public Void visit(RunEndEncodedVector vector, Void value) {
+ validateVectorCommon(vector);
+ int valueCount = vector.getValueCount();
+ FieldVector runEndsVector = vector.getRunEndsVector();
+
+ if (runEndsVector != null) {
Review Comment:
Ah, good point. We can leave it then.
##########
java/vector/src/test/java/org/apache/arrow/vector/TestRunEndEncodedVector.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.List;
+import java.util.function.Function;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.vector.compare.Range;
+import org.apache.arrow.vector.compare.RangeEqualsVisitor;
+import org.apache.arrow.vector.complex.RunEndEncodedVector;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.ArrowType.RunEndEncoded;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+public class TestRunEndEncodedVector {
+
+ private BufferAllocator allocator;
+
+ @BeforeEach
+ public void init() {
+ allocator = new DirtyRootAllocator(Long.MAX_VALUE, (byte) 100);
Review Comment:
Hmm, I suppose it doesn't hurt to keep it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1767973977
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -185,6 +185,12 @@ public List<ArrowBuf> visit(ArrowType.Union type) {
}
}
+
Review Comment:
```suggestion
```
Make spotless happy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #43888:
URL: https://github.com/apache/arrow/pull/43888#issuecomment-2357560573
> Thanks, just one small thing.
>
> @vibhatha do you want to take a look as well?
Yes @lidavidm I am going to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "ViggoC (via GitHub)" <gi...@apache.org>.
ViggoC commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1765129002
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,683 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param runEndsVector The vector represents run ends. Only Zero vector or type int vector with
+ * size 16, 32 is allowed
+ * @param valuesVector The vector represents values
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ return null;
+ }
+
+ /**
+ * Set the initial record capacity.
+ *
+ * @param numRecords the initial record capacity.
+ */
+ @Override
+ public void setInitialCapacity(int numRecords) {}
+
+ /**
+ * Returns the maximum number of values that can be stored in this vector instance.
+ *
+ * @return the maximum number of values that can be stored in this vector instance.
+ */
+ @Override
+ public int getValueCapacity() {
+ return getChildrenFromFields().stream()
+ .mapToInt(item -> item != null ? item.getValueCapacity() : 0)
+ .min()
+ .orElseThrow(NoSuchElementException::new);
+ }
+
+ /** Alternative to clear(). Allows use as an AutoCloseable in try-with-resources. */
+ @Override
+ public void close() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.close();
+ }
+ }
+
+ /**
+ * Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has
+ * any child vectors, they will also be cleared.
+ */
+ @Override
+ public void clear() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.clear();
+ }
+ }
+
+ /**
+ * Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer
+ * capacities will remain unchanged and any previous data will be zeroed out. This includes
+ * buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be
+ * reset.
+ */
+ @Override
+ public void reset() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reset();
+ }
+ valueCount = 0;
+ }
+
+ /**
+ * Get information about how this field is materialized.
+ *
+ * @return the field corresponding to this vector
+ */
+ @Override
+ public Field getField() {
+ return field;
+ }
+
+ @Override
+ public MinorType getMinorType() {
+ return MinorType.RUNENDENCODED;
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(BufferAllocator allocator) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(BufferAllocator)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator) {
+ return getTransferPair(ref, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator) {
+ return getTransferPair(field, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(String, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(Field, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * Makes a new transfer pair used to transfer underlying buffers.
+ *
+ * @param target the target for the transfer
+ * @return a new {@link org.apache.arrow.vector.util.TransferPair transfer pair} that is used to
+ * transfer underlying buffers into the target vector.
+ */
+ @Override
+ public TransferPair makeTransferPair(ValueVector target) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support makeTransferPair(ValueVector)");
+ }
+
+ /**
+ * Get a reader for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.reader.FieldReader field reader} that supports
+ * reading values from this vector.
+ */
+ @Override
+ public FieldReader getReader() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get a writer for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.writer.FieldWriter field writer} that supports
+ * writing values to this vector.
+ */
+ public FieldWriter getWriter() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get the number of bytes used by this vector.
+ *
+ * @return the number of bytes that is used by this vector instance.
+ */
+ @Override
+ public int getBufferSize() {
+ int bufferSize = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ bufferSize += v.getBufferSize();
+ }
+ return bufferSize;
+ }
+
+ /**
+ * Returns the number of bytes that is used by this vector if it holds the given number of values.
+ * The result will be the same as if setValueCount() were called, followed by calling
+ * getBufferSize(), but without any of the closing side-effects that setValueCount() implies wrt
+ * finishing off the population of a vector. Some operations might wish to use this to determine
+ * how much memory has been used by a vector so far, even though it is not finished being
+ * populated.
+ *
+ * @param valueCount the number of values to assume this vector contains
+ * @return the buffer size if this vector is holding valueCount values
+ */
+ @Override
+ public int getBufferSizeFor(int valueCount) {
+ return 0;
+ }
+
+ /**
+ * Return the underlying buffers associated with this vector. Note that this doesn't impact the
+ * reference counts for this buffer so it only should be used for in-context access. Also note
+ * that this buffer changes regularly thus external classes shouldn't hold a reference to it
+ * (unless they change it).
+ *
+ * @param clear Whether to clear vector before returning; the buffers will still be refcounted;
+ * but the returned array will be the only reference to them
+ * @return The underlying {@link ArrowBuf buffers} that is used by this vector instance.
+ */
+ @Override
+ public ArrowBuf[] getBuffers(boolean clear) {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with validity vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getValidityBuffer() {
+ return null;
Review Comment:
@vibhatha what do you mean "for StringView, we used unsupported", I can see getValidityBuffer is implemented in BaseVariableWidthViewVector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #43888:
URL: https://github.com/apache/arrow/pull/43888#issuecomment-2363726160
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 7d96d9434028fb4d5fcf677d93dd584284cc713c.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/30433775377) has more details. It also includes information about 19 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #43888:
URL: https://github.com/apache/arrow/pull/43888#issuecomment-2362903280
@ViggoC thanks for getting this PR to the finish line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #43888:
URL: https://github.com/apache/arrow/pull/43888#issuecomment-2359976247
@lidavidm let's re-approve the workflows, PR LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39982: [Java] Add RunEndEncodedVector [arrow]
Posted by "ViggoC (via GitHub)" <gi...@apache.org>.
ViggoC commented on code in PR #43888:
URL: https://github.com/apache/arrow/pull/43888#discussion_r1767840499
##########
java/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java:
##########
@@ -0,0 +1,683 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.complex;
+
+import static org.apache.arrow.util.Preconditions.checkArgument;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.ByteFunctionHelpers;
+import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.vector.BaseIntVector;
+import org.apache.arrow.vector.BaseValueVector;
+import org.apache.arrow.vector.BufferBacked;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.ValueVector;
+import org.apache.arrow.vector.ZeroVector;
+import org.apache.arrow.vector.compare.VectorVisitor;
+import org.apache.arrow.vector.complex.reader.FieldReader;
+import org.apache.arrow.vector.complex.writer.FieldWriter;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.types.Types.MinorType;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.util.CallBack;
+import org.apache.arrow.vector.util.TransferPair;
+
+/**
+ * A run-end encoded vector contains only two child vectors: a run_end vector of type int and a
+ * values vector of any type. There are no buffers associated with the parent vector.
+ */
+public class RunEndEncodedVector extends BaseValueVector implements FieldVector {
+ public static final FieldVector DEFAULT_VALUE_VECTOR = ZeroVector.INSTANCE;
+ public static final FieldVector DEFAULT_RUN_END_VECTOR = ZeroVector.INSTANCE;
+
+ public static RunEndEncodedVector empty(String name, BufferAllocator allocator) {
+ return new RunEndEncodedVector(
+ name, allocator, FieldType.notNullable(ArrowType.RunEndEncoded.INSTANCE), null);
+ }
+
+ protected final CallBack callBack;
+ protected Field field;
+ protected FieldVector runEndsVector;
+ protected FieldVector valuesVector;
+ protected int valueCount;
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param name The name of the instance.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param fieldType The type of the array that is run-end encoded.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) {
+ this(new Field(name, fieldType, null), allocator, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(Field field, BufferAllocator allocator, CallBack callBack) {
+ this(field, allocator, DEFAULT_RUN_END_VECTOR, DEFAULT_VALUE_VECTOR, callBack);
+ }
+
+ /**
+ * Constructs a new instance.
+ *
+ * @param field The field materialized by this vector.
+ * @param allocator The allocator to use for allocating/reallocating buffers.
+ * @param runEndsVector The vector represents run ends. Only Zero vector or type int vector with
+ * size 16, 32 is allowed
+ * @param valuesVector The vector represents values
+ * @param callBack A schema change callback.
+ */
+ public RunEndEncodedVector(
+ Field field,
+ BufferAllocator allocator,
+ FieldVector runEndsVector,
+ FieldVector valuesVector,
+ CallBack callBack) {
+ super(allocator);
+ this.field = field;
+ this.callBack = callBack;
+ this.valueCount = 0;
+ this.runEndsVector = runEndsVector;
+ this.valuesVector = valuesVector;
+ }
+
+ /** ValueVector interface */
+
+ /**
+ * Allocate new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @throws OutOfMemoryException Thrown if no memory can be allocated.
+ */
+ @Override
+ public void allocateNew() throws OutOfMemoryException {
+ if (!allocateNewSafe()) {
+ throw new OutOfMemoryException("Failure while allocating memory");
+ }
+ }
+
+ /**
+ * Allocates new buffers. ValueVector implements logic to determine how much to allocate.
+ *
+ * @return Returns true if allocation was successful.
+ */
+ @Override
+ public boolean allocateNewSafe() {
+ initializeChildrenFromFields(field.getChildren());
+ for (FieldVector v : getChildrenFromFields()) {
+ boolean isAllocated = v.allocateNewSafe();
+ if (!isAllocated) {
+ v.clear();
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's
+ * buffer with new buffer, and release old one
+ */
+ @Override
+ public void reAlloc() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reAlloc();
+ }
+ }
+
+ @Override
+ public BufferAllocator getAllocator() {
+ return allocator;
+ }
+
+ @Override
+ protected FieldReader getReaderImpl() {
+ return null;
+ }
+
+ /**
+ * Set the initial record capacity.
+ *
+ * @param numRecords the initial record capacity.
+ */
+ @Override
+ public void setInitialCapacity(int numRecords) {}
+
+ /**
+ * Returns the maximum number of values that can be stored in this vector instance.
+ *
+ * @return the maximum number of values that can be stored in this vector instance.
+ */
+ @Override
+ public int getValueCapacity() {
+ return getChildrenFromFields().stream()
+ .mapToInt(item -> item != null ? item.getValueCapacity() : 0)
+ .min()
+ .orElseThrow(NoSuchElementException::new);
+ }
+
+ /** Alternative to clear(). Allows use as an AutoCloseable in try-with-resources. */
+ @Override
+ public void close() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.close();
+ }
+ }
+
+ /**
+ * Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has
+ * any child vectors, they will also be cleared.
+ */
+ @Override
+ public void clear() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.clear();
+ }
+ }
+
+ /**
+ * Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer
+ * capacities will remain unchanged and any previous data will be zeroed out. This includes
+ * buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be
+ * reset.
+ */
+ @Override
+ public void reset() {
+ for (FieldVector v : getChildrenFromFields()) {
+ v.reset();
+ }
+ valueCount = 0;
+ }
+
+ /**
+ * Get information about how this field is materialized.
+ *
+ * @return the field corresponding to this vector
+ */
+ @Override
+ public Field getField() {
+ return field;
+ }
+
+ @Override
+ public MinorType getMinorType() {
+ return MinorType.RUNENDENCODED;
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(BufferAllocator allocator) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(BufferAllocator)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator) {
+ return getTransferPair(ref, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator) {
+ return getTransferPair(field, allocator, null);
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param ref the name of the vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(String, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * To transfer quota responsibility.
+ *
+ * @param field the Field object used by the target vector
+ * @param allocator the target allocator
+ * @param callBack A schema change callback.
+ * @return a {@link org.apache.arrow.vector.util.TransferPair transfer pair}, creating a new
+ * target vector of the same type.
+ */
+ @Override
+ public TransferPair getTransferPair(Field field, BufferAllocator allocator, CallBack callBack) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support getTransferPair(Field, BufferAllocator, CallBack)");
+ }
+
+ /**
+ * Makes a new transfer pair used to transfer underlying buffers.
+ *
+ * @param target the target for the transfer
+ * @return a new {@link org.apache.arrow.vector.util.TransferPair transfer pair} that is used to
+ * transfer underlying buffers into the target vector.
+ */
+ @Override
+ public TransferPair makeTransferPair(ValueVector target) {
+ throw new UnsupportedOperationException(
+ "RunEndEncodedVector does not support makeTransferPair(ValueVector)");
+ }
+
+ /**
+ * Get a reader for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.reader.FieldReader field reader} that supports
+ * reading values from this vector.
+ */
+ @Override
+ public FieldReader getReader() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get a writer for this vector.
+ *
+ * @return a {@link org.apache.arrow.vector.complex.writer.FieldWriter field writer} that supports
+ * writing values to this vector.
+ */
+ public FieldWriter getWriter() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ /**
+ * Get the number of bytes used by this vector.
+ *
+ * @return the number of bytes that is used by this vector instance.
+ */
+ @Override
+ public int getBufferSize() {
+ int bufferSize = 0;
+ for (FieldVector v : getChildrenFromFields()) {
+ bufferSize += v.getBufferSize();
+ }
+ return bufferSize;
+ }
+
+ /**
+ * Returns the number of bytes that is used by this vector if it holds the given number of values.
+ * The result will be the same as if setValueCount() were called, followed by calling
+ * getBufferSize(), but without any of the closing side-effects that setValueCount() implies wrt
+ * finishing off the population of a vector. Some operations might wish to use this to determine
+ * how much memory has been used by a vector so far, even though it is not finished being
+ * populated.
+ *
+ * @param valueCount the number of values to assume this vector contains
+ * @return the buffer size if this vector is holding valueCount values
+ */
+ @Override
+ public int getBufferSizeFor(int valueCount) {
+ return 0;
+ }
+
+ /**
+ * Return the underlying buffers associated with this vector. Note that this doesn't impact the
+ * reference counts for this buffer so it only should be used for in-context access. Also note
+ * that this buffer changes regularly thus external classes shouldn't hold a reference to it
+ * (unless they change it).
+ *
+ * @param clear Whether to clear vector before returning; the buffers will still be refcounted;
+ * but the returned array will be the only reference to them
+ * @return The underlying {@link ArrowBuf buffers} that is used by this vector instance.
+ */
+ @Override
+ public ArrowBuf[] getBuffers(boolean clear) {
+ return null;
+ }
+
+ /**
+ * Gets the underlying buffer associated with validity vector.
+ *
+ * @return buffer
+ */
+ @Override
+ public ArrowBuf getValidityBuffer() {
+ return null;
Review Comment:
fixed in [14ba5bb](https://github.com/apache/arrow/pull/43888/commits/14ba5bbff29398ab8772ac3ae2817ee744e270b4)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org