You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "CurtHagenlocher (via GitHub)" <gi...@apache.org> on 2026/04/07 12:34:36 UTC

[PR] feat: Add support for Run-End Encoded arrays [arrow-dotnet]

CurtHagenlocher opened a new pull request, #308:
URL: https://github.com/apache/arrow-dotnet/pull/308

   ## What's Changed
   
   This PR adds basic support for Run-End Encoded arrays by following established codebase patterns.
   
   Notably:
   - New `ArrowTypeId` added.
   - New array type `RunEndEncodedArray` added.
   - New visitor method to handle the new array type.
   - New entry in the IPC serializer field type switch.
   - New `RunEndEncodedType` nested type.
   - Basic feature tests.
   - C API support
   - Concatenation support
   
   Co-authored-by: Jorge Candeias <jo...@outcompute.com>
   
   Supercedes #260
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: Add support for Run-End Encoded arrays [arrow-dotnet]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher merged PR #308:
URL: https://github.com/apache/arrow-dotnet/pull/308


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: Add support for Run-End Encoded arrays [arrow-dotnet]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on code in PR #308:
URL: https://github.com/apache/arrow-dotnet/pull/308#discussion_r3045603754


##########
src/Apache.Arrow/Arrays/ArrayDataConcatenator.cs:
##########
@@ -286,6 +287,201 @@ public void Visit(UnionType type)
 
             public void Visit(MapType type) => ConcatenateLists(type.UnsortedKey()); /* Can't tell if the output is still sorted */
 
+            public void Visit(RunEndEncodedType type)
+            {
+                ArrowTypeId runEndsTypeId = type.RunEndsDataType.TypeId;
+                if (runEndsTypeId != ArrowTypeId.Int16 &&
+                    runEndsTypeId != ArrowTypeId.Int32 &&
+                    runEndsTypeId != ArrowTypeId.Int64)
+                {
+                    throw new InvalidOperationException(
+                        $"Run-ends array must be Int16, Int32, or Int64, but got {runEndsTypeId}");
+                }
+
+                var slicedValues = new List<ArrayData>(_arrayDataList.Count);
+                ArrowBuffer.Builder<short> int16Builder = null;
+                ArrowBuffer.Builder<int> int32Builder = null;
+                ArrowBuffer.Builder<long> int64Builder = null;
+
+                switch (runEndsTypeId)
+                {
+                    case ArrowTypeId.Int16: int16Builder = new ArrowBuffer.Builder<short>(); break;
+                    case ArrowTypeId.Int32: int32Builder = new ArrowBuffer.Builder<int>(); break;
+                    case ArrowTypeId.Int64: int64Builder = new ArrowBuffer.Builder<long>(); break;
+                }
+
+                long baseOffset = 0;
+                int physicalRunCount = 0;
+
+                foreach (ArrayData arrayData in _arrayDataList)
+                {
+                    arrayData.EnsureDataType(type.TypeId);
+
+                    ArrayData runEndsData = arrayData.Children[0];
+                    ArrayData valuesData = arrayData.Children[1];
+
+                    if (runEndsData.DataType.TypeId != runEndsTypeId)
+                    {
+                        throw new ArgumentException(
+                            $"All run-end encoded arrays must have the same run-ends type. Expected <{runEndsTypeId}> but got <{runEndsData.DataType.TypeId}>.");
+                    }
+                    if (valuesData.DataType.TypeId != type.ValuesDataType.TypeId)
+                    {
+                        throw new ArgumentException(
+                            $"All run-end encoded arrays must have the same values type. Expected <{type.ValuesDataType.TypeId}> but got <{valuesData.DataType.TypeId}>.");
+                    }
+
+                    if (arrayData.Length == 0)
+                    {
+                        continue;
+                    }
+
+                    int logicalOffset = arrayData.Offset;
+                    int logicalLength = arrayData.Length;
+                    int logicalEnd = logicalOffset + logicalLength;
+
+                    int physicalStart;
+                    int physicalEndExclusive;
+
+                    switch (runEndsTypeId)
+                    {
+                        case ArrowTypeId.Int16:
+                            {
+                                ReadOnlySpan<short> re = runEndsData.Buffers[1].Span.CastTo<short>()
+                                    .Slice(runEndsData.Offset, runEndsData.Length);
+                                physicalStart = FindPhysicalStartInt16(re, logicalOffset);
+                                physicalEndExclusive = FindPhysicalEndExclusiveInt16(re, logicalEnd, physicalStart);
+                                for (int p = physicalStart; p < physicalEndExclusive; p++)
+                                {
+                                    int adjustedEnd = Math.Min((int)re[p], logicalEnd) - logicalOffset;
+                                    int16Builder.Append(checked((short)(baseOffset + adjustedEnd)));
+                                }

Review Comment:
   I'm okay with this for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: Add support for Run-End Encoded arrays [arrow-dotnet]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on code in PR #308:
URL: https://github.com/apache/arrow-dotnet/pull/308#discussion_r3048367571


##########
src/Apache.Arrow/Arrays/RunEndEncodedArray.cs:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using Apache.Arrow.Memory;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow;
+
+/// <summary>
+/// Represents a run-end encoded array.
+/// A run-end encoded array stores consecutive runs of the same value more efficiently.
+/// It contains two child arrays: run_ends (Int16/Int32/Int64) and values (any type).
+/// The run_ends array stores the cumulative end positions of each run.
+/// </summary>
+public class RunEndEncodedArray : Array
+{
+    /// <summary>
+    /// Gets the run ends array (Int16Array, Int32Array, or Int64Array).
+    /// This array contains the cumulative end indices for each run.
+    /// </summary>
+    public IArrowArray RunEnds { get; }
+
+    /// <summary>
+    /// Gets the values array.
+    /// This array contains the actual values that are run-length encoded.
+    /// </summary>
+    public IArrowArray Values { get; }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray from ArrayData.
+    /// </summary>
+    /// <param name="data">The array data containing run ends and values as children.</param>
+    public RunEndEncodedArray(ArrayData data)
+        : this(data, ArrowArrayFactory.BuildArray(data.Children[0]), ArrowArrayFactory.BuildArray(data.Children[1]))
+    {
+    }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray with specified run ends and values arrays.
+    /// </summary>
+    /// <param name="runEnds">The run ends array (must be Int16Array, Int32Array, or Int64Array).</param>
+    /// <param name="values">The values array (can be any type).</param>
+    public RunEndEncodedArray(IArrowArray runEnds, IArrowArray values)
+        : this(CreateArrayData(runEnds, values), runEnds, values)
+    {
+    }
+
+    private RunEndEncodedArray(ArrayData data, IArrowArray runEnds, IArrowArray values)
+        : base(data)
+    {
+        data.EnsureBufferCount(0); // REE arrays have no buffers, only children
+        data.EnsureDataType(ArrowTypeId.RunEndEncoded);
+
+        if (data.NullCount != 0)
+        {
+            throw new ArgumentException(
+                $"Run-end encoded arrays have no top-level validity bitmap and must report null count 0, but got {data.NullCount}.",
+                nameof(data));
+        }
+
+        ValidateRunEndsType(runEnds);
+
+        if (runEnds.Length != values.Length)
+        {
+            throw new ArgumentException(
+                $"Run ends array length ({runEnds.Length}) must equal values array length ({values.Length}).");
+        }

Review Comment:
   I think I won't do this for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: Add support for Run-End Encoded arrays [arrow-dotnet]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #308:
URL: https://github.com/apache/arrow-dotnet/pull/308#discussion_r3045654722


##########
test/Apache.Arrow.Tests/RunEndEncodedArrayTests.cs:
##########
@@ -0,0 +1,366 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.IO;
+using Apache.Arrow.Ipc;
+using Apache.Arrow.Types;
+using Xunit;
+
+namespace Apache.Arrow.Tests;
+
+public class RunEndEncodedArrayTests
+{
+    [Fact]
+    public void TestRunEndEncodedTypeCreation()
+    {
+        // Test with explicit fields
+        var runEndsField = new Field("run_ends", Int32Type.Default, nullable: false);
+        var valuesField = new Field("values", StringType.Default, nullable: true);
+        var reeType = new RunEndEncodedType(runEndsField, valuesField);
+
+        Assert.Equal(ArrowTypeId.RunEndEncoded, reeType.TypeId);
+        Assert.Equal("run_end_encoded", reeType.Name);
+        Assert.Equal(runEndsField, reeType.RunEndsField);
+        Assert.Equal(valuesField, reeType.ValuesField);
+        Assert.Equal(Int32Type.Default.TypeId, reeType.RunEndsDataType.TypeId);
+        Assert.Equal(StringType.Default.TypeId, reeType.ValuesDataType.TypeId);
+    }
+
+    [Fact]
+    public void TestRunEndEncodedTypeCreationWithDataTypes()
+    {
+        // Test with data types (uses default field names)
+        var reeType = new RunEndEncodedType(Int32Type.Default, StringType.Default);
+
+        Assert.Equal(ArrowTypeId.RunEndEncoded, reeType.TypeId);
+        Assert.Equal("run_ends", reeType.RunEndsField.Name);
+        Assert.Equal("values", reeType.ValuesField.Name);
+    }
+
+    [Fact]
+    public void TestRunEndEncodedTypeValidation()
+    {
+        // Invalid run ends type (must be Int16, Int32, or Int64)
+        Assert.Throws<ArgumentException>(() => new RunEndEncodedType(Int8Type.Default, StringType.Default));
+        Assert.Throws<ArgumentException>(() => new RunEndEncodedType(FloatType.Default, StringType.Default));
+        Assert.Throws<ArgumentException>(() => new RunEndEncodedType(StringType.Default, StringType.Default));
+
+        // Valid run ends types
+        Assert.NotNull(new RunEndEncodedType(Int16Type.Default, StringType.Default)); // Should not throw
+        Assert.NotNull(new RunEndEncodedType(Int32Type.Default, StringType.Default)); // Should not throw
+        Assert.NotNull(new RunEndEncodedType(Int64Type.Default, StringType.Default)); // Should not throw
+    }
+
+    [Fact]
+    public void TestRunEndEncodedArrayWithInt32RunEnds()
+    {
+        // Create run ends: [3, 7, 10, 15]
+        // This represents: 3 'A's, 4 'B's, 3 'C's, 5 'D's
+        var runEndsBuilder = new Int32Array.Builder();
+        runEndsBuilder.AppendRange([3, 7, 10, 15]);
+        Int32Array runEnds = runEndsBuilder.Build();
+
+        // Create values: ['A', 'B', 'C', 'D']
+        var valuesBuilder = new StringArray.Builder();
+        valuesBuilder.AppendRange(["A", "B", "C", "D"]);
+        StringArray values = valuesBuilder.Build();
+
+        // Create REE array
+        var reeArray = new RunEndEncodedArray(runEnds, values);
+
+        Assert.Equal(15, reeArray.Length); // Logical length is the last run end value
+        Assert.Equal(0, reeArray.NullCount); // REE arrays don't have nulls at the top level
+        Assert.Equal(runEnds, reeArray.RunEnds);
+        Assert.Equal(values, reeArray.Values);
+    }
+
+    [Fact]
+    public void TestRunEndEncodedArrayWithInt16RunEnds()
+    {
+        var runEndsBuilder = new Int16Array.Builder();
+        runEndsBuilder.AppendRange([2, 5, 8]);
+        Int16Array runEnds = runEndsBuilder.Build();
+
+        var valuesBuilder = new Int32Array.Builder();
+        valuesBuilder.AppendRange([100, 200, 300]);
+        Int32Array values = valuesBuilder.Build();
+
+        var reeArray = new RunEndEncodedArray(runEnds, values);
+
+        Assert.Equal(8, reeArray.Length);
+        Assert.Equal(runEnds, reeArray.RunEnds);
+        Assert.Equal(values, reeArray.Values);
+    }
+
+    [Fact]
+    public void TestRunEndEncodedArrayWithInt64RunEnds()
+    {
+        var runEndsBuilder = new Int64Array.Builder();
+        runEndsBuilder.AppendRange([1000, 2000, 3000]);
+        Int64Array runEnds = runEndsBuilder.Build();
+
+        var valuesBuilder = new DoubleArray.Builder();
+        valuesBuilder.AppendRange([1.5, 2.5, 3.5]);
+        DoubleArray values = valuesBuilder.Build();
+
+        var reeArray = new RunEndEncodedArray(runEnds, values);
+
+        Assert.Equal(3000, reeArray.Length);
+        Assert.Equal(runEnds, reeArray.RunEnds);
+        Assert.Equal(values, reeArray.Values);
+    }
+
+    [Fact]
+    public void TestRunEndEncodedArrayInvalidRunEndsType()
+    {
+        Int8Array invalidRunEnds = new Int8Array.Builder().AppendRange([1, 2, 3]).Build();
+        StringArray values = new StringArray.Builder().AppendRange(["A", "B", "C"]).Build();
+
+        Assert.Throws<ArgumentException>(() => new RunEndEncodedArray(invalidRunEnds, values));
+    }
+
+    [Fact]
+    public void TestRunEndEncodedArrayEmpty()
+    {
+        Int32Array runEnds = new Int32Array.Builder().Build();
+        StringArray values = new StringArray.Builder().Build();
+
+        var reeArray = new RunEndEncodedArray(runEnds, values);
+
+        Assert.Equal(0, reeArray.Length);
+    }
+
+    [Fact]
+    public void TestFindPhysicalIndexInt32()
+    {
+        // Run ends: [3, 7, 10, 15] means:
+        // Logical indices 0-2 map to physical index 0 (value 'A')
+        // Logical indices 3-6 map to physical index 1 (value 'B')
+        // Logical indices 7-9 map to physical index 2 (value 'C')
+        // Logical indices 10-14 map to physical index 3 (value 'D')
+        Int32Array runEnds = new Int32Array.Builder()
+            .AppendRange([3, 7, 10, 15])
+            .Build();
+        StringArray values = new StringArray.Builder()
+            .AppendRange(["A", "B", "C", "D"])
+            .Build();
+
+        var reeArray = new RunEndEncodedArray(runEnds, values);
+
+        Assert.Equal(0, reeArray.FindPhysicalIndex(0));
+        Assert.Equal(0, reeArray.FindPhysicalIndex(1));
+        Assert.Equal(0, reeArray.FindPhysicalIndex(2));
+        Assert.Equal(1, reeArray.FindPhysicalIndex(3));
+        Assert.Equal(1, reeArray.FindPhysicalIndex(4));
+        Assert.Equal(1, reeArray.FindPhysicalIndex(5));
+        Assert.Equal(1, reeArray.FindPhysicalIndex(6));
+        Assert.Equal(2, reeArray.FindPhysicalIndex(7));
+        Assert.Equal(2, reeArray.FindPhysicalIndex(8));
+        Assert.Equal(2, reeArray.FindPhysicalIndex(9));
+        Assert.Equal(3, reeArray.FindPhysicalIndex(10));
+        Assert.Equal(3, reeArray.FindPhysicalIndex(11));
+        Assert.Equal(3, reeArray.FindPhysicalIndex(14));
+    }
+
+    [Fact]
+    public void TestRunEndEncodedLengthMismatchThrows()
+    {
+        Int32Array runEnds = new Int32Array.Builder().AppendRange([3, 7]).Build();
+        StringArray values = new StringArray.Builder().AppendRange(["A", "B", "C"]).Build();
+
+        var ex = Assert.Throws<ArgumentException>(() => new RunEndEncodedArray(runEnds, values));
+        Assert.Contains("length", ex.Message);
+    }
+
+    [Fact]
+    public void TestRunEndEncodedNullRunEndsThrows()
+    {
+        Int32Array runEnds = new Int32Array.Builder().Append(3).AppendNull().Build();
+        StringArray values = new StringArray.Builder().AppendRange(["A", "B"]).Build();
+
+        var ex = Assert.Throws<ArgumentException>(() => new RunEndEncodedArray(runEnds, values));
+        Assert.Contains("null", ex.Message);
+    }
+
+    [Fact]
+    public void TestFindPhysicalIndexOnSlicedArray()
+    {
+        // Run ends: [3, 7, 10] → run 0 (A): positions 0..2, run 1 (B): 3..6, run 2 (C): 7..9
+        Int32Array runEnds = new Int32Array.Builder().AppendRange([3, 7, 10]).Build();
+        StringArray values = new StringArray.Builder().AppendRange(["A", "B", "C"]).Build();
+        var reeArray = new RunEndEncodedArray(runEnds, values);
+
+        // Slice covering underlying positions 2..7 (length 6).
+        // Slice positions: 0→A(2), 1→B(3), 2→B(4), 3→B(5), 4→B(6), 5→C(7)
+        var sliced = (RunEndEncodedArray)ArrowArrayFactory.Slice(reeArray, 2, 6);
+
+        Assert.Equal(6, sliced.Length);
+        Assert.Equal(0, sliced.FindPhysicalIndex(0));
+        Assert.Equal(1, sliced.FindPhysicalIndex(1));
+        Assert.Equal(1, sliced.FindPhysicalIndex(2));
+        Assert.Equal(1, sliced.FindPhysicalIndex(3));
+        Assert.Equal(1, sliced.FindPhysicalIndex(4));
+        Assert.Equal(2, sliced.FindPhysicalIndex(5));
+
+        // Out-of-range against the slice length, even though they would be valid against
+        // the underlying physical array.
+        Assert.Throws<ArgumentOutOfRangeException>(() => sliced.FindPhysicalIndex(6));
+    }
+
+    [Fact]
+    public void TestFindPhysicalIndexOutOfRange()
+    {
+        Int32Array runEnds = new Int32Array.Builder().AppendRange([3, 7]).Build();
+        StringArray values = new StringArray.Builder().AppendRange(["A", "B"]).Build();
+        var reeArray = new RunEndEncodedArray(runEnds, values);
+
+        Assert.Throws<ArgumentOutOfRangeException>(() => reeArray.FindPhysicalIndex(-1));
+        Assert.Throws<ArgumentOutOfRangeException>(() => reeArray.FindPhysicalIndex(7));
+        Assert.Throws<ArgumentOutOfRangeException>(() => reeArray.FindPhysicalIndex(100));
+    }
+
+    [Fact]
+    public void TestRunEndEncodedArraySerialization()
+    {
+        // Create a REE array
+        Int32Array runEnds = new Int32Array.Builder().AppendRange([3, 7, 10]).Build();
+        StringArray values = new StringArray.Builder().AppendRange(["foo", "bar", "baz"]).Build();
+        var reeArray = new RunEndEncodedArray(runEnds, values);
+
+        // Create a record batch with the REE array
+        var reeField = new Field("ree_column", reeArray.Data.DataType, nullable: false);
+        var schema = new Schema([reeField], null);
+        var recordBatch = new RecordBatch(schema, [reeArray], reeArray.Length);
+

Review Comment:
   The serialization test only covers a non-sliced RunEndEncodedArray (Offset=0). Given REE requires special handling to serialize slices correctly (since IPC always encodes arrays with offset=0), it would be good to add a round-trip test for a sliced REE array (e.g., ArrowArrayFactory.Slice on a larger REE) to prevent regressions in slice normalization during write/read.



##########
src/Apache.Arrow/Ipc/ArrowReaderImplementation.cs:
##########
@@ -276,6 +276,9 @@ private ArrayData LoadField(
             {
                 case ArrowTypeId.Null:
                     return new ArrayData(field.DataType, fieldLength, fieldNullCount, 0, System.Array.Empty<ArrowBuffer>());
+                case ArrowTypeId.RunEndEncoded:

Review Comment:
   When reading RunEndEncoded arrays, the IPC field node’s nullCount should always be 0 (REE has no top-level validity bitmap). If a message provides a non-zero nullCount, the resulting ArrayData will have NullCount != 0 but BufferCount==0, which can later trigger IndexOutOfRangeException via Array.IsValid/NullBitmapBuffer access. Consider validating fieldNullCount==0 here and throwing InvalidDataException for non-canonical/invalid input.
   ```suggestion
                   case ArrowTypeId.RunEndEncoded:
                       if (fieldNullCount != 0)
                       {
                           throw new InvalidDataException("RunEndEncoded array top-level null count must be 0");
                       }
   ```



##########
src/Apache.Arrow/Ipc/ArrowStreamWriter.cs:
##########
@@ -358,6 +359,15 @@ public void Visit(DictionaryArray array)
                 array.Indices.Accept(this);
             }
 
+            public void Visit(RunEndEncodedArray array)
+            {
+                // REE arrays have no buffers at the top level, only child arrays
+                // Visit the run_ends array
+                VisitArray(array.RunEnds);
+                // Visit the values array
+                VisitArray(array.Values);
+            }

Review Comment:
   Arrow IPC serialization doesn’t account for sliced RunEndEncodedArray instances. When array.Offset != 0 (or when serializing a logical slice), this currently writes the full RunEnds/Values children without normalizing run_ends to the slice range, but the stream format always encodes arrays with offset=0. This will round-trip to incorrect data (logical index 0 in the deserialized array won’t correspond to the slice’s start). Consider serializing a normalized/sliced REE representation: compute the physical run range for [array.Offset, array.Offset+array.Length) and emit adjusted run_ends (ending at array.Length) plus the corresponding values slice.



##########
src/Apache.Arrow/Arrays/RunEndEncodedArray.cs:
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow;
+
+/// <summary>
+/// Represents a run-end encoded array.
+/// A run-end encoded array stores consecutive runs of the same value more efficiently.
+/// It contains two child arrays: run_ends (Int16/Int32/Int64) and values (any type).
+/// The run_ends array stores the cumulative end positions of each run.
+/// </summary>
+public class RunEndEncodedArray : Array
+{
+    /// <summary>
+    /// Gets the run ends array (Int16Array, Int32Array, or Int64Array).
+    /// This array contains the cumulative end indices for each run.
+    /// </summary>
+    public IArrowArray RunEnds { get; }
+
+    /// <summary>
+    /// Gets the values array.
+    /// This array contains the actual values that are run-length encoded.
+    /// </summary>
+    public IArrowArray Values { get; }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray from ArrayData.
+    /// </summary>
+    /// <param name="data">The array data containing run ends and values as children.</param>
+    public RunEndEncodedArray(ArrayData data)
+        : this(data, ArrowArrayFactory.BuildArray(data.Children[0]), ArrowArrayFactory.BuildArray(data.Children[1]))
+    {
+    }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray with specified run ends and values arrays.
+    /// </summary>
+    /// <param name="runEnds">The run ends array (must be Int16Array, Int32Array, or Int64Array).</param>
+    /// <param name="values">The values array (can be any type).</param>
+    public RunEndEncodedArray(IArrowArray runEnds, IArrowArray values)
+        : this(CreateArrayData(runEnds, values), runEnds, values)
+    {
+    }
+
+    private RunEndEncodedArray(ArrayData data, IArrowArray runEnds, IArrowArray values)
+        : base(data)
+    {
+        data.EnsureBufferCount(0); // REE arrays have no buffers, only children
+        data.EnsureDataType(ArrowTypeId.RunEndEncoded);
+

Review Comment:
   RunEndEncodedArray relies on NullCount==0 to avoid Array.IsValid touching NullBitmapBuffer (since buffer count is 0). If a RunEndEncodedArray is constructed from ArrayData with non-zero nullCount (e.g., malformed IPC/C data), IsValid/IsNull can throw IndexOutOfRangeException when it tries to access Data.Buffers[0]. Consider validating data.NullCount==0 in this constructor (and throwing ArgumentException/InvalidDataException) to fail fast with a clearer message.
   ```suggestion
   
           if (data.NullCount != 0)
           {
               throw new ArgumentException(
                   $"Run-end encoded arrays must not have parent nulls, but had {data.NullCount} null(s).",
                   nameof(data));
           }
   ```



##########
src/Apache.Arrow/C/CArrowArrayImporter.cs:
##########
@@ -206,6 +206,11 @@ private ArrayData GetAsArrayData(CArrowArray* cArray, IArrowType type)
                         children = ProcessListChildren(cArray, mapType.Fields[0].DataType);
                         buffers = ImportListBuffers(cArray);
                         break;
+                    case ArrowTypeId.RunEndEncoded:
+                        RunEndEncodedType reeType = (RunEndEncodedType)storageType;
+                        children = ProcessStructChildren(cArray, reeType.Fields);

Review Comment:
   For REE import, buffers are set to empty (correct for canonical REE), but ArrayData created later will still carry cArray->null_count. If a C producer sets null_count > 0, the resulting RunEndEncodedArray can later throw when IsValid accesses Data.Buffers[0]. Consider validating cArray->null_count==0 for ArrowTypeId.RunEndEncoded (or otherwise rejecting/normalizing) to avoid constructing an invalid in-memory array shape.
   ```suggestion
                           children = ProcessStructChildren(cArray, reeType.Fields);
                           if (cArray->null_count != 0)
                           {
                               throw new InvalidOperationException("Imported RunEndEncoded arrays must have null_count == 0.");
                           }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: Add support for Run-End Encoded arrays [arrow-dotnet]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #308:
URL: https://github.com/apache/arrow-dotnet/pull/308#discussion_r3045035711


##########
src/Apache.Arrow/Arrays/RunEndEncodedArray.cs:
##########
@@ -0,0 +1,222 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow;
+
+/// <summary>
+/// Represents a run-end encoded array.
+/// A run-end encoded array stores consecutive runs of the same value more efficiently.
+/// It contains two child arrays: run_ends (Int16/Int32/Int64) and values (any type).
+/// The run_ends array stores the cumulative end positions of each run.
+/// </summary>
+public class RunEndEncodedArray : Array
+{
+    /// <summary>
+    /// Gets the run ends array (Int16Array, Int32Array, or Int64Array).
+    /// This array contains the cumulative end indices for each run.
+    /// </summary>
+    public IArrowArray RunEnds { get; }
+
+    /// <summary>
+    /// Gets the values array.
+    /// This array contains the actual values that are run-length encoded.
+    /// </summary>
+    public IArrowArray Values { get; }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray from ArrayData.
+    /// </summary>
+    /// <param name="data">The array data containing run ends and values as children.</param>
+    public RunEndEncodedArray(ArrayData data)
+        : this(data, ArrowArrayFactory.BuildArray(data.Children[0]), ArrowArrayFactory.BuildArray(data.Children[1]))
+    {
+    }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray with specified run ends and values arrays.
+    /// </summary>
+    /// <param name="runEnds">The run ends array (must be Int16Array, Int32Array, or Int64Array).</param>
+    /// <param name="values">The values array (can be any type).</param>
+    public RunEndEncodedArray(IArrowArray runEnds, IArrowArray values)
+        : this(CreateArrayData(runEnds, values), runEnds, values)
+    {
+    }
+
+    private RunEndEncodedArray(ArrayData data, IArrowArray runEnds, IArrowArray values)
+        : base(data)
+    {
+        data.EnsureBufferCount(0); // REE arrays have no buffers, only children
+        data.EnsureDataType(ArrowTypeId.RunEndEncoded);
+
+        ValidateRunEndsType(runEnds);
+        RunEnds = runEnds;
+        Values = values;
+    }
+
+    private static ArrayData CreateArrayData(IArrowArray runEnds, IArrowArray values)
+    {
+        ValidateRunEndsType(runEnds);
+
+        // The logical length of a REE array is determined by the last value in run_ends
+        int logicalLength = GetLogicalLength(runEnds);
+
+        var dataType = new RunEndEncodedType(runEnds.Data.DataType, values.Data.DataType);
+
+        return new ArrayData(
+            dataType,
+            logicalLength,
+            nullCount: 0, // REE arrays don't have a validity bitmap
+            offset: 0,
+            buffers: [],
+            children: [runEnds.Data, values.Data]);
+    }
+
+    private static void ValidateRunEndsType(IArrowArray runEnds)
+    {
+        ArrowTypeId typeId = runEnds.Data.DataType.TypeId;
+        if (typeId != ArrowTypeId.Int16 &&
+            typeId != ArrowTypeId.Int32 &&
+            typeId != ArrowTypeId.Int64)
+        {
+            throw new ArgumentException(
+                $"Run ends array must be Int16, Int32, or Int64, but got {typeId}",
+                nameof(runEnds));
+        }
+    }
+
+    private static int GetLogicalLength(IArrowArray runEnds)
+    {
+        if (runEnds.Length == 0)
+        {
+            return 0;
+        }
+
+        // Get the last run end value which represents the logical length
+        switch (runEnds)
+        {
+            case Int16Array int16Array:
+                return int16Array.GetValue(int16Array.Length - 1) ?? throw new ArgumentException("invalid length");
+            case Int32Array int32Array:
+                return int32Array.GetValue(int32Array.Length - 1) ?? throw new ArgumentException("invalid length");
+            case Int64Array int64Array:
+                {
+                    long? lastValue = int64Array.GetValue(int64Array.Length - 1);
+                    if (lastValue.HasValue && lastValue.Value > int.MaxValue)
+                    {
+                        throw new ArgumentException("Run ends value exceeds maximum supported length.");
+                    }
+                    return (int)(lastValue ?? throw new ArgumentException("invalid length"));
+                }
+            default:
+                throw new InvalidOperationException($"Unexpected run ends array type: {runEnds.Data.DataType.TypeId}");
+        }
+    }
+
+    /// <summary>
+    /// Finds the physical index in the run_ends array that contains the specified logical index.
+    /// </summary>
+    /// <param name="logicalIndex">The logical index in the decoded array.</param>
+    /// <returns>The physical index in the run_ends/values arrays.</returns>
+    public int FindPhysicalIndex(int logicalIndex)
+    {
+        if (logicalIndex < 0 || logicalIndex >= Length)
+        {
+            throw new ArgumentOutOfRangeException(nameof(logicalIndex));
+        }
+
+        // Binary search to find the run that contains this logical index
+        return RunEnds switch
+        {
+            Int16Array int16Array => BinarySearchRunEnds(int16Array, logicalIndex),
+            Int32Array int32Array => BinarySearchRunEnds(int32Array, logicalIndex),
+            Int64Array int64Array => BinarySearchRunEnds(int64Array, logicalIndex),

Review Comment:
   FindPhysicalIndex doesn't account for the REE array's Data.Offset (slices). For a sliced RunEndEncodedArray, logical indices are relative to the slice, but the binary search compares against run_ends values from the unsliced physical representation, producing incorrect physical indices. Adjust the searched logical index by Data.Offset (or otherwise make FindPhysicalIndex slice-aware) and add a test covering FindPhysicalIndex on a sliced REE array.
   ```suggestion
           int searchLogicalIndex = logicalIndex + Data.Offset;
   
           // Binary search to find the run that contains this logical index.
           // For sliced arrays, run_ends remain in the original logical coordinate
           // space, so we need to account for the array offset before searching.
           return RunEnds switch
           {
               Int16Array int16Array => BinarySearchRunEnds(int16Array, searchLogicalIndex),
               Int32Array int32Array => BinarySearchRunEnds(int32Array, searchLogicalIndex),
               Int64Array int64Array => BinarySearchRunEnds(int64Array, searchLogicalIndex),
   ```



##########
src/Apache.Arrow/Arrays/RunEndEncodedArray.cs:
##########
@@ -0,0 +1,222 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow;
+
+/// <summary>
+/// Represents a run-end encoded array.
+/// A run-end encoded array stores consecutive runs of the same value more efficiently.
+/// It contains two child arrays: run_ends (Int16/Int32/Int64) and values (any type).
+/// The run_ends array stores the cumulative end positions of each run.
+/// </summary>
+public class RunEndEncodedArray : Array
+{
+    /// <summary>
+    /// Gets the run ends array (Int16Array, Int32Array, or Int64Array).
+    /// This array contains the cumulative end indices for each run.
+    /// </summary>
+    public IArrowArray RunEnds { get; }
+
+    /// <summary>
+    /// Gets the values array.
+    /// This array contains the actual values that are run-length encoded.
+    /// </summary>
+    public IArrowArray Values { get; }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray from ArrayData.
+    /// </summary>
+    /// <param name="data">The array data containing run ends and values as children.</param>
+    public RunEndEncodedArray(ArrayData data)
+        : this(data, ArrowArrayFactory.BuildArray(data.Children[0]), ArrowArrayFactory.BuildArray(data.Children[1]))
+    {
+    }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray with specified run ends and values arrays.
+    /// </summary>
+    /// <param name="runEnds">The run ends array (must be Int16Array, Int32Array, or Int64Array).</param>
+    /// <param name="values">The values array (can be any type).</param>
+    public RunEndEncodedArray(IArrowArray runEnds, IArrowArray values)
+        : this(CreateArrayData(runEnds, values), runEnds, values)
+    {
+    }
+
+    private RunEndEncodedArray(ArrayData data, IArrowArray runEnds, IArrowArray values)
+        : base(data)
+    {
+        data.EnsureBufferCount(0); // REE arrays have no buffers, only children
+        data.EnsureDataType(ArrowTypeId.RunEndEncoded);
+
+        ValidateRunEndsType(runEnds);
+        RunEnds = runEnds;
+        Values = values;
+    }
+
+    private static ArrayData CreateArrayData(IArrowArray runEnds, IArrowArray values)
+    {
+        ValidateRunEndsType(runEnds);
+
+        // The logical length of a REE array is determined by the last value in run_ends
+        int logicalLength = GetLogicalLength(runEnds);
+
+        var dataType = new RunEndEncodedType(runEnds.Data.DataType, values.Data.DataType);
+
+        return new ArrayData(
+            dataType,
+            logicalLength,
+            nullCount: 0, // REE arrays don't have a validity bitmap
+            offset: 0,
+            buffers: [],
+            children: [runEnds.Data, values.Data]);
+    }
+
+    private static void ValidateRunEndsType(IArrowArray runEnds)
+    {
+        ArrowTypeId typeId = runEnds.Data.DataType.TypeId;
+        if (typeId != ArrowTypeId.Int16 &&
+            typeId != ArrowTypeId.Int32 &&
+            typeId != ArrowTypeId.Int64)
+        {
+            throw new ArgumentException(
+                $"Run ends array must be Int16, Int32, or Int64, but got {typeId}",
+                nameof(runEnds));
+        }
+    }
+
+    private static int GetLogicalLength(IArrowArray runEnds)
+    {
+        if (runEnds.Length == 0)
+        {
+            return 0;
+        }
+
+        // Get the last run end value which represents the logical length
+        switch (runEnds)
+        {
+            case Int16Array int16Array:
+                return int16Array.GetValue(int16Array.Length - 1) ?? throw new ArgumentException("invalid length");
+            case Int32Array int32Array:
+                return int32Array.GetValue(int32Array.Length - 1) ?? throw new ArgumentException("invalid length");
+            case Int64Array int64Array:
+                {
+                    long? lastValue = int64Array.GetValue(int64Array.Length - 1);
+                    if (lastValue.HasValue && lastValue.Value > int.MaxValue)
+                    {
+                        throw new ArgumentException("Run ends value exceeds maximum supported length.");
+                    }
+                    return (int)(lastValue ?? throw new ArgumentException("invalid length"));
+                }
+            default:
+                throw new InvalidOperationException($"Unexpected run ends array type: {runEnds.Data.DataType.TypeId}");
+        }

Review Comment:
   GetLogicalLength can return a negative logical length if the last run_end value is negative (Int16/Int32/Int64). ArrayData doesn't validate length >= 0, so this can create invalid arrays and downstream failures. Validate that the final run_end is >= 0 (and ideally > 0 when runEnds.Length > 0) and throw a clear ArgumentException when invalid.



##########
src/Apache.Arrow/Arrays/ArrayDataConcatenator.cs:
##########
@@ -286,6 +287,201 @@ public void Visit(UnionType type)
 
             public void Visit(MapType type) => ConcatenateLists(type.UnsortedKey()); /* Can't tell if the output is still sorted */
 
+            public void Visit(RunEndEncodedType type)
+            {
+                ArrowTypeId runEndsTypeId = type.RunEndsDataType.TypeId;
+                if (runEndsTypeId != ArrowTypeId.Int16 &&
+                    runEndsTypeId != ArrowTypeId.Int32 &&
+                    runEndsTypeId != ArrowTypeId.Int64)
+                {
+                    throw new InvalidOperationException(
+                        $"Run-ends array must be Int16, Int32, or Int64, but got {runEndsTypeId}");
+                }
+
+                var slicedValues = new List<ArrayData>(_arrayDataList.Count);
+                ArrowBuffer.Builder<short> int16Builder = null;
+                ArrowBuffer.Builder<int> int32Builder = null;
+                ArrowBuffer.Builder<long> int64Builder = null;
+
+                switch (runEndsTypeId)
+                {
+                    case ArrowTypeId.Int16: int16Builder = new ArrowBuffer.Builder<short>(); break;
+                    case ArrowTypeId.Int32: int32Builder = new ArrowBuffer.Builder<int>(); break;
+                    case ArrowTypeId.Int64: int64Builder = new ArrowBuffer.Builder<long>(); break;
+                }
+
+                long baseOffset = 0;
+                int physicalRunCount = 0;
+
+                foreach (ArrayData arrayData in _arrayDataList)
+                {
+                    arrayData.EnsureDataType(type.TypeId);
+
+                    ArrayData runEndsData = arrayData.Children[0];
+                    ArrayData valuesData = arrayData.Children[1];
+
+                    if (runEndsData.DataType.TypeId != runEndsTypeId)
+                    {
+                        throw new ArgumentException(
+                            $"All run-end encoded arrays must have the same run-ends type. Expected <{runEndsTypeId}> but got <{runEndsData.DataType.TypeId}>.");
+                    }
+                    if (valuesData.DataType.TypeId != type.ValuesDataType.TypeId)
+                    {
+                        throw new ArgumentException(
+                            $"All run-end encoded arrays must have the same values type. Expected <{type.ValuesDataType.TypeId}> but got <{valuesData.DataType.TypeId}>.");
+                    }
+
+                    if (arrayData.Length == 0)
+                    {
+                        continue;
+                    }
+
+                    int logicalOffset = arrayData.Offset;
+                    int logicalLength = arrayData.Length;
+                    int logicalEnd = logicalOffset + logicalLength;
+
+                    int physicalStart;
+                    int physicalEndExclusive;
+
+                    switch (runEndsTypeId)
+                    {
+                        case ArrowTypeId.Int16:
+                            {
+                                ReadOnlySpan<short> re = runEndsData.Buffers[1].Span.CastTo<short>()
+                                    .Slice(runEndsData.Offset, runEndsData.Length);
+                                physicalStart = FindPhysicalStartInt16(re, logicalOffset);
+                                physicalEndExclusive = FindPhysicalEndExclusiveInt16(re, logicalEnd, physicalStart);
+                                for (int p = physicalStart; p < physicalEndExclusive; p++)
+                                {
+                                    int adjustedEnd = Math.Min((int)re[p], logicalEnd) - logicalOffset;
+                                    int16Builder.Append(checked((short)(baseOffset + adjustedEnd)));
+                                }

Review Comment:
   For Int16 run_ends, concatenation appends checked((short)(baseOffset + adjustedEnd)). If the concatenated logical length exceeds Int16.MaxValue, this will throw OverflowException mid-build. Consider proactively validating that the final logical length fits in the chosen run_ends type and throw a targeted ArgumentException with a clearer message.



##########
src/Apache.Arrow/Arrays/RunEndEncodedArray.cs:
##########
@@ -0,0 +1,222 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow;
+
+/// <summary>
+/// Represents a run-end encoded array.
+/// A run-end encoded array stores consecutive runs of the same value more efficiently.
+/// It contains two child arrays: run_ends (Int16/Int32/Int64) and values (any type).
+/// The run_ends array stores the cumulative end positions of each run.
+/// </summary>
+public class RunEndEncodedArray : Array
+{
+    /// <summary>
+    /// Gets the run ends array (Int16Array, Int32Array, or Int64Array).
+    /// This array contains the cumulative end indices for each run.
+    /// </summary>
+    public IArrowArray RunEnds { get; }
+
+    /// <summary>
+    /// Gets the values array.
+    /// This array contains the actual values that are run-length encoded.
+    /// </summary>
+    public IArrowArray Values { get; }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray from ArrayData.
+    /// </summary>
+    /// <param name="data">The array data containing run ends and values as children.</param>
+    public RunEndEncodedArray(ArrayData data)
+        : this(data, ArrowArrayFactory.BuildArray(data.Children[0]), ArrowArrayFactory.BuildArray(data.Children[1]))
+    {
+    }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray with specified run ends and values arrays.
+    /// </summary>
+    /// <param name="runEnds">The run ends array (must be Int16Array, Int32Array, or Int64Array).</param>
+    /// <param name="values">The values array (can be any type).</param>
+    public RunEndEncodedArray(IArrowArray runEnds, IArrowArray values)
+        : this(CreateArrayData(runEnds, values), runEnds, values)
+    {
+    }
+
+    private RunEndEncodedArray(ArrayData data, IArrowArray runEnds, IArrowArray values)
+        : base(data)
+    {
+        data.EnsureBufferCount(0); // REE arrays have no buffers, only children
+        data.EnsureDataType(ArrowTypeId.RunEndEncoded);
+
+        ValidateRunEndsType(runEnds);
+        RunEnds = runEnds;
+        Values = values;
+    }
+
+    private static ArrayData CreateArrayData(IArrowArray runEnds, IArrowArray values)
+    {
+        ValidateRunEndsType(runEnds);

Review Comment:
   RunEndEncodedArray doesn't validate that runEnds.Length == values.Length (one value per run). Creating an instance with mismatched lengths yields an invalid REE array and can lead to out-of-range access in consumers. Add a check in the constructor(s) (and potentially also validate runEnds.NullCount == 0).
   ```suggestion
           ValidateRunEndsType(runEnds);
           ValidateRunEndEncodedChildren(runEnds, values);
           RunEnds = runEnds;
           Values = values;
       }
   
       private static void ValidateRunEndEncodedChildren(IArrowArray runEnds, IArrowArray values)
       {
           if (runEnds.Length != values.Length)
           {
               throw new ArgumentException("Run-end encoded arrays require runEnds.Length to equal values.Length.", nameof(values));
           }
   
           if (runEnds.NullCount != 0)
           {
               throw new ArgumentException("Run-end encoded arrays require runEnds to have no nulls.", nameof(runEnds));
           }
       }
   
       private static ArrayData CreateArrayData(IArrowArray runEnds, IArrowArray values)
       {
           ValidateRunEndsType(runEnds);
           ValidateRunEndEncodedChildren(runEnds, values);
   ```



##########
src/Apache.Arrow/Arrays/RunEndEncodedArray.cs:
##########
@@ -0,0 +1,222 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow;
+
+/// <summary>
+/// Represents a run-end encoded array.
+/// A run-end encoded array stores consecutive runs of the same value more efficiently.
+/// It contains two child arrays: run_ends (Int16/Int32/Int64) and values (any type).
+/// The run_ends array stores the cumulative end positions of each run.
+/// </summary>
+public class RunEndEncodedArray : Array
+{
+    /// <summary>
+    /// Gets the run ends array (Int16Array, Int32Array, or Int64Array).
+    /// This array contains the cumulative end indices for each run.
+    /// </summary>
+    public IArrowArray RunEnds { get; }
+
+    /// <summary>
+    /// Gets the values array.
+    /// This array contains the actual values that are run-length encoded.
+    /// </summary>
+    public IArrowArray Values { get; }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray from ArrayData.
+    /// </summary>
+    /// <param name="data">The array data containing run ends and values as children.</param>
+    public RunEndEncodedArray(ArrayData data)
+        : this(data, ArrowArrayFactory.BuildArray(data.Children[0]), ArrowArrayFactory.BuildArray(data.Children[1]))
+    {
+    }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray with specified run ends and values arrays.
+    /// </summary>
+    /// <param name="runEnds">The run ends array (must be Int16Array, Int32Array, or Int64Array).</param>
+    /// <param name="values">The values array (can be any type).</param>
+    public RunEndEncodedArray(IArrowArray runEnds, IArrowArray values)
+        : this(CreateArrayData(runEnds, values), runEnds, values)
+    {
+    }
+
+    private RunEndEncodedArray(ArrayData data, IArrowArray runEnds, IArrowArray values)
+        : base(data)
+    {
+        data.EnsureBufferCount(0); // REE arrays have no buffers, only children
+        data.EnsureDataType(ArrowTypeId.RunEndEncoded);
+
+        ValidateRunEndsType(runEnds);
+        RunEnds = runEnds;
+        Values = values;
+    }
+
+    private static ArrayData CreateArrayData(IArrowArray runEnds, IArrowArray values)
+    {
+        ValidateRunEndsType(runEnds);
+
+        // The logical length of a REE array is determined by the last value in run_ends
+        int logicalLength = GetLogicalLength(runEnds);
+
+        var dataType = new RunEndEncodedType(runEnds.Data.DataType, values.Data.DataType);
+
+        return new ArrayData(
+            dataType,
+            logicalLength,
+            nullCount: 0, // REE arrays don't have a validity bitmap
+            offset: 0,
+            buffers: [],
+            children: [runEnds.Data, values.Data]);
+    }
+
+    private static void ValidateRunEndsType(IArrowArray runEnds)
+    {
+        ArrowTypeId typeId = runEnds.Data.DataType.TypeId;
+        if (typeId != ArrowTypeId.Int16 &&
+            typeId != ArrowTypeId.Int32 &&
+            typeId != ArrowTypeId.Int64)
+        {
+            throw new ArgumentException(
+                $"Run ends array must be Int16, Int32, or Int64, but got {typeId}",
+                nameof(runEnds));
+        }
+    }
+
+    private static int GetLogicalLength(IArrowArray runEnds)
+    {
+        if (runEnds.Length == 0)
+        {
+            return 0;
+        }
+
+        // Get the last run end value which represents the logical length
+        switch (runEnds)
+        {
+            case Int16Array int16Array:
+                return int16Array.GetValue(int16Array.Length - 1) ?? throw new ArgumentException("invalid length");
+            case Int32Array int32Array:
+                return int32Array.GetValue(int32Array.Length - 1) ?? throw new ArgumentException("invalid length");
+            case Int64Array int64Array:
+                {
+                    long? lastValue = int64Array.GetValue(int64Array.Length - 1);
+                    if (lastValue.HasValue && lastValue.Value > int.MaxValue)
+                    {
+                        throw new ArgumentException("Run ends value exceeds maximum supported length.");
+                    }
+                    return (int)(lastValue ?? throw new ArgumentException("invalid length"));
+                }
+            default:
+                throw new InvalidOperationException($"Unexpected run ends array type: {runEnds.Data.DataType.TypeId}");
+        }
+    }
+
+    /// <summary>
+    /// Finds the physical index in the run_ends array that contains the specified logical index.
+    /// </summary>
+    /// <param name="logicalIndex">The logical index in the decoded array.</param>
+    /// <returns>The physical index in the run_ends/values arrays.</returns>
+    public int FindPhysicalIndex(int logicalIndex)
+    {
+        if (logicalIndex < 0 || logicalIndex >= Length)
+        {
+            throw new ArgumentOutOfRangeException(nameof(logicalIndex));
+        }
+
+        // Binary search to find the run that contains this logical index
+        return RunEnds switch
+        {
+            Int16Array int16Array => BinarySearchRunEnds(int16Array, logicalIndex),
+            Int32Array int32Array => BinarySearchRunEnds(int32Array, logicalIndex),
+            Int64Array int64Array => BinarySearchRunEnds(int64Array, logicalIndex),
+            _ => throw new InvalidOperationException($"Unexpected run ends array type: {RunEnds.GetType()}"),
+        };
+    }
+
+    private static int BinarySearchRunEnds(Int16Array runEnds, int logicalIndex)
+    {
+        int left = 0;
+        int right = runEnds.Length - 1;
+
+        while (left < right)
+        {
+            int mid = left + (right - left) / 2;
+            int runEnd = runEnds.GetValue(mid) ?? throw new ArgumentException("invalid length"); ;
+
+            if (logicalIndex < runEnd)
+            {
+                right = mid;
+            }
+            else
+            {
+                left = mid + 1;
+            }
+        }
+
+        return left;
+    }
+
+    private static int BinarySearchRunEnds(Int32Array runEnds, int logicalIndex)
+    {
+        int left = 0;
+        int right = runEnds.Length - 1;
+
+        while (left < right)
+        {
+            int mid = left + (right - left) / 2;
+            int runEnd = runEnds.GetValue(mid) ?? 0;
+
+            if (logicalIndex < runEnd)
+            {
+                right = mid;
+            }
+            else
+            {
+                left = mid + 1;
+            }
+        }
+
+        return left;
+    }
+
+    private static int BinarySearchRunEnds(Int64Array runEnds, int logicalIndex)
+    {
+        int left = 0;
+        int right = runEnds.Length - 1;
+
+        while (left < right)
+        {
+            int mid = left + (right - left) / 2;
+            long runEnd = runEnds.GetValue(mid) ?? 0;
+

Review Comment:
   BinarySearchRunEnds for Int32/Int64 treats null run_end values as 0 ("?? 0"), which can silently return incorrect physical indices and mask invalid data. This should be consistent with the Int16 path (throw if a run_end is null) since run_ends must be non-null.



##########
src/Apache.Arrow/Arrays/ArrayDataConcatenator.cs:
##########
@@ -286,6 +287,201 @@ public void Visit(UnionType type)
 
             public void Visit(MapType type) => ConcatenateLists(type.UnsortedKey()); /* Can't tell if the output is still sorted */
 
+            public void Visit(RunEndEncodedType type)
+            {
+                ArrowTypeId runEndsTypeId = type.RunEndsDataType.TypeId;
+                if (runEndsTypeId != ArrowTypeId.Int16 &&
+                    runEndsTypeId != ArrowTypeId.Int32 &&
+                    runEndsTypeId != ArrowTypeId.Int64)
+                {
+                    throw new InvalidOperationException(
+                        $"Run-ends array must be Int16, Int32, or Int64, but got {runEndsTypeId}");
+                }
+
+                var slicedValues = new List<ArrayData>(_arrayDataList.Count);
+                ArrowBuffer.Builder<short> int16Builder = null;
+                ArrowBuffer.Builder<int> int32Builder = null;
+                ArrowBuffer.Builder<long> int64Builder = null;
+
+                switch (runEndsTypeId)
+                {
+                    case ArrowTypeId.Int16: int16Builder = new ArrowBuffer.Builder<short>(); break;
+                    case ArrowTypeId.Int32: int32Builder = new ArrowBuffer.Builder<int>(); break;
+                    case ArrowTypeId.Int64: int64Builder = new ArrowBuffer.Builder<long>(); break;
+                }
+
+                long baseOffset = 0;
+                int physicalRunCount = 0;
+
+                foreach (ArrayData arrayData in _arrayDataList)
+                {
+                    arrayData.EnsureDataType(type.TypeId);
+
+                    ArrayData runEndsData = arrayData.Children[0];
+                    ArrayData valuesData = arrayData.Children[1];
+
+                    if (runEndsData.DataType.TypeId != runEndsTypeId)
+                    {
+                        throw new ArgumentException(
+                            $"All run-end encoded arrays must have the same run-ends type. Expected <{runEndsTypeId}> but got <{runEndsData.DataType.TypeId}>.");
+                    }
+                    if (valuesData.DataType.TypeId != type.ValuesDataType.TypeId)
+                    {
+                        throw new ArgumentException(
+                            $"All run-end encoded arrays must have the same values type. Expected <{type.ValuesDataType.TypeId}> but got <{valuesData.DataType.TypeId}>.");
+                    }
+
+                    if (arrayData.Length == 0)
+                    {
+                        continue;
+                    }
+
+                    int logicalOffset = arrayData.Offset;
+                    int logicalLength = arrayData.Length;
+                    int logicalEnd = logicalOffset + logicalLength;
+
+                    int physicalStart;
+                    int physicalEndExclusive;
+
+                    switch (runEndsTypeId)
+                    {
+                        case ArrowTypeId.Int16:
+                            {
+                                ReadOnlySpan<short> re = runEndsData.Buffers[1].Span.CastTo<short>()
+                                    .Slice(runEndsData.Offset, runEndsData.Length);
+                                physicalStart = FindPhysicalStartInt16(re, logicalOffset);
+                                physicalEndExclusive = FindPhysicalEndExclusiveInt16(re, logicalEnd, physicalStart);
+                                for (int p = physicalStart; p < physicalEndExclusive; p++)
+                                {
+                                    int adjustedEnd = Math.Min((int)re[p], logicalEnd) - logicalOffset;
+                                    int16Builder.Append(checked((short)(baseOffset + adjustedEnd)));
+                                }
+                                break;
+                            }
+                        case ArrowTypeId.Int32:
+                            {
+                                ReadOnlySpan<int> re = runEndsData.Buffers[1].Span.CastTo<int>()
+                                    .Slice(runEndsData.Offset, runEndsData.Length);
+                                physicalStart = FindPhysicalStartInt32(re, logicalOffset);
+                                physicalEndExclusive = FindPhysicalEndExclusiveInt32(re, logicalEnd, physicalStart);
+                                for (int p = physicalStart; p < physicalEndExclusive; p++)
+                                {
+                                    int adjustedEnd = Math.Min(re[p], logicalEnd) - logicalOffset;
+                                    int32Builder.Append(checked((int)(baseOffset + adjustedEnd)));
+                                }
+                                break;
+                            }
+                        default: // Int64
+                            {
+                                ReadOnlySpan<long> re = runEndsData.Buffers[1].Span.CastTo<long>()
+                                    .Slice(runEndsData.Offset, runEndsData.Length);
+                                physicalStart = FindPhysicalStartInt64(re, logicalOffset);
+                                physicalEndExclusive = FindPhysicalEndExclusiveInt64(re, logicalEnd, physicalStart);
+                                for (int p = physicalStart; p < physicalEndExclusive; p++)
+                                {
+                                    long adjustedEnd = Math.Min(re[p], (long)logicalEnd) - logicalOffset;
+                                    int64Builder.Append(baseOffset + adjustedEnd);
+                                }
+                                break;
+                            }
+                    }
+
+                    int physicalCount = physicalEndExclusive - physicalStart;
+                    physicalRunCount += physicalCount;
+                    slicedValues.Add(valuesData.Slice(physicalStart, physicalCount));
+                    baseOffset += logicalLength;
+                }
+
+                ArrowBuffer runEndsValueBuffer = runEndsTypeId switch
+                {
+                    ArrowTypeId.Int16 => int16Builder.Build(_allocator),
+                    ArrowTypeId.Int32 => int32Builder.Build(_allocator),
+                    _ => int64Builder.Build(_allocator),
+                };
+
+                ArrayData runEndsResult = new ArrayData(
+                    type.RunEndsDataType, physicalRunCount, 0, 0,
+                    new[] { ArrowBuffer.Empty, runEndsValueBuffer });
+
+                ArrayData valuesResult = slicedValues.Count == 0
+                    ? new ArrayData(type.ValuesDataType, 0, 0, 0,
+                        new[] { ArrowBuffer.Empty })
+                    : Concatenate(slicedValues, _allocator);
+

Review Comment:
   When concatenating REE arrays, valuesResult for the all-empty case is created with a single empty buffer. This is invalid for many values types (e.g., primitive arrays require 2 buffers, binary requires 3, etc.) and will fail when building the values child array. Construct an empty ArrayData for type.ValuesDataType with the correct buffer/child layout (or reuse existing concatenation logic to produce an empty of the right shape).
   ```suggestion
                   ArrayData valuesResult;
                   if (slicedValues.Count == 0)
                   {
                       if (_arrayDataList.Count == 0)
                       {
                           throw new InvalidOperationException(
                               "Cannot construct REE values array data without any input arrays.");
                       }
   
                       // Reuse an existing empty values child so the buffer/child layout
                       // matches type.ValuesDataType for all Arrow value types.
                       valuesResult = _arrayDataList[0].Children[1];
                   }
                   else
                   {
                       valuesResult = Concatenate(slicedValues, _allocator);
                   }
   ```



##########
src/Apache.Arrow/Types/RunEndEncodedType.cs:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+
+namespace Apache.Arrow.Types;
+
+/// <summary>
+/// Represents a run-end encoded array type.
+/// Contains two child arrays: run_ends and values.
+/// The run_ends child array must be a 16/32/64-bit signed integer array
+/// which encodes the indices at which the run with the value in 
+/// each corresponding index in the values child array ends.
+/// </summary>
+public sealed class RunEndEncodedType : NestedType
+{
+    public override ArrowTypeId TypeId => ArrowTypeId.RunEndEncoded;
+    public override string Name => "run_end_encoded";
+
+    /// <summary>
+    /// Gets the run ends field (must be Int16, Int32, or Int64).
+    /// </summary>
+    public Field RunEndsField => Fields[0];
+
+    /// <summary>
+    /// Gets the values field (can be any type).
+    /// </summary>
+    public Field ValuesField => Fields[1];
+
+    /// <summary>
+    /// Gets the data type of the run ends array.
+    /// </summary>
+    public IArrowType RunEndsDataType => RunEndsField.DataType;
+
+    /// <summary>
+    /// Gets the data type of the values array.
+    /// </summary>
+    public IArrowType ValuesDataType => ValuesField.DataType;
+
+    /// <summary>
+    /// Creates a new RunEndEncodedType with the specified run ends and values fields.
+    /// </summary>
+    /// <param name="runEndsField">The run ends field (must be Int16, Int32, or Int64).</param>
+    /// <param name="valuesField">The values field (can be any type).</param>
+    public RunEndEncodedType(Field runEndsField, Field valuesField)
+        : base([runEndsField, valuesField])
+    {
+        ValidateRunEndsType(runEndsField.DataType);
+    }

Review Comment:
   ValidateRunEndsType throws an ArgumentException with paramName "runEndsDataType", but in the Field-based constructor the invalid input originates from runEndsField.DataType. Consider passing a paramName that matches the public constructor parameter (e.g., nameof(runEndsField)) to make exception reporting clearer for callers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: Add support for Run-End Encoded arrays [arrow-dotnet]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #308:
URL: https://github.com/apache/arrow-dotnet/pull/308#discussion_r3047801024


##########
src/Apache.Arrow/Ipc/ArrowStreamWriter.cs:
##########
@@ -797,6 +840,7 @@ private protected void WriteRecordBatchInternal(RecordBatch recordBatch)
                 recordBatchOffset, recordBatchBuilder.TotalLength);
 
             long bufferLength = WriteBufferData(recordBatchBuilder.Buffers);
+            recordBatchBuilder.DisposeDeferredArrays();
 

Review Comment:
   DisposeDeferredArrays() is only called after WriteBufferData. If WriteMessage/WriteBufferData throws, the normalized REE arrays held in _deferredDisposals will leak (and may retain native buffers) because they’re never disposed. Wrap the message/body write in a try/finally (or make ArrowRecordBatchFlatBufferBuilder IDisposable and dispose it in finally) so deferred arrays are always released.



##########
src/Apache.Arrow/Ipc/ArrowStreamWriter.cs:
##########
@@ -837,6 +881,7 @@ private protected async Task WriteRecordBatchInternalAsync(RecordBatch recordBat
                 cancellationToken).ConfigureAwait(false);
 
             long bufferLength = await WriteBufferDataAsync(recordBatchBuilder.Buffers, cancellationToken).ConfigureAwait(false);
+            recordBatchBuilder.DisposeDeferredArrays();
 

Review Comment:
   DisposeDeferredArrays() should run in a finally block here as well. As written, an exception from WriteMessageAsync/WriteBufferDataAsync will skip disposal of any deferred normalized arrays, causing a memory/resource leak.



##########
src/Apache.Arrow/Ipc/ArrowStreamWriter.cs:
##########
@@ -986,6 +1031,7 @@ private protected void WriteDictionary(long id, IArrowType valueType, IArrowArra
                 dictionaryBatchOffset, recordBatchBuilder.TotalLength);
 
             long bufferLength = WriteBufferData(recordBatchBuilder.Buffers);
+            recordBatchBuilder.DisposeDeferredArrays();
 

Review Comment:
   Same issue as record batches: DisposeDeferredArrays() is not in a finally. If WriteMessage/WriteBufferData throws while writing a dictionary batch, deferred arrays (e.g., normalized REE slices) won’t be disposed.



##########
src/Apache.Arrow/Arrays/RunEndEncodedArray.cs:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using Apache.Arrow.Memory;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow;
+
+/// <summary>
+/// Represents a run-end encoded array.
+/// A run-end encoded array stores consecutive runs of the same value more efficiently.
+/// It contains two child arrays: run_ends (Int16/Int32/Int64) and values (any type).
+/// The run_ends array stores the cumulative end positions of each run.
+/// </summary>
+public class RunEndEncodedArray : Array
+{
+    /// <summary>
+    /// Gets the run ends array (Int16Array, Int32Array, or Int64Array).
+    /// This array contains the cumulative end indices for each run.
+    /// </summary>
+    public IArrowArray RunEnds { get; }
+
+    /// <summary>
+    /// Gets the values array.
+    /// This array contains the actual values that are run-length encoded.
+    /// </summary>
+    public IArrowArray Values { get; }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray from ArrayData.
+    /// </summary>
+    /// <param name="data">The array data containing run ends and values as children.</param>
+    public RunEndEncodedArray(ArrayData data)
+        : this(data, ArrowArrayFactory.BuildArray(data.Children[0]), ArrowArrayFactory.BuildArray(data.Children[1]))
+    {
+    }
+
+    /// <summary>
+    /// Creates a new RunEndEncodedArray with specified run ends and values arrays.
+    /// </summary>
+    /// <param name="runEnds">The run ends array (must be Int16Array, Int32Array, or Int64Array).</param>
+    /// <param name="values">The values array (can be any type).</param>
+    public RunEndEncodedArray(IArrowArray runEnds, IArrowArray values)
+        : this(CreateArrayData(runEnds, values), runEnds, values)
+    {
+    }
+
+    private RunEndEncodedArray(ArrayData data, IArrowArray runEnds, IArrowArray values)
+        : base(data)
+    {
+        data.EnsureBufferCount(0); // REE arrays have no buffers, only children
+        data.EnsureDataType(ArrowTypeId.RunEndEncoded);
+
+        if (data.NullCount != 0)
+        {
+            throw new ArgumentException(
+                $"Run-end encoded arrays have no top-level validity bitmap and must report null count 0, but got {data.NullCount}.",
+                nameof(data));
+        }
+
+        ValidateRunEndsType(runEnds);
+
+        if (runEnds.Length != values.Length)
+        {
+            throw new ArgumentException(
+                $"Run ends array length ({runEnds.Length}) must equal values array length ({values.Length}).");
+        }

Review Comment:
   When constructing from ArrayData, the top-level Length/Offset is not validated against the run_ends values. This allows malformed ArrayData where Offset+Length exceeds the last run_end, which will break Normalize()/FindPhysicalIndex semantics. Add a validation that the slice range (data.Offset + data.Length) does not exceed the logical length implied by the last run_end value.



##########
src/Apache.Arrow/Ipc/ArrowStreamWriter.cs:
##########
@@ -1010,6 +1056,7 @@ private protected async Task WriteDictionaryAsync(long id, IArrowType valueType,
                 dictionaryBatchOffset, recordBatchBuilder.TotalLength, cancellationToken).ConfigureAwait(false);
 
             long bufferLength = await WriteBufferDataAsync(recordBatchBuilder.Buffers, cancellationToken).ConfigureAwait(false);
+            recordBatchBuilder.DisposeDeferredArrays();
 

Review Comment:
   Async dictionary writing has the same leak hazard: DisposeDeferredArrays() should be guaranteed via try/finally so deferred normalized arrays are disposed even when WriteMessageAsync/WriteBufferDataAsync throws.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: Add support for Run-End Encoded arrays [arrow-dotnet]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on code in PR #308:
URL: https://github.com/apache/arrow-dotnet/pull/308#discussion_r3048363746


##########
src/Apache.Arrow/Ipc/ArrowStreamWriter.cs:
##########
@@ -797,6 +840,7 @@ private protected void WriteRecordBatchInternal(RecordBatch recordBatch)
                 recordBatchOffset, recordBatchBuilder.TotalLength);
 
             long bufferLength = WriteBufferData(recordBatchBuilder.Buffers);
+            recordBatchBuilder.DisposeDeferredArrays();
 

Review Comment:
   I'm going to address this partially. An exception inside the factory method will still end up "leaking" the buffer (it will live until its finalizer runs) but one that happens during writing will get properly disposed. To do any better requires restructuring of the code is as better achieved in a separate refactoring PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org