You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "CurtHagenlocher (via GitHub)" <gi...@apache.org> on 2026/09/27 16:47:40 UTC

[PR] fix: Concatenate null and dictionary arrays [arrow-dotnet]

CurtHagenlocher opened a new pull request, #450:
URL: https://github.com/apache/arrow-dotnet/pull/450

   ## What's Changed
   
   `ArrayDataConcatenator` (and so `ArrowArrayConcatenator`) couldn't handle two array types:
   
   - **Null arrays** had no visitor case and threw `NotImplementedException`, at the top level or as a child (e.g. a null-typed struct field, which Parquet readers hit for pyarrow `pa.null()` columns). They now concatenate to a null array of the combined length.
   - **Dictionary arrays** went through `Visit(FixedWidthType)` because `DictionaryType` derives from `FixedWidthType`. That concatenated the indices but dropped the dictionary, so `ArrowArrayConcatenator` threw `Dictionary must not be null` and `ArrayDataConcatenator` silently returned dictionary-typed data with no dictionary.
   
   For dictionaries:
   - If every non-empty input shares the same dictionary (the same `ArrayData`, or distinct `ArrayData` over the same memory, e.g. after `Retain`/`SliceShared`), the indices are concatenated and that dictionary is kept.
   - Otherwise the dictionaries of the non-empty inputs are concatenated and each input's indices are shifted by the combined length of the dictionaries before it. Null slots get index 0 instead of a shifted, possibly out-of-range value. If the combined dictionary is too big for the index type, this throws `OverflowException` instead of wrapping around.
   - Inputs with different index types or value types are rejected with `ArgumentException`.
   
   Dictionaries that are equal in content but stored separately take the concatenation path. The result is correct but not deduplicated. Unifying dictionaries could be added later if needed.
   
   New tests cover null arrays, a struct with a null field, shared and different dictionaries (including nulls, slices and empty inputs), index overflow at the `int8` boundary, and `ArrayDataConcatenator` keeping the dictionary.
   
   Closes #446.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Concatenate null and dictionary arrays [arrow-dotnet]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #450:
URL: https://github.com/apache/arrow-dotnet/pull/450#discussion_r4116149110


##########
src/Apache.Arrow/Arrays/ArrayDataConcatenator.cs:
##########
@@ -125,6 +127,200 @@ public void Visit(FixedWidthType type)
                 Result = new ArrayData(resolvedType, _totalLength, _totalNullCount, 0, new ArrowBuffer[] { validityBuffer, valueBuffer });
             }
 
+            public void Visit(NullType type)
+            {
+                foreach (ArrayData arrayData in _arrayDataList)
+                {
+                    arrayData.EnsureDataType(type.TypeId);
+                }
+
+                // A null array has no buffers; every slot is null.
+                Result = new ArrayData(type, _totalLength, _totalLength, 0, System.Array.Empty<ArrowBuffer>());
+            }
+
+            public void Visit(DictionaryType type)
+            {
+                CheckData(type, 2);
+                var indexType = (IntegerType)type.IndexType;
+
+                // Inputs with no rows don't contribute any dictionary entries.
+                var contributing = new List<ArrayData>(_arrayDataList.Count);
+                foreach (ArrayData arrayData in _arrayDataList)
+                {
+                    var otherType = (DictionaryType)arrayData.DataType;
+                    if (otherType.IndexType.TypeId != indexType.TypeId)
+                    {
+                        throw new ArgumentException(
+                            $"Cannot concatenate dictionary arrays with different index types: {indexType.Name} vs {otherType.IndexType.Name}");

Review Comment:
   The result always uses the first input's `DictionaryType`, including its `Ordered` flag, but the fallback dictionary is built by appending dictionaries end to end. If the first dictionary is marked ordered, inputs such as `[b]` followed by `[a]` produce `[b, a]` while still advertising an ordered dictionary, which is false. Reject incompatible ordered inputs or construct a non-ordered result when dictionaries must be combined.
   
   This issue also appears on line 156 of the same file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Concatenate null and dictionary arrays [arrow-dotnet]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on code in PR #450:
URL: https://github.com/apache/arrow-dotnet/pull/450#discussion_r4116206223


##########
src/Apache.Arrow/Arrays/ArrayDataConcatenator.cs:
##########
@@ -125,6 +127,200 @@ public void Visit(FixedWidthType type)
                 Result = new ArrayData(resolvedType, _totalLength, _totalNullCount, 0, new ArrowBuffer[] { validityBuffer, valueBuffer });
             }
 
+            public void Visit(NullType type)
+            {
+                foreach (ArrayData arrayData in _arrayDataList)
+                {
+                    arrayData.EnsureDataType(type.TypeId);
+                }
+
+                // A null array has no buffers; every slot is null.
+                Result = new ArrayData(type, _totalLength, _totalLength, 0, System.Array.Empty<ArrowBuffer>());
+            }
+
+            public void Visit(DictionaryType type)
+            {
+                CheckData(type, 2);
+                var indexType = (IntegerType)type.IndexType;
+
+                // Inputs with no rows don't contribute any dictionary entries.
+                var contributing = new List<ArrayData>(_arrayDataList.Count);
+                foreach (ArrayData arrayData in _arrayDataList)
+                {
+                    var otherType = (DictionaryType)arrayData.DataType;
+                    if (otherType.IndexType.TypeId != indexType.TypeId)
+                    {
+                        throw new ArgumentException(
+                            $"Cannot concatenate dictionary arrays with different index types: {indexType.Name} vs {otherType.IndexType.Name}");

Review Comment:
   Good catch, fixed in e2d1b28. When different dictionaries are appended, the result is now never ordered: even if every input is ordered, the appended entries aren't necessarily in order, and nothing verifies that. When every input shares one dictionary, nothing is appended, so the result is ordered only if every input says so. That commit also moves the dictionary concatenation ahead of the buffer allocations and releases the buffers if building the indices throws, which covers the leak from the overview. It also adds a test for dictionaries held in separate `ArrayData` objects over the same memory that checks the result still works after the inputs are disposed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Concatenate null and dictionary arrays [arrow-dotnet]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on PR #450:
URL: https://github.com/apache/arrow-dotnet/pull/450#issuecomment-6134027007

   FWIW there's something I don't like in this PR and I'll come back to it shortly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Concatenate null and dictionary arrays [arrow-dotnet]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher merged PR #450:
URL: https://github.com/apache/arrow-dotnet/pull/450


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org