You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/05/27 22:45:06 UTC
[I] [Feature] support size() for MapType inputs [datafusion-comet]
andygrove opened a new issue, #4472:
URL: https://github.com/apache/datafusion-comet/issues/4472
## Describe the bug
Spark's `size(expr)` accepts both `ArrayType` and `MapType` inputs (`Size.inputTypes = Seq(TypeCollection(ArrayType, MapType))` in `collectionOperations.scala`, identical across 3.4.3 / 3.5.8 / 4.0.1 / 4.1.1). Comet's `CometSize` only supports `ArrayType`; for `MapType` it returns `Unsupported(Some("size does not support map inputs"))` and falls back to Spark.
Surfaced by the collection-expressions audit in apache/datafusion-comet#4471.
## Steps to reproduce
```sql
CREATE TABLE t(m map<string, int>) USING parquet;
INSERT INTO t VALUES (map('a', 1, 'b', 2));
SELECT size(m) FROM t;
```
Spark returns `2`. Comet falls back to Spark for the entire plan node.
## Expected behavior
Native support for `size(<map>)`. Arrow's `MapArray` carries a length per row that can drive the same `numElements` semantics Spark uses, with the existing `legacySizeOfNull` config-driven null handling that `CometSize` already implements for arrays.
## Additional context
- Serde: `CometSize` in `spark/src/main/scala/org/apache/comet/serde/arrays.scala` (line ~640)
- Native: routes through `size` scalar function in `comet_scalar_funcs.rs`; the size UDF would need a `MapType` branch.
- Related: `cardinality` is an alias for `size` in Spark and would benefit from the same fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Feature] support size() for MapType inputs [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #4472: [Feature] support size() for MapType inputs
URL: https://github.com/apache/datafusion-comet/issues/4472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Feature] support size() for MapType inputs [datafusion-comet]
Posted by "marvelshan (via GitHub)" <gi...@apache.org>.
marvelshan commented on issue #4472:
URL: https://github.com/apache/datafusion-comet/issues/4472#issuecomment-4608909405
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org