You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2025/05/25 03:09:18 UTC
[I] Support `map_values` [datafusion-comet]
comphead opened a new issue, #1789:
URL: https://github.com/apache/datafusion-comet/issues/1789
`map_values` will be addressed in following PR. It got a correctness issue
```
- read map[struct, struct] from parquet *** FAILED *** (1 second, 478 milliseconds)
Results do not match for query:
Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
Timezone Env:
== Parsed Logical Plan ==
Project [map_values(map1#508970).id2 AS map_values(map1).id2#509026]
+- SubqueryAlias tbl
+- View (`tbl`, [map1#508970])
+- Relation [map1#508970] parquet
== Analyzed Logical Plan ==
map_values(map1).id2: array<bigint>
Project [map_values(map1#508970).id2 AS map_values(map1).id2#509026]
+- SubqueryAlias tbl
+- View (`tbl`, [map1#508970])
+- Relation [map1#508970] parquet
== Optimized Logical Plan ==
Project [map_values(map1#508970).id2 AS map_values(map1).id2#509026]
+- Relation [map1#508970] parquet
== Physical Plan ==
*(1) CometColumnarToRow
+- CometProject [map_values(map1).id2#509026], [map_values(map1#508970).id2 AS map_values(map1).id2#509026]
+- CometScan parquet [map1#508970] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/__w/datafusion-comet/datafusion-comet/spark/target/tmp/spark-062..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<map1:map<struct<id:bigint,id2:bigint,id3:bigint>,struct<id2:bigint>>>
== Results ==
== Results ==
!== Correct Answer - 5 == == Spark Answer - 5 ==
struct<map_values(map1).id2:array<bigint>> struct<map_values(map1).id2:array<bigint>>
![ArrayBuffer(3)] [ArrayBuffer(0)]
![ArrayBuffer(4)] [ArrayBuffer(3)]
![ArrayBuffer(null)] [ArrayBuffer(4)]
[null] [null]
[null] [null] (QueryTest.scala:244)
```
_Originally posted by @comphead in https://github.com/apache/datafusion-comet/issues/1788#issuecomment-2907578088_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `map_values` [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #1789:
URL: https://github.com/apache/datafusion-comet/issues/1789#issuecomment-2920846368
So the issue goes down to the reader, the problem is Spark can understand on fly parquet schema whereas DataFusion still static. To reproduce the problem
Create some data
```
val q = """
| select map(str0, str1) c0 from
| (
| select named_struct('a', cast(3 as long), 'b', cast(4 as long), 'c', cast(5 as long)) str0,
| named_struct('x', cast(6 as long), 'y', 'abc', 'z', cast(8 as long)) str1 union all
| select named_struct('a', cast(31 as long), 'b', cast(41 as long), 'c', cast(51 as long)), null
| )
|""".stripMargin
spark.sql(q).repartition(1).write.parquet("/tmp/t1")
```
Check parquet file metadata. The file meta shows both the fields `x, y, z` are required, but entire group `value` is optional. This is quite weird. Spark schema in properties shows `value` is not nullable although it contains null
```
parquet meta /tmp/t1/part-00000-340a4bdf-2e2c-42a8-a38a-01b47ab7d3c0-c000.snappy.parquet
File path: /tmp/t1/part-00000-340a4bdf-2e2c-42a8-a38a-01b47ab7d3c0-c000.snappy.parquet
Created by: parquet-mr version 1.13.1 (build db4183109d5b734ec5930d870cdae161e408ddba)
Properties:
org.apache.spark.version: 3.5.5
org.apache.spark.sql.parquet.row.metadata: {"type":"struct","fields":[{"name":"c0","type":{"type":"map","keyType":{"type":"struct","fields":[{"name":"a","type":"long","nullable":false,"metadata":{}},{"name":"b","type":"long","nullable":false,"metadata":{}},{"name":"c","type":"long","nullable":false,"metadata":{}}]},"valueType":{"type":"struct","fields":[{"name":"x","type":"long","nullable":false,"metadata":{}},{"name":"y","type":"string","nullable":false,"metadata":{}},{"name":"z","type":"long","nullable":false,"metadata":{}}]},"valueContainsNull":true},"nullable":false,"metadata":{}}]}
Schema:
message spark_schema {
required group c0 (MAP) {
repeated group key_value {
required group key {
required int64 a;
required int64 b;
required int64 c;
}
optional group value {
required int64 x;
required binary y (STRING);
required int64 z;
}
}
}
}
```
Reading the file through Spark and print schema. Note `x, y, z` are nullable now although it is not in Parquet.
```
scala> spark.read.parquet("/tmp/t1").printSchema
root
|-- c0: map (nullable = true)
| |-- key: struct
| | |-- a: long (nullable = true)
| | |-- b: long (nullable = true)
| | |-- c: long (nullable = true)
| |-- value: struct (valueContainsNull = true)
| | |-- x: long (nullable = true)
| | |-- y: string (nullable = true)
| | |-- z: long (nullable = true)
```
I assuma Spark is smart enough to infer nullability of `x,y,z` based on the `value` is optional in Parquet file and tweak the schema accordingly. DataFusion cannot do that afaik.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `map_values` [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead closed issue #1789: Support `map_values`
URL: https://github.com/apache/datafusion-comet/issues/1789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `map_values` [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1789:
URL: https://github.com/apache/datafusion-comet/issues/1789#issuecomment-2922749343
The investigation seems to be diverging from `map_values` and seems more focused on map value nullability, and in particular struct field nullability and how that's encoded and read back in Parquet.
If that's the case, I suggest starting from simpler questions without concerning maps:
- Can a _struct_ that is non-nullable have nullable fields (I believe yes)? How is that encoded in Parquet by Spark, and how does DF treat that?
- Can a _struct_ that is nullable have non-nullable fields (I believe yes)? How is that encoded in Parquet by Spark, and how does DF treat that?
Once we know that I suggest we go back to the map issue with structs as keys and values and try to make more sense of what's going on there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `map_values` [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #1789:
URL: https://github.com/apache/datafusion-comet/issues/1789#issuecomment-2920864833
Datafusion test
```
let batch = session_ctx
.read_parquet("/tmp/t1/part-00000-340a4bdf-2e2c-42a8-a38a-01b47ab7d3c0-c000.snappy.parquet",
ParquetReadOptions::default()).await?.collect().await?;
print_batches(&*batch);
dbg!(&batch);
```
the print shows correctly
```
+--------------------------------------------+
| c0 |
+--------------------------------------------+
| {{a: 3, b: 4, c: 5}: {x: 6, y: abc, z: 8}} |
| {{a: 31, b: 41, c: 51}: } |
+--------------------------------------------+
```
But the batch itself is not
```
-- child 0: "x" (Int64)
PrimitiveArray<Int64>
[
0,
]
-- child 1: "y" (Utf8)
StringArray
[
"",
]
-- child 2: "z" (Int64)
PrimitiveArray<Int64>
[
0,
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `map_values` [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #1789:
URL: https://github.com/apache/datafusion-comet/issues/1789#issuecomment-2920869741
UPD: apparently DF checks validity on runtime and Comet does not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org