You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jamesmcm (via GitHub)" <gi...@apache.org> on 2024/05/23 16:10:52 UTC
[I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]
jamesmcm opened a new issue, #10639:
URL: https://github.com/apache/datafusion/issues/10639
### Is your feature request related to a problem or challenge?
This job in Ballista fails:
```rust
let avro_file = "gs://...";
let metadata_df = ctx
.read_avro(avro_file, AvroReadOptions::default())
.await?
.select(vec![col("id"), col("nested_array")])?
.unnest_column("nested_array")?
```
```
Error: DataFusionError(Internal("failed to serialize logical plan: Internal(\"LogicalPlan serde is not yet implemented for Unnest\")"))
```
Due to the lack of ProtoBuf serialisation here - https://github.com/apache/datafusion/blob/main/datafusion/proto/src/logical_plan/mod.rs#L1527-L1529
### Describe the solution you'd like
ProtoBuf Serialisation would be added so that Unnest will be usable in Ballista jobs.
### Describe alternatives you've considered
I also couldn't get taking just the head to work:
```rust
let metadata_df = ctx
.read_avro(avro_file, AvroReadOptions::default())
.await?
.select(vec![
col("id"),
col("nested_array")
.index(lit(1)) // Note index starts at 1
.field("array_id_1")
.alias("array_id_1"),
col("nested_array")
.index(lit(1))
.field("array_id_2")
.alias("array_id_2"),
])?;
```
which gave:
```
Error: DataFusionError(Plan("Only ints are valid as an indexed field in a list"))
```
But really unnesting support is needed anyway.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]
Posted by "akoshchiy (via GitHub)" <gi...@apache.org>.
akoshchiy commented on issue #10639:
URL: https://github.com/apache/datafusion/issues/10639#issuecomment-2132192230
@alamb Hi! Can I try to fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #10639: Implement protobuf serialization for LogicalPlan::Unnest
URL: https://github.com/apache/datafusion/issues/10639
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10639:
URL: https://github.com/apache/datafusion/issues/10639#issuecomment-2134982310
Yeah thanks a alot @akoshchiy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]
Posted by "akoshchiy (via GitHub)" <gi...@apache.org>.
akoshchiy commented on issue #10639:
URL: https://github.com/apache/datafusion/issues/10639#issuecomment-2132921567
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]
Posted by "jamesmcm (via GitHub)" <gi...@apache.org>.
jamesmcm commented on issue #10639:
URL: https://github.com/apache/datafusion/issues/10639#issuecomment-2134927449
Wow, thanks for the quick response! I'll test out the PR later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10639:
URL: https://github.com/apache/datafusion/issues/10639#issuecomment-2131196191
Thanks @jamesmcm -- this would be a good first issue I think for someone with Rust experience looking to help with DataFusion. The protobuf serialization code is a bit hairy I find, but there are many examples of how to do it that we can follow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org