You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jamesmcm (via GitHub)" <gi...@apache.org> on 2024/05/23 16:10:52 UTC

[I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]

jamesmcm opened a new issue, #10639:
URL: https://github.com/apache/datafusion/issues/10639

   ### Is your feature request related to a problem or challenge?
   
   This job in Ballista fails:
   
   ```rust
       let avro_file = "gs://...";
   
       let metadata_df = ctx
           .read_avro(avro_file, AvroReadOptions::default())
           .await?
           .select(vec![col("id"), col("nested_array")])?
           .unnest_column("nested_array")?
   ```
   
   ```
   Error: DataFusionError(Internal("failed to serialize logical plan: Internal(\"LogicalPlan serde is not yet implemented for Unnest\")"))
   ```
   
   Due to the lack of ProtoBuf serialisation here - https://github.com/apache/datafusion/blob/main/datafusion/proto/src/logical_plan/mod.rs#L1527-L1529
   
   ### Describe the solution you'd like
   
   ProtoBuf Serialisation would be added so that Unnest will be usable in Ballista jobs.
   
   ### Describe alternatives you've considered
   
   I also couldn't get taking just the head to work:
   
   ```rust
       let metadata_df = ctx
           .read_avro(avro_file, AvroReadOptions::default())
           .await?
           .select(vec![
               col("id"),
               col("nested_array")
                   .index(lit(1)) // Note index starts at 1
                   .field("array_id_1")
                   .alias("array_id_1"),
               col("nested_array")
                   .index(lit(1))
                   .field("array_id_2")
                   .alias("array_id_2"),
           ])?;
   ```
   
   which gave:
   
   ```
   Error: DataFusionError(Plan("Only ints are valid as an indexed field in a list"))
   ```
   
   But really unnesting support is needed anyway.
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]

Posted by "akoshchiy (via GitHub)" <gi...@apache.org>.
akoshchiy commented on issue #10639:
URL: https://github.com/apache/datafusion/issues/10639#issuecomment-2132192230

   @alamb Hi! Can I try to fix it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #10639: Implement protobuf serialization for LogicalPlan::Unnest
URL: https://github.com/apache/datafusion/issues/10639


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10639:
URL: https://github.com/apache/datafusion/issues/10639#issuecomment-2134982310

   Yeah thanks a  alot @akoshchiy 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]

Posted by "akoshchiy (via GitHub)" <gi...@apache.org>.
akoshchiy commented on issue #10639:
URL: https://github.com/apache/datafusion/issues/10639#issuecomment-2132921567

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]

Posted by "jamesmcm (via GitHub)" <gi...@apache.org>.
jamesmcm commented on issue #10639:
URL: https://github.com/apache/datafusion/issues/10639#issuecomment-2134927449

   Wow, thanks for the quick response! I'll test out the PR later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Implement protobuf serialization for LogicalPlan::Unnest [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10639:
URL: https://github.com/apache/datafusion/issues/10639#issuecomment-2131196191

   Thanks @jamesmcm  -- this would be a good first issue I think for someone with Rust experience looking to help with DataFusion. The protobuf serialization code is a bit hairy I find, but there are many examples of how to do it that we can follow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org