You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "milenkovicm (via GitHub)" <gi...@apache.org> on 2025/07/28 12:26:45 UTC

[I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

milenkovicm opened a new issue, #16944:
URL: https://github.com/apache/datafusion/issues/16944

   ### Is your feature request related to a problem or challenge?
   
   At the moment ballista overrides `LogicalExtensionCodec::try_decode_file_format` & `LogicalExtensionCodec::try_encode_file_format` providing support for:
   
   ```rust
   file_format_codecs: vec![
               Arc::new(ParquetLogicalExtensionCodec {}),
               Arc::new(CsvLogicalExtensionCodec {}),
               Arc::new(JsonLogicalExtensionCodec {}),
               Arc::new(ArrowLogicalExtensionCodec {}),
               Arc::new(AvroLogicalExtensionCodec {}),
           ],
   ```
   
   as seen at [1]. Should we want to integrate ballista with datafusion python we would need to provide a custom `LogicalExtensionCodec` implementing same logic or reusing ballista `LogicalExtensionCodec` implementation. As this file types are supported out of the box in datafusion would it make sense to implement encoder/decoder or them in `DefaultLogicalExtensionCodec`?
   
   [1]: https://github.com/milenkovicm/arrow-ballista/blob/e1e9f6ca423fd558664a7f2fb3b1bc3ed07d7db8/ballista/core/src/serde/mod.rs#L164-L165
   
   ### Describe the solution you'd like
   
   Should this proposal make sense, implement support for it in `DefaultLogicalExtensionCodec` similar to what is supported in ballista already:
   
   ```rust
     fn try_decode_file_format(
           &self,
           buf: &[u8],
           ctx: &datafusion::prelude::SessionContext,
       ) -> Result<Arc<dyn datafusion::datasource::file_format::FileFormatFactory>> {
           let proto = FileFormatProto::decode(buf)
               .map_err(|e| DataFusionError::Internal(e.to_string()))?;
   
           let codec = self
               .file_format_codecs
               .get(proto.encoder_position as usize)
               .ok_or(DataFusionError::Internal(
                   "Can't find required codec in file codec list".to_owned(),
               ))?;
   
           codec.try_decode_file_format(&proto.blob, ctx)
       }
   
       fn try_encode_file_format(
           &self,
           buf: &mut Vec<u8>,
           node: Arc<dyn datafusion::datasource::file_format::FileFormatFactory>,
       ) -> Result<()> {
           let mut blob = vec![];
           let (encoder_position, _) =
               self.try_any(|codec| codec.try_encode_file_format(&mut blob, node.clone()))?;
   
           let proto = FileFormatProto {
               encoder_position,
               blob,
           };
           proto
               .encode(buf)
               .map_err(|e| DataFusionError::Internal(e.to_string()))
       }
   ```
   
   https://github.com/milenkovicm/arrow-ballista/blob/e1e9f6ca423fd558664a7f2fb3b1bc3ed07d7db8/ballista/core/src/serde/mod.rs#L214-L215
   
   ### Describe alternatives you've considered
   
   alternative would be to keep everything as it is.
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

Posted by "jizezhang (via GitHub)" <gi...@apache.org>.
jizezhang commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3508846189

   Hi, if no one is actively working on this task, I am interested. To check my understanding, it seems like the goal is to move some functionality from ballista codec to `datafusion_proto::logical_plan::DefaultLogicalExtensionCodec`? Specifically,
   - add a `default` method for `DefaultLogicalExtensionCodec` that populates a list of file format codecs under `datafusion_proto::logical_plan::file_formats` module
   - add a `FileFormatProto` struct as in ballista
   - implement  `try_decode_file_format` and `try_encode_file_format` based on ballista implementation?
   
   If those sound correct, I can work on getting a PR. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm closed issue #16944: make DefaultLogicalExtensionCodec support serialisation of build in file formats
URL: https://github.com/apache/datafusion/issues/16944


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3224141088

   /take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

Posted by "Acfboy (via GitHub)" <gi...@apache.org>.
Acfboy commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3763610202

   Hi @peterxcli, are you still working on this? I notice relevant pr has some conflicts and pending review comments. Since there hasn't been activity for a while, if you're busy, I'd be happy to take over the PR and drive it to completion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3431276831

   any progress on this one @peterxcli 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3778092702

   No problem @peterxcli, I’m not in a hurry at all. This task was opened to give new contributors a chance to get involved with datafusion. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3224141624

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

Posted by "Acfboy (via GitHub)" <gi...@apache.org>.
Acfboy commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3977028599

   > hey [@Acfboy](https://github.com/Acfboy) if you're still interested to implement this please go ahead
   
   Thanks, I would like to give it a try.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3777757880

   @Acfboy Hi, thanks for asking! I have low bandwidth right now, so I can only start working on this next week. Thanks for your patience. šŸ™
   
   @milenkovicm sorry for letting you wait for so long


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3224135550

   I would like to give it a try~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

Posted by "Standing-Man (via GitHub)" <gi...@apache.org>.
Standing-Man commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3706962970

   untake


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3777683966

   hey @Acfboy if you're able to address the comments i'd be happy to review them


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]

Posted by "Standing-Man (via GitHub)" <gi...@apache.org>.
Standing-Man commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3127557742

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org