You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "milenkovicm (via GitHub)" <gi...@apache.org> on 2025/07/28 12:26:45 UTC
[I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
milenkovicm opened a new issue, #16944:
URL: https://github.com/apache/datafusion/issues/16944
### Is your feature request related to a problem or challenge?
At the moment ballista overrides `LogicalExtensionCodec::try_decode_file_format` & `LogicalExtensionCodec::try_encode_file_format` providing support for:
```rust
file_format_codecs: vec![
Arc::new(ParquetLogicalExtensionCodec {}),
Arc::new(CsvLogicalExtensionCodec {}),
Arc::new(JsonLogicalExtensionCodec {}),
Arc::new(ArrowLogicalExtensionCodec {}),
Arc::new(AvroLogicalExtensionCodec {}),
],
```
as seen at [1]. Should we want to integrate ballista with datafusion python we would need to provide a custom `LogicalExtensionCodec` implementing same logic or reusing ballista `LogicalExtensionCodec` implementation. As this file types are supported out of the box in datafusion would it make sense to implement encoder/decoder or them in `DefaultLogicalExtensionCodec`?
[1]: https://github.com/milenkovicm/arrow-ballista/blob/e1e9f6ca423fd558664a7f2fb3b1bc3ed07d7db8/ballista/core/src/serde/mod.rs#L164-L165
### Describe the solution you'd like
Should this proposal make sense, implement support for it in `DefaultLogicalExtensionCodec` similar to what is supported in ballista already:
```rust
fn try_decode_file_format(
&self,
buf: &[u8],
ctx: &datafusion::prelude::SessionContext,
) -> Result<Arc<dyn datafusion::datasource::file_format::FileFormatFactory>> {
let proto = FileFormatProto::decode(buf)
.map_err(|e| DataFusionError::Internal(e.to_string()))?;
let codec = self
.file_format_codecs
.get(proto.encoder_position as usize)
.ok_or(DataFusionError::Internal(
"Can't find required codec in file codec list".to_owned(),
))?;
codec.try_decode_file_format(&proto.blob, ctx)
}
fn try_encode_file_format(
&self,
buf: &mut Vec<u8>,
node: Arc<dyn datafusion::datasource::file_format::FileFormatFactory>,
) -> Result<()> {
let mut blob = vec![];
let (encoder_position, _) =
self.try_any(|codec| codec.try_encode_file_format(&mut blob, node.clone()))?;
let proto = FileFormatProto {
encoder_position,
blob,
};
proto
.encode(buf)
.map_err(|e| DataFusionError::Internal(e.to_string()))
}
```
https://github.com/milenkovicm/arrow-ballista/blob/e1e9f6ca423fd558664a7f2fb3b1bc3ed07d7db8/ballista/core/src/serde/mod.rs#L214-L215
### Describe alternatives you've considered
alternative would be to keep everything as it is.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
Posted by "jizezhang (via GitHub)" <gi...@apache.org>.
jizezhang commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3508846189
Hi, if no one is actively working on this task, I am interested. To check my understanding, it seems like the goal is to move some functionality from ballista codec to `datafusion_proto::logical_plan::DefaultLogicalExtensionCodec`? Specifically,
- add a `default` method for `DefaultLogicalExtensionCodec` that populates a list of file format codecs under `datafusion_proto::logical_plan::file_formats` module
- add a `FileFormatProto` struct as in ballista
- implement `try_decode_file_format` and `try_encode_file_format` based on ballista implementation?
If those sound correct, I can work on getting a PR. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm closed issue #16944: make DefaultLogicalExtensionCodec support serialisation of build in file formats
URL: https://github.com/apache/datafusion/issues/16944
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3224141088
/take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
Posted by "Acfboy (via GitHub)" <gi...@apache.org>.
Acfboy commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3763610202
Hi @peterxcli, are you still working on this? I notice relevant pr has some conflicts and pending review comments. Since there hasn't been activity for a while, if you're busy, I'd be happy to take over the PR and drive it to completion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3431276831
any progress on this one @peterxcli
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3778092702
No problem @peterxcli, Iām not in a hurry at all. This task was opened to give new contributors a chance to get involved with datafusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3224141624
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
Posted by "Acfboy (via GitHub)" <gi...@apache.org>.
Acfboy commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3977028599
> hey [@Acfboy](https://github.com/Acfboy) if you're still interested to implement this please go ahead
Thanks, I would like to give it a try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3777757880
@Acfboy Hi, thanks for asking! I have low bandwidth right now, so I can only start working on this next week. Thanks for your patience. š
@milenkovicm sorry for letting you wait for so long
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3224135550
I would like to give it a try~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
Posted by "Standing-Man (via GitHub)" <gi...@apache.org>.
Standing-Man commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3706962970
untake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3777683966
hey @Acfboy if you're able to address the comments i'd be happy to review them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make DefaultLogicalExtensionCodec support serialisation of build in file formats [datafusion]
Posted by "Standing-Man (via GitHub)" <gi...@apache.org>.
Standing-Man commented on issue #16944:
URL: https://github.com/apache/datafusion/issues/16944#issuecomment-3127557742
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org