You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jayshrivastava (via GitHub)" <gi...@apache.org> on 2026/08/13 15:52:46 UTC
[I] ComposedPhysicalExtensionCodec should not rely on positional encoding [datafusion]
jayshrivastava opened a new issue, #24331:
URL: https://github.com/apache/datafusion/issues/24331
### Is your feature request related to a problem or challenge?
ComposedPhysicalExtensionCodec should identify codecs by stable ID instead of list position
`ComposedPhysicalExtensionCodec` records the position of each codec in the `DataEncoderTuple`:
```rust
struct DataEncoderTuple {
pub encoder_position: u32,
pub blob: Vec<u8>,
}
The problem is that if someone adds, removes, or reorders a codec, you get really confusing error messages. For example:
1. A writer with [CodecA, CodecB] encodes protobuf `A` and specifies `encoder_position: 0`
2. A reader with [CodecB, CodecA] decodes the proto using `encoder_position: 0` and errors with "CodecB cannot decode message"
```
It's hard to see that this was caused a breaking protocol change. Since both codecs are present, it doesn't seem like a breaking change, but it is.
### Describe the solution you'd like
Maybe we can introduce an id to look up encoders:
```rust
struct DataEncoderTuple {
// Retained for decoding payloads written by older versions.
pub encoder_position: u32,
pub blob: Vec<u8>,
pub codec_id: Option<String>,
}
Decoding would:
1. Use codec_id when present.
2. Return an explicit error containing the unknown identifier when it is not registered.
3. Fall back to encoder_position for legacy payloads.
```
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ComposedPhysicalExtensionCodec should not rely on positional encoding [datafusion]
Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on issue #24331:
URL: https://github.com/apache/datafusion/issues/24331#issuecomment-5598049733
Was just coincidentally drafting something about this here: https://github.com/apache/datafusion/pull/24826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ComposedPhysicalExtensionCodec should not rely on positional encoding [datafusion]
Posted by "imtherealnaska (via GitHub)" <gi...@apache.org>.
imtherealnaska commented on issue #24331:
URL: https://github.com/apache/datafusion/issues/24331#issuecomment-5311219658
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ComposedPhysicalExtensionCodec should not rely on positional encoding [datafusion]
Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on issue #24331:
URL: https://github.com/apache/datafusion/issues/24331#issuecomment-5316406954
@imtherealnaska Because this is a significant change, it may be good to get a committer's opinion on if it's a good idea first :) cc @adriangb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ComposedPhysicalExtensionCodec should not rely on positional encoding [datafusion]
Posted by "imtherealnaska (via GitHub)" <gi...@apache.org>.
imtherealnaska commented on issue #24331:
URL: https://github.com/apache/datafusion/issues/24331#issuecomment-5316605639
yeah true 👍🏼
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org