You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/13 18:27:11 UTC

[GitHub] [beam] steveniemitz opened a new issue, #23213: [Feature Request]: AvroSource should use the standard CodecFactory to decompress blocks

steveniemitz opened a new issue, #23213:
URL: https://github.com/apache/beam/issues/23213

   ### What would you like to happen?
   
   Currently AvroSource has its own hand-rolled avro parser, given the class is over 6 years old I assume this is due to it being written before Avro had DataFileReader, etc, with good support for partially reading an avro container file.
   
   One of the downsides of this is that it doesn't share the same CodecFactory infrastructure as avro proper does, so users can't plug in custom CodecFactory instances using `CodecFactory.addCodec`.
   
   It should be possible to refactor AvroSource to use DataFileReader rather than its own handmade parser, and in doing so would automatically get the benefit of using the normal CodecFactory infrastructure.
   
   ### Issue Priority
   
   Priority: 3
   
   ### Issue Component
   
   Component: io-java-avro


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] steveniemitz commented on issue #23213: [Feature Request]: AvroSource should use the standard CodecFactory to decompress blocks

Posted by GitBox <gi...@apache.org>.
steveniemitz commented on issue #23213:
URL: https://github.com/apache/beam/issues/23213#issuecomment-1246074823

   Yeah that was what I figured, all the tests pass using DataFileReader so that's cool.  I'm going to run a bigger test with our own internal test suite to make sure it works in the real world too, but the test coverage seems pretty complete.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lukecwik commented on issue #23213: [Feature Request]: AvroSource should use the standard CodecFactory to decompress blocks

Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #23213:
URL: https://github.com/apache/beam/issues/23213#issuecomment-1246026933

   From what I vaguely remember, the existing APIs at the time weren't able to support the splitting protocol that was necessary. As a quick sanity check you could compare the DataFileReader interface at the time to the current one to see if there are new APIs related to seeking to specific offsets and knowing what offset we are at.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] iemejia closed issue #23213: [Feature Request]: AvroSource should use the standard CodecFactory to decompress blocks

Posted by GitBox <gi...@apache.org>.
iemejia closed issue #23213: [Feature Request]: AvroSource should use the standard CodecFactory to decompress blocks
URL: https://github.com/apache/beam/issues/23213


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org