[GitHub] [beam] zeidoo commented on pull request #12748: [BEAM-8715] update avro - extract avro to it's own module [WIP]

[GitBox <gi...@apache.org>]

zeidoo commented on pull request #12748:
URL: https://github.com/apache/beam/pull/12748#issuecomment-690946531


   Per module breakdown of the high level changes.
   !!! = someone should probably confirm this is alright.
   
   !!! beam.sdks.java.extensions.avro: (New module)
   - moved from core.coders: AvroCoder, AvroGenericCoder
   - moved from core.io: AvroIO, AvroSchemaIOProvider, AvroSink, AvroSource, ConstantAvroDestination, DynamicAvroDestinations, GenericRecordWriteConverter, SerializableAvroCodecFactory
   - moved from core.schemas: AvroByteBuddyUtils, AvroUtils, AvroRecordSchema
   - moved from core.test: user.avsc, test.avsc, AvroCoderTest, AvroCoderTestPojo, AvroIOTest, AvroSourceTest, SerializableAvroCodecFactoryTest, AvroGenerators, AvroUtilsTest, AvroSchemaCoderTest, AvroSchemaTest
   
   beam:
   - settings.gradle -> include ref to avro module 
   
   beam.examples.java:
   - gradle dep on Avro module, needs Avro coder
   - only imports changed
   
   beam.examples.kotlin:
   - only imports (1 instance) changed
   
   beam.runners.core-construction-java:
   - gradle dep on Avro module, needs AvroGenericCoder and AvroCoder (for tests only)
   - !!! AvroGenericCoderRegistrar uses the ClassLoader to check if the AvroGenericCoder class can be loaded, otherwise DOES NOT try to load that class
   
   beam.runners.google-cloud-dataflow-java:
   - gradle dep on Avro module, needs AvroCoder and AvroReader
   - only imports changed
   
   beam.runners.spark:
   - NO gradle dep on Avro module, depends on beam.runners.google-cloud-dataflow-java which depends on Avro
   - only imports changed (in 1 test file, uses AvroIO)
   
   beam.sdks.java.core:
   - !!! DefaultCoderProvider is now public (previously private)
   - !!! BlockBasedReader is now public (previously was protected)
   - !!! BlockBasedReader.isAtSplitPoint() is now public (previously was protected)
   - !!! CounterMarkCoder introduced, previously was serialized with AvroCoder
   - !!! DefaultFilenamePolicy.constructName() is now public (previously private)
   - !!! FileIO.getEmptyMatchTreatment() and FileIO.getWatchInterval are now public (previously private)
   - !!! new MyValueACoder for CoderRegistryTest.MyValueA, previously using AvroCoder
   - !!! DefaultCoderTest Avro parts of these tests moved into the new Avro module
   - !!! TextIOWriteTest added a UserWriteTypeCoder for UserWriteType, previously using AvroCoder
   - !!! TextIOWriteTest.testDynamicDefaultFilenamePolicy, UserWriteTypeCoder now registered on the pipeline, previously this wasn't needed 
   - !!! SchemasTest Avro parts of these tests moved into the new Avro module
   - !!! LatestTest, replaced AvroCoder usages with StringUtf8Coder
   
   beam.sdks.java.extensions.sketching:
   - gradle dep on Avro module, needs AvroCoder for tests
   
   beam.sdk.extensions.sql:
   - gradle dep on Avro module, needs AvroIO and AvroSchemaIOProvider
   
   beam.sdks.java.io.file-based-io:
   - NO gradle dep on Avro module, indirect dep needed for AvroCoder and AvroIO
   
   beam.sdks.java.io.google-cloud-platform:
   - gradle dep on Avro module, needs AvroSource, AvroCoder and AvroUtils
   - copy from core.test: user.avsc
   
   beam.sdks.java.io.hadoop-format:
   - gradle dep on Avro module, needs AvroCoder for tests only
   
   beam.sdks.java.io.kafka:
   - gradle dep on Avro module, and added applyAvroNature(), needs AvroCoder
   - moved from core.test: user.avsc
   
   beam.sdks.java.io.parquet:
   - gradle dep on Avro module, needs AvroUtils and AvroCoder
   
   beam.sdks.java.io.snowflake:
   - gradle dep on Avro module, and added applyAvroNature(), needs AvroCoder for tests only
   - copy from core.test: user.avsc
   
   beam.sdks.java.testing.expansion-service:
   - gradle dep on Avro module, needs AvroCoder for tests only
   
   beam.sdks.java.testing.nexmark:
   - gradle dep on Avro module, needs AvroIO and AvroCoder


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org