You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by echauchot <gi...@git.apache.org> on 2017/10/05 13:51:30 UTC
[GitHub] beam pull request #3950: [BEAM-2993] AvroIO.write without specifying a schem...
GitHub user echauchot opened a pull request:
https://github.com/apache/beam/pull/3950
[BEAM-2993] AvroIO.write without specifying a schema
Follow this checklist to help us incorporate your contribution quickly and easily:
- [X] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
- [X] Each commit in the pull request should have a meaningful subject line and body.
- [X] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue.
- [X] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
- [X] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
- [X] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
---
This PR adds the ability to use `AvroIO.write()` and related methods without specifying a schema.
The schema is determined at the first call of `AvroSink.write()`: the `DataFileWriter` is lazy initialized (at first write) once we have the value to get the schema from.
This PR also makes the schema optional in `ConstantAvroDestination` and depreciate write methods that take schema as parameter. Tell me if I'm missing something that prevents deprecation of these methods.
To use `AvoIO.write()` with no schema, all the elements of the input PCollection must have the same schema, but it is the same with current AvroIO.write(schema) implementation because this schema instance is passed to the `TypedWrite` then to the `ConstantAvroDestination` that is used in `AvroSink`. Please tell me if I'm missing something here.
My only concern is with empty bundles, `AvroSink.write()` will not be called resulting in the `DataFileWriter` not being initialized.
Please merge the PR bellow before this one because it is used as a base for the tests
https://github.com/apache/beam/pull/3948
R: @jkff
R: @reuvenlax
CC: @lukecwik
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/echauchot/beam AvroIOWriteSchemaLess2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3950.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3950
----
commit 43ef4d42d7d224b1997278832ec645bccb945792
Author: Etienne Chauchot <ec...@gmail.com>
Date: 2017-10-05T09:45:12Z
[BEAM-3019] Make AvroIOWriteTransformTest more generic
make runTestWrite() more generic to be able to use GenericRecord[] as input for writeGenericRecords test in place of AvroGeneratedUser
make readAvroFile() generic to be able to read GenericRecords using GenericDatumReader for writeGenericRecords test
commit 84074e36085d76f569c89d4a29a647fc40b22531
Author: Etienne Chauchot <ec...@gmail.com>
Date: 2017-10-02T15:08:55Z
[BEAM-2993] AvroIO.write without specifying a schema
Lazy init (at first write) of the dataFileWriter once we have the value to get the schema from.
Make schema optional in ConstantAvroDestination and depreciate write methods that take schema as parameter
Cleaning
commit d19c2cb3538e5981e8138522d0c2138b455dec46
Author: Etienne Chauchot <ec...@gmail.com>
Date: 2017-10-05T12:08:04Z
Add tests of the schema less write methods
Cleaning
commit da95342353bd191c55d6d7768d4c052c531b8cf1
Author: Etienne Chauchot <ec...@gmail.com>
Date: 2017-10-05T12:43:29Z
Fixups
----
---
[GitHub] beam pull request #3950: [BEAM-2993] AvroIO.write without specifying a schem...
Posted by echauchot <gi...@git.apache.org>.
Github user echauchot closed the pull request at:
https://github.com/apache/beam/pull/3950
---