You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/21 15:16:30 UTC
[GitHub] [beam] steveniemitz opened a new pull request #11479: [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
steveniemitz opened a new pull request #11479:
URL: https://github.com/apache/beam/pull/11479
There are cases where we'd like to further control the user record -> avro process. For example, I might not want to write generic records, but instead use my own DatumWriter instance to do the avro writing.
This PR simply allows further customizing the AvroRowWriter that is created during pipeline construction, allowing the user to hook into the DatumWriter construction.
R: @pabloem @chamikaramj
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz edited a comment on pull request #11479: [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
Posted by GitBox <gi...@apache.org>.
steveniemitz edited a comment on pull request #11479:
URL: https://github.com/apache/beam/pull/11479#issuecomment-622067456
thanks! I'm going to have a similar PR for AvroIO coming up, do you know who's best to review that?
edit: hmm looks like this pre-commit broke again w/ the license error, do I need to rebase this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11479: [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11479:
URL: https://github.com/apache/beam/pull/11479#issuecomment-622084154
since it's only that task that failed, we can merge anyway.
For AvroIO - I think @iemejia may be a good candidate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on issue #11479: [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on issue #11479:
URL: https://github.com/apache/beam/pull/11479#issuecomment-618074012
thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on issue #11479: [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11479:
URL: https://github.com/apache/beam/pull/11479#issuecomment-618067907
Hi Steve! thanks for the feature. I'll take a look hopefully by friday
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11479: [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11479:
URL: https://github.com/apache/beam/pull/11479#issuecomment-622031167
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #11479: [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #11479:
URL: https://github.com/apache/beam/pull/11479#issuecomment-622067456
thanks! I'm going to have a similar PR for AvroIO coming up, do you know who's best to review that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11479: [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11479:
URL: https://github.com/apache/beam/pull/11479#issuecomment-622031007
ah yes sorry Steve. Thanks for the contribution. This looks good to me. I'll just rerun precomits to get a green run, and run postcommits as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on issue #11479: [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on issue #11479:
URL: https://github.com/apache/beam/pull/11479#issuecomment-617292747
hm, looks like the precommit is failing with:
```
13:01:36 FAILURE: Build failed with an exception.
13:01:36
13:01:36 * What went wrong:
13:01:36 Execution failed for task ':sdks:java:container:generateThirdPartyLicenses'.
13:01:36 > Process 'command './sdks/java/container/license_scripts/license_script.sh'' finished with non-zero exit value 1
```
which hopefully my change didn't break ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #11479: [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #11479:
URL: https://github.com/apache/beam/pull/11479#issuecomment-621854433
bump?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11479: [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11479:
URL: https://github.com/apache/beam/pull/11479#issuecomment-622031083
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11479: [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11479:
URL: https://github.com/apache/beam/pull/11479#issuecomment-622087181
thanks @steveniemitz !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org