You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/08/17 12:56:01 UTC
[GitHub] [beam] steveniemitz opened a new pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
steveniemitz opened a new pull request #15338:
URL: https://github.com/apache/beam/pull/15338
This adds two improvements to the PipelineOptionFactory command line parser:
- Failure to parse complex types due to missing quotes is retried with the input automatically quoted
- @JsonDeserialize is now supported on getters, the specified deserializer in the annotation will be used to parse the input string.
R: @lukecwik @kennknowles since you two are the last to make any changes here.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-909399531
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-902989250
> @lukecwik should be good to review now.
I'm away camping, will take a look when I'm back on Wednesday next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-902304706
Run Java_Examples_Dataflow PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2ab0024) into [master](https://codecov.io/gh/apache/beam/commit/af59192ac95a564ca3d29b6385a6d1448f5a407d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (af59192) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2ab0024 differs from pull request most recent head 13807ab. Consider uploading reports for the commit 13807ab to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
- Coverage 83.81% 83.81% -0.01%
==========================================
Files 441 441
Lines 59773 59773
==========================================
- Hits 50101 50096 -5
- Misses 9672 9677 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.47% <0.00%> (-0.78%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.44% <0.00%> (-0.35%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.85% <0.00%> (-0.16%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.73% <0.00%> (-0.15%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...13807ab](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690640631
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
Thanks, now I understand but Jackson does seem to ignore that when the object type being converted to is a String since on https://github.com/apache/beam/blob/2ab002486345f5d97f11ccfbeb86541796c080e0/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java#L1676 it seems as to allow the conversion and there are a few tests in PipelineOptionsFactoryTest that pass even without the quoting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-908601444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.03%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head f9b9959. Consider uploading reports for the commit f9b9959 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.83% +0.03%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50207 +92
+ Misses 9689 9679 -10
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [18 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...f9b9959](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-909506552
Thanks, if you have some time and can look at why the test is flaking it would be greatly appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-908782473
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r697707454
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1058,13 +1086,29 @@ private static void validateMethodAnnotations(
validateGettersHaveConsistentAnnotation(
methodNameToAllMethodMap, descriptors, AnnotationPredicates.DEFAULT_VALUE);
+ // Verify that there is no getter with a mixed @JsonDeserialize annotation.
+ validateGettersHaveConsistentAnnotation(
Review comment:
Does it make sense that if `@JsonDeserialize` is on the getter then we must also have `@JsonSerialize` and vice versa?
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1791,6 +1984,11 @@ private static void checkEmptyStringAllowed(
private final Map<Set<Class<? extends PipelineOptions>>, Registration<?>> combinedCache =
Maps.newConcurrentMap();
+ private final Map<Method, JsonDeserializer<Object>> deserializerCache = Maps.newConcurrentMap();
Review comment:
We should use `Optional` here as well to prevent recomputation all the time since computeIfAbsent will always compute again and again when null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kw2542 commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
kw2542 commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910496463
I am not 100% sure if this is the RC but looks like it.
```
java.lang.RuntimeException: Failed to invoke samza job
at org.apache.beam.runners.samza.SamzaPipelineRunner.run(SamzaPipelineRunner.java:83)
at org.apache.beam.runners.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:86)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Failed to serialize PipelineOptions
at org.apache.beam.runners.samza.translation.ConfigBuilder.build(ConfigBuilder.java:110)
at org.apache.beam.runners.samza.SamzaRunner.runPortablePipeline(SamzaRunner.java:93)
at org.apache.beam.runners.samza.SamzaPipelineRunner.run(SamzaPipelineRunner.java:73)
... 7 more
Caused by: java.lang.IllegalArgumentException: Failed to serialize PipelineOptions
at org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:70)
at org.apache.beam.runners.core.construction.SerializablePipelineOptions.<init>(SerializablePipelineOptions.java:44)
at org.apache.beam.runners.samza.translation.ConfigBuilder.build(ConfigBuilder.java:102)
... 9 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Error while populating display data for component 'org.apache.beam.sdk.options.ProxyInvocationHandler$PipelineOptionsDisplayData': null
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._wrapAsIOE(DefaultSerializerProvider.java:509)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:482)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4487)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3742)
at org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:68)
... 11 more
Caused by: org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder$PopulateDisplayDataException: Error while populating display data for component 'org.apache.beam.sdk.options.ProxyInvocationHandler$PipelineOptionsDisplayData': null
at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:784)
at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.delegate(DisplayData.java:740)
at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:157)
at com.sun.proxy.$Proxy0.populateDisplayData(Unknown Source)
at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:775)
at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.access$100(DisplayData.java:703)
at org.apache.beam.sdk.transforms.display.DisplayData.from(DisplayData.java:81)
at org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:706)
at org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:654)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
... 15 more
Caused by: java.lang.NullPointerException
at com.fasterxml.jackson.databind.DeserializationContext.isEnabled(DeserializationContext.java:395)
at com.fasterxml.jackson.databind.deser.std.StdDeserializer._checkFromStringCoercion(StdDeserializer.java:1364)
at com.fasterxml.jackson.databind.deser.std.StdDeserializer._parseIntPrimitive(StdDeserializer.java:728)
at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$IntegerDeserializer.deserialize(NumberDeserializers.java:529)
at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$IntegerDeserializer.deserialize(NumberDeserializers.java:506)
at org.apache.beam.sdk.options.PipelineOptionsFactory.deserializeNode(PipelineOptionsFactory.java:1791)
at org.apache.beam.sdk.options.ProxyInvocationHandler.getValueFromJson(ProxyInvocationHandler.java:517)
at org.apache.beam.sdk.options.ProxyInvocationHandler.getValueFromJson(ProxyInvocationHandler.java:512)
at org.apache.beam.sdk.options.ProxyInvocationHandler.access$400(ProxyInvocationHandler.java:93)
at org.apache.beam.sdk.options.ProxyInvocationHandler$PipelineOptionsDisplayData.populateDisplayData(ProxyInvocationHandler.java:359)
at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:775)
... 24 more
ERROR:root:java.lang.NullPointerException
```
Here is the stacktrace that failed these tests, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kw2542 edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
kw2542 edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910489301
@steveniemitz @lukecwik This commit seems to be causing bunch of post commit tests to failure:
https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/172/
https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/1347/
https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/5024/
Should we revert this PR for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-909464429
@lukecwik finally jammed all the tests through!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-908805853
Run Java_Examples_Dataflow PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910533042
roll-forward PR up as well: #15439
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-908601444
> np! Hope you had fun camping. I agree about requiring both attributes if either is present, I was toying around with the same thing on our internal branch too. I'll address these in the next day or so.
Sounds great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head 87f2c72. Consider uploading reports for the commit 87f2c72 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.80% +0.01%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50190 +75
- Misses 9689 9696 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [12 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...87f2c72](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690488023
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
+ } catch (JsonParseException inner) {
+ // rethrow the original exception rather the one thrown from the fallback attempt
+ throw e;
+ }
+ } else {
+ throw e;
+ }
+ }
+
+ JsonParser parser = new TreeTraversingParser(tree, MAPPER);
+ parser.nextToken();
+ return DESERIALIZATION_CONTEXT.readPropertyValue(parser, prop, type);
Review comment:
As a point of conversation, I'm curious what people think of using the built-in jackson `@JsonDeserialize` vs making our own attribute? The downside I see of the built-in attribute is that other things may parse pipeline options (dataflow for example writes it out as json), and this will affect that deserialization as well.
We could have a `@ParseFunction` attribute or similar to map String -> T instead possibly, which would only be used from PipelineOptionsFactory.
Curious what people's thoughts are.
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
jackson expects the input to be well formed json, which means if it's a string it needs to be double quoted. You can see it in a test just by removing these catch block and running the test, it'll fail with something like:
```com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'test': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (String)"test"; line: 1, column: 5]
```
The deserialization is a different problem (which is addressed by the JsonDeserialize attribute support).
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
+ } catch (JsonParseException inner) {
+ // rethrow the original exception rather the one thrown from the fallback attempt
+ throw e;
+ }
+ } else {
+ throw e;
+ }
+ }
+
+ JsonParser parser = new TreeTraversingParser(tree, MAPPER);
+ parser.nextToken();
+ return DESERIALIZATION_CONTEXT.readPropertyValue(parser, prop, type);
Review comment:
I'll double check, but I think it should already be supported. The places I've seen pass the options into the mapper directly, which will use the annotations. We only need this workaround here since the PipelineOptions class isn't directly deserialized.
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
+ } catch (JsonParseException inner) {
+ // rethrow the original exception rather the one thrown from the fallback attempt
+ throw e;
+ }
+ } else {
+ throw e;
+ }
+ }
+
+ JsonParser parser = new TreeTraversingParser(tree, MAPPER);
+ parser.nextToken();
+ return DESERIALIZATION_CONTEXT.readPropertyValue(parser, prop, type);
Review comment:
> But be aware that you'll want to ensure that all instances used via the "as" transform use the same annotation
ack, I had saw this but didn't implement it yet pending agreement on using the jackson annotation. I'll add it in.
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
jackson expects the input to be well formed json, which means if it's a string it needs to be double quoted. You can see it in a test just by removing these catch block and running the test, it'll fail with something like:
```
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'test': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (String)"test"; line: 1, column: 5]
```
The deserialization is a different problem (which is addressed by the JsonDeserialize attribute support).
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
ah yeah `convertValue` vs `readValue` use different code paths. afaik convert will serialize `from` to json first, then read it back, which is slightly different than the complex object case here, since the input string potentially is an actual json value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2ab0024) into [master](https://codecov.io/gh/apache/beam/commit/af59192ac95a564ca3d29b6385a6d1448f5a407d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (af59192) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2ab0024 differs from pull request most recent head 400c6ec. Consider uploading reports for the commit 400c6ec to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
- Coverage 83.81% 83.81% -0.01%
==========================================
Files 441 441
Lines 59773 59773
==========================================
- Hits 50101 50096 -5
- Misses 9672 9677 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.47% <0.00%> (-0.78%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.44% <0.00%> (-0.35%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.85% <0.00%> (-0.16%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.73% <0.00%> (-0.15%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...400c6ec](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-901428245
> I don't see any logic around `@JsonSerialize/@JsonIgnore` as we discussed.
I'm not sure what I'd be testing/adding logic for here? The places that serialize PipelineOptions I've seen pass the typed PipelineOptions object to an ObjectMapper directly, is there some other path you were thinking about?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910528005
> and run the additional failing tests in the forward fix.
Forgive the noob question, is there a way to run these post-commit tests on a PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690614446
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
+ } catch (JsonParseException inner) {
+ // rethrow the original exception rather the one thrown from the fallback attempt
+ throw e;
+ }
+ } else {
+ throw e;
+ }
+ }
+
+ JsonParser parser = new TreeTraversingParser(tree, MAPPER);
+ parser.nextToken();
+ return DESERIALIZATION_CONTEXT.readPropertyValue(parser, prop, type);
Review comment:
This sounds like a useful improvement but we'll want `@JsonSerialize` as well since we need the value to round trip to JSON and back (unless the getter is marked with `@JsonIgnore` in that case we don't need the `@JsonSerialize`).
But be aware that you'll want to ensure that all instances used via the "as" transform use the same annotation since multiple PipelineOptions can define the same property effectively and we would want to limit how inconsistent conversions can appear so this will limit the feature a little.
For example with `@JsonIgnore` https://github.com/apache/beam/blob/2ab002486345f5d97f11ccfbeb86541796c080e0/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java#L941
https://github.com/apache/beam/blob/2ab002486345f5d97f11ccfbeb86541796c080e0/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java#L1054
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
I saw the example in the test but am lost as two why converting `--myInstant=2021-07-17` actually fails. Would you be able to add an example of the failure to the tests?
Also, is this an issue with the type deserialize not being able to handle JSON string -> object conversion?
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
I saw the example in the JIRA but am lost as two why converting `--myInstant=2021-07-17` actually fails. Would you be able to add an example of the failure to the tests?
Also, is this an issue with the type deserialize not being able to handle JSON string -> object conversion?
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
Thanks, now I understand but Jackson does seem to ignore that when the object type being converted to is a String since on https://github.com/apache/beam/blob/2ab002486345f5d97f11ccfbeb86541796c080e0/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java#L1676 it seems as to allow the conversion and there are a few tests in PipelineOptionsFactoryTest that pass even without the quoting.
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
Thanks, now I understand but Jackson does seem to ignore that when the object type being converted to is a String since on https://github.com/apache/beam/blob/2ab002486345f5d97f11ccfbeb86541796c080e0/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java#L1676 it seems as to allow the conversion and there are a few tests in PipelineOptionsFactoryTest that pass even without the quoting.
I think adding the quoting makes sense unless there is a way to fix JsonDeserializer types to support unquoted complex objects.
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
Good point, sounds good to add the automatic quoting feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.03%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head 87f2c72. Consider uploading reports for the commit 87f2c72 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.83% +0.03%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50207 +92
+ Misses 9689 9679 -10
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [18 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...87f2c72](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2ab0024) into [master](https://codecov.io/gh/apache/beam/commit/af59192ac95a564ca3d29b6385a6d1448f5a407d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (af59192) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2ab0024 differs from pull request most recent head bd17b09. Consider uploading reports for the commit bd17b09 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
- Coverage 83.81% 83.81% -0.01%
==========================================
Files 441 441
Lines 59773 59773
==========================================
- Hits 50101 50096 -5
- Misses 9672 9677 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.47% <0.00%> (-0.78%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.44% <0.00%> (-0.35%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.85% <0.00%> (-0.16%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.73% <0.00%> (-0.15%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...bd17b09](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2ab0024) into [master](https://codecov.io/gh/apache/beam/commit/af59192ac95a564ca3d29b6385a6d1448f5a407d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (af59192) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2ab0024 differs from pull request most recent head 62d2909. Consider uploading reports for the commit 62d2909 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
- Coverage 83.81% 83.81% -0.01%
==========================================
Files 441 441
Lines 59773 59773
==========================================
- Hits 50101 50096 -5
- Misses 9672 9677 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.47% <0.00%> (-0.78%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.44% <0.00%> (-0.35%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.85% <0.00%> (-0.16%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.73% <0.00%> (-0.15%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...62d2909](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-908694619
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head 87f2c72. Consider uploading reports for the commit 87f2c72 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.82% +0.02%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50197 +82
Misses 9689 9689
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [16 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...87f2c72](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910503342
I think the patch is actually really simple, I can get another PR going right now, or we can revert and re-apply with it fixed. I'm fine with either.
```
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
index 030c56ba59..e20a8641b6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
@@ -44,6 +44,7 @@ import com.fasterxml.jackson.databind.node.TreeTraversingParser;
import com.fasterxml.jackson.databind.ser.DefaultSerializerProvider;
import com.fasterxml.jackson.databind.type.TypeBindings;
import com.fasterxml.jackson.databind.util.SimpleBeanPropertyDefinition;
+import com.fasterxml.jackson.databind.util.TokenBuffer;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.beans.BeanInfo;
import java.beans.IntrospectionException;
@@ -499,7 +500,8 @@ public class PipelineOptionsFactory {
private static final DefaultDeserializationContext DESERIALIZATION_CONTEXT =
new DefaultDeserializationContext.Impl(MAPPER.getDeserializationContext().getFactory())
- .createInstance(MAPPER.getDeserializationConfig(), null, null);
+ .createInstance(
+ MAPPER.getDeserializationConfig(), new TokenBuffer(MAPPER, false).asParser(), null);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.03%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head 87f2c72. Consider uploading reports for the commit 87f2c72 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.83% +0.03%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50206 +91
+ Misses 9689 9680 -9
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [18 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...87f2c72](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690613345
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
I saw the example in the JIRA but am lost as two why converting `--myInstant=2021-07-17` actually fails. Would you be able to add an example of the failure to the tests?
Also, is this an issue with the type deserialize not being able to handle JSON string -> object conversion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-901337967
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-908810040
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r698722243
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1791,6 +1984,11 @@ private static void checkEmptyStringAllowed(
private final Map<Set<Class<? extends PipelineOptions>>, Registration<?>> combinedCache =
Maps.newConcurrentMap();
+ private final Map<Method, JsonDeserializer<Object>> deserializerCache = Maps.newConcurrentMap();
Review comment:
`computeDeserializerForMethod` should never return null here (it returns the default if there isn't one set)
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1058,13 +1086,29 @@ private static void validateMethodAnnotations(
validateGettersHaveConsistentAnnotation(
methodNameToAllMethodMap, descriptors, AnnotationPredicates.DEFAULT_VALUE);
+ // Verify that there is no getter with a mixed @JsonDeserialize annotation.
+ validateGettersHaveConsistentAnnotation(
Review comment:
Docs are updated, lemme know what you think of them, happy to improve/reword if you have suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kw2542 edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
kw2542 edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910496463
I am not 100% sure if this is the RC but looks like it.
```
java.lang.RuntimeException: Failed to invoke samza job
at org.apache.beam.runners.samza.SamzaPipelineRunner.run(SamzaPipelineRunner.java:83)
at org.apache.beam.runners.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:86)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Failed to serialize PipelineOptions
at org.apache.beam.runners.samza.translation.ConfigBuilder.build(ConfigBuilder.java:110)
at org.apache.beam.runners.samza.SamzaRunner.runPortablePipeline(SamzaRunner.java:93)
at org.apache.beam.runners.samza.SamzaPipelineRunner.run(SamzaPipelineRunner.java:73)
... 7 more
Caused by: java.lang.IllegalArgumentException: Failed to serialize PipelineOptions
at org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:70)
at org.apache.beam.runners.core.construction.SerializablePipelineOptions.<init>(SerializablePipelineOptions.java:44)
at org.apache.beam.runners.samza.translation.ConfigBuilder.build(ConfigBuilder.java:102)
... 9 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Error while populating display data for component 'org.apache.beam.sdk.options.ProxyInvocationHandler$PipelineOptionsDisplayData': null
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._wrapAsIOE(DefaultSerializerProvider.java:509)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:482)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4487)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3742)
at org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:68)
... 11 more
Caused by: org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder$PopulateDisplayDataException: Error while populating display data for component 'org.apache.beam.sdk.options.ProxyInvocationHandler$PipelineOptionsDisplayData': null
at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:784)
at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.delegate(DisplayData.java:740)
at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:157)
at com.sun.proxy.$Proxy0.populateDisplayData(Unknown Source)
at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:775)
at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.access$100(DisplayData.java:703)
at org.apache.beam.sdk.transforms.display.DisplayData.from(DisplayData.java:81)
at org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:706)
at org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:654)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
... 15 more
Caused by: java.lang.NullPointerException
at com.fasterxml.jackson.databind.DeserializationContext.isEnabled(DeserializationContext.java:395)
at com.fasterxml.jackson.databind.deser.std.StdDeserializer._checkFromStringCoercion(StdDeserializer.java:1364)
at com.fasterxml.jackson.databind.deser.std.StdDeserializer._parseIntPrimitive(StdDeserializer.java:728)
at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$IntegerDeserializer.deserialize(NumberDeserializers.java:529)
at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$IntegerDeserializer.deserialize(NumberDeserializers.java:506)
at org.apache.beam.sdk.options.PipelineOptionsFactory.deserializeNode(PipelineOptionsFactory.java:1791)
at org.apache.beam.sdk.options.ProxyInvocationHandler.getValueFromJson(ProxyInvocationHandler.java:517)
at org.apache.beam.sdk.options.ProxyInvocationHandler.getValueFromJson(ProxyInvocationHandler.java:512)
at org.apache.beam.sdk.options.ProxyInvocationHandler.access$400(ProxyInvocationHandler.java:93)
at org.apache.beam.sdk.options.ProxyInvocationHandler$PipelineOptionsDisplayData.populateDisplayData(ProxyInvocationHandler.java:359)
at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:775)
... 24 more
ERROR:root:java.lang.NullPointerException
```
This is the stacktrace in Samza runner that failed these tests, where a similar stacktrace in Flink and Sparker Runner, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kw2542 commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
kw2542 commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910604528
> looks like the failing tests are fixed in #15439 now, so we can either roll forward with that or roll back w/ the revert. I'd prefer to roll forward but I'll leave that up to you :)
+1, let's fix forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-908810201
Run Java_Examples_Dataflow_Java11 PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-902421970
Run Java_Examples_Dataflow PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690696582
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
Good point, sounds good to add the automatic quoting feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head 87f2c72. Consider uploading reports for the commit 87f2c72 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.82% +0.02%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50200 +85
+ Misses 9689 9686 -3
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [17 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...87f2c72](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kw2542 commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
kw2542 commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910528854
> > and run the additional failing tests in the forward fix.
>
> Forgive the noob question, is there a way to run these post-commit tests on a PR?
I am testing out the rollback at https://github.com/apache/beam/pull/15438 where you can see how these tests are being triggered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-902315338
~~I'm still tracking down one more issue w/ serialization now when serializing custom ValueProviders~~
edit: nm there wasn't an issue, I was just using an out-of-date build on my tests =/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2ab0024) into [master](https://codecov.io/gh/apache/beam/commit/af59192ac95a564ca3d29b6385a6d1448f5a407d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (af59192) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2ab0024 differs from pull request most recent head 183b6d8. Consider uploading reports for the commit 183b6d8 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
- Coverage 83.81% 83.81% -0.01%
==========================================
Files 441 441
Lines 59773 59773
==========================================
- Hits 50101 50096 -5
- Misses 9672 9677 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.47% <0.00%> (-0.78%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.44% <0.00%> (-0.35%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.85% <0.00%> (-0.16%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.73% <0.00%> (-0.15%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...183b6d8](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-902131179
> > I don't see any logic around `@JsonSerialize/@JsonIgnore` as we discussed.
>
> ~I'm not sure what I'd be testing/adding logic for here? The places that serialize PipelineOptions I've seen pass the typed PipelineOptions object to an ObjectMapper directly, is there some other path you were thinking about?~
>
> never mind I see what you mean now, I didn't realize the PipelineOptions had a custom serializer/deserializer as well. Will update that.
Great, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690488023
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
+ } catch (JsonParseException inner) {
+ // rethrow the original exception rather the one thrown from the fallback attempt
+ throw e;
+ }
+ } else {
+ throw e;
+ }
+ }
+
+ JsonParser parser = new TreeTraversingParser(tree, MAPPER);
+ parser.nextToken();
+ return DESERIALIZATION_CONTEXT.readPropertyValue(parser, prop, type);
Review comment:
As a point of conversation, I'm curious what people think of using the built-in jackson `@JsonDeserialize` vs making our own attribute? The downside I see of the built-in attribute is that other things may parse pipeline options (dataflow for example writes it out as json), and this will affect that deserialization as well.
We could have a `@ParseFunction` attribute or similar to map String -> T instead possibly, which would only be used from PipelineOptionsFactory.
Curious what people's thoughts are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690640631
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
Thanks, now I understand but Jackson does seem to ignore that when the object type being converted to is a String since on https://github.com/apache/beam/blob/2ab002486345f5d97f11ccfbeb86541796c080e0/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java#L1676 it seems as to allow the conversion and there are a few tests in PipelineOptionsFactoryTest that pass even without the quoting.
I think adding the quoting makes sense unless there is a way to fix JsonDeserializer types to support unquoted complex objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2ab0024) into [master](https://codecov.io/gh/apache/beam/commit/af59192ac95a564ca3d29b6385a6d1448f5a407d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (af59192) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2ab0024 differs from pull request most recent head 9e0c248. Consider uploading reports for the commit 9e0c248 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
- Coverage 83.81% 83.81% -0.01%
==========================================
Files 441 441
Lines 59773 59773
==========================================
- Hits 50101 50096 -5
- Misses 9672 9677 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.47% <0.00%> (-0.78%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.44% <0.00%> (-0.35%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.85% <0.00%> (-0.16%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.73% <0.00%> (-0.15%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...9e0c248](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-902810421
@lukecwik should be good to review now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-902304308
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910524439
> I think the patch is actually really simple, I can get another PR going right now, or we can revert and re-apply with it fixed. I'm fine with either.
>
> ```
> diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
> index 030c56ba59..e20a8641b6 100644
> --- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
> +++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
> @@ -44,6 +44,7 @@ import com.fasterxml.jackson.databind.node.TreeTraversingParser;
> import com.fasterxml.jackson.databind.ser.DefaultSerializerProvider;
> import com.fasterxml.jackson.databind.type.TypeBindings;
> import com.fasterxml.jackson.databind.util.SimpleBeanPropertyDefinition;
> +import com.fasterxml.jackson.databind.util.TokenBuffer;
> import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
> import java.beans.BeanInfo;
> import java.beans.IntrospectionException;
> @@ -499,7 +500,8 @@ public class PipelineOptionsFactory {
>
> private static final DefaultDeserializationContext DESERIALIZATION_CONTEXT =
> new DefaultDeserializationContext.Impl(MAPPER.getDeserializationContext().getFactory())
> - .createInstance(MAPPER.getDeserializationConfig(), null, null);
> + .createInstance(
> + MAPPER.getDeserializationConfig(), new TokenBuffer(MAPPER, false).asParser(), null);
> ```
You can get both PRs out (forward fix and rollback) and run the additional failing tests in the forward fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik merged pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik merged pull request #15338:
URL: https://github.com/apache/beam/pull/15338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.03%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head 87f2c72. Consider uploading reports for the commit 87f2c72 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.83% +0.03%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50206 +91
+ Misses 9689 9680 -9
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [18 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...87f2c72](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r698760690
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1058,13 +1086,29 @@ private static void validateMethodAnnotations(
validateGettersHaveConsistentAnnotation(
methodNameToAllMethodMap, descriptors, AnnotationPredicates.DEFAULT_VALUE);
+ // Verify that there is no getter with a mixed @JsonDeserialize annotation.
+ validateGettersHaveConsistentAnnotation(
Review comment:
Docs are updated, lemme know what you think of them, happy to improve/reword if you have suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-901252936
hm, this says there's changes requested still but I can't actually find them in the review... @lukecwik I think I addressed all your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-901428245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.03%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head 16224d3. Consider uploading reports for the commit 16224d3 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.83% +0.03%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50207 +92
+ Misses 9689 9679 -10
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [18 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...16224d3](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2ab0024) into [master](https://codecov.io/gh/apache/beam/commit/af59192ac95a564ca3d29b6385a6d1448f5a407d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (af59192) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2ab0024 differs from pull request most recent head 87f2c72. Consider uploading reports for the commit 87f2c72 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
- Coverage 83.81% 83.81% -0.01%
==========================================
Files 441 441
Lines 59773 59773
==========================================
- Hits 50101 50096 -5
- Misses 9672 9677 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.47% <0.00%> (-0.78%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.44% <0.00%> (-0.35%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.85% <0.00%> (-0.16%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.73% <0.00%> (-0.15%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...87f2c72](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-907472083
np! Hope you had fun camping. I agree about requiring both attributes if either is present, I was toying around with the same thing on our internal branch too. I'll address these in the next day or so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-909278581
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910533042
roll-forward PR up as well: #15439 , I'm not sure if the trigger phrase worked for that one though, maybe I don't have permissions?
I'd love to figure out how to reproduce this in a unit test as well, it seems like we're missing coverage over something pretty fundamental.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r691583028
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1601,80 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ private static JsonDeserializer<Object> deserializerForMethod(Method method, JavaType type)
+ throws JsonMappingException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ Object maybeDeserializerClass =
+ DESERIALIZATION_CONTEXT.getAnnotationIntrospector().findDeserializer(annotatedMethod);
+
+ JsonDeserializer<Object> jsonDeserializer =
+ DESERIALIZATION_CONTEXT.deserializerInstance(annotatedMethod, maybeDeserializerClass);
+
+ if (jsonDeserializer == null) {
+ jsonDeserializer = DESERIALIZATION_CONTEXT.findContextualValueDeserializer(type, prop);
+ }
+ return jsonDeserializer;
+ }
+
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
Review comment:
```suggestion
* used, otherwise the default ObjectMapper deserialization strategy is used.
```
##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsFactoryTest.java
##########
@@ -2080,6 +2082,93 @@ static String myStaticMethod(OptionsWithStaticMethod o) {
}
}
+ public static class SimpleParsedObject {
+ public String value;
+
+ public SimpleParsedObject(String value) {
+ this.value = value;
+ }
+ }
+
+ public interface OptionsWithParsing extends PipelineOptions {
+ SimpleParsedObject getSimple();
+
+ void setSimple(SimpleParsedObject value);
+ }
+
+ @Test
+ public void testAutoQuoteStringArgumentsForComplexObjects() {
+ OptionsWithParsing options =
+ PipelineOptionsFactory.fromArgs("--simple=test").as(OptionsWithParsing.class);
+
+ assertEquals("test", options.getSimple().value);
+ }
+
+ public static class ComplexType2 {
+ public String value;
+ }
+
+ public interface OptionsWithJsonDeserialize1 extends PipelineOptions {
+ @JsonDeserialize(using = ComplexType2Deserializer1.class)
+ ComplexType2 getComplexType();
+
+ void setComplexType(ComplexType2 value);
+ }
+
+ public interface OptionsWithJsonDeserialize2 extends PipelineOptions {
+ @JsonDeserialize(using = ComplexType2Deserializer2.class)
+ ComplexType2 getComplexType();
+
+ void setComplexType(ComplexType2 value);
+ }
+
+ public static class ComplexType2Deserializer1 extends StdDeserializer<ComplexType2> {
+ public ComplexType2Deserializer1() {
+ super(ComplexType2.class);
+ }
+
+ @Override
+ public ComplexType2 deserialize(JsonParser p, DeserializationContext ctxt)
+ throws IOException, JsonProcessingException {
+ ComplexType2 ct = new ComplexType2();
+ ct.value = p.getText();
+ return ct;
+ }
+ }
+
+ public static class ComplexType2Deserializer2 extends StdDeserializer<ComplexType2> {
+ public ComplexType2Deserializer2() {
+ super(ComplexType2.class);
+ }
+
+ @Override
+ public ComplexType2 deserialize(JsonParser p, DeserializationContext ctxt)
+ throws IOException, JsonProcessingException {
+ ComplexType2 ct = new ComplexType2();
+ ct.value = p.getText();
+ return ct;
+ }
+ }
+
+ @Test
+ public void testJsonDeserializeAttribute_NoConflict() {
+ OptionsWithJsonDeserialize1 options =
+ PipelineOptionsFactory.fromArgs("--complexType=test").as(OptionsWithJsonDeserialize1.class);
+
+ assertEquals("test", options.getComplexType().value);
+ }
+
+ @Test
+ public void testJsonDeserializeAttribute_Conflict() {
+ OptionsWithJsonDeserialize1 options =
+ PipelineOptionsFactory.fromArgs("--complexType=test").as(OptionsWithJsonDeserialize1.class);
+
+ expectedException.expect(IllegalArgumentException.class);
+ expectedException.expectMessage(
Review comment:
`ExpectedException` is deprecated, please use `assertThrows(..., lambda that throws exception);`
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1601,80 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ private static JsonDeserializer<Object> deserializerForMethod(Method method, JavaType type)
Review comment:
nit: method comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik merged pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik merged pull request #15338:
URL: https://github.com/apache/beam/pull/15338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-908782473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910490820
hm, weird. Yeah feel free to revert, I can look into the failures
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910605061
> Forgive the noob question, is there a way to run these post-commit tests on a PR?
There is a link in PR template: See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-909506552
Thanks, if you have some time and can look at why the test is flaking it would be greatly appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910531746
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.03%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head 87f2c72. Consider uploading reports for the commit 87f2c72 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.83% +0.03%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50206 +91
+ Misses 9689 9680 -9
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [18 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...87f2c72](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2ab0024) into [master](https://codecov.io/gh/apache/beam/commit/af59192ac95a564ca3d29b6385a6d1448f5a407d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (af59192) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2ab0024 differs from pull request most recent head 1c80016. Consider uploading reports for the commit 1c80016 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
- Coverage 83.81% 83.81% -0.01%
==========================================
Files 441 441
Lines 59773 59773
==========================================
- Hits 50101 50096 -5
- Misses 9672 9677 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.47% <0.00%> (-0.78%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.44% <0.00%> (-0.35%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.85% <0.00%> (-0.16%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.73% <0.00%> (-0.15%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...1c80016](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head 87f2c72. Consider uploading reports for the commit 87f2c72 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.82% +0.02%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50200 +85
+ Misses 9689 9686 -3
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [17 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...87f2c72](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2ab0024) into [master](https://codecov.io/gh/apache/beam/commit/af59192ac95a564ca3d29b6385a6d1448f5a407d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (af59192) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2ab0024 differs from pull request most recent head 7afc8c6. Consider uploading reports for the commit 7afc8c6 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
- Coverage 83.81% 83.81% -0.01%
==========================================
Files 441 441
Lines 59773 59773
==========================================
- Hits 50101 50096 -5
- Misses 9672 9677 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.47% <0.00%> (-0.78%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.44% <0.00%> (-0.35%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.85% <0.00%> (-0.16%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.73% <0.00%> (-0.15%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...7afc8c6](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690634935
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
jackson expects the input to be well formed json, which means if it's a string it needs to be double quoted. You can see it in a test just by removing these catch block and running the test, it'll fail with something like:
```
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'test': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (String)"test"; line: 1, column: 5]
```
The deserialization is a different problem (which is addressed by the JsonDeserialize attribute support).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-901252936
hm, this says there's changes requested still but I can't actually find them in the review... @lukecwik I think I addressed all your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690614446
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
+ } catch (JsonParseException inner) {
+ // rethrow the original exception rather the one thrown from the fallback attempt
+ throw e;
+ }
+ } else {
+ throw e;
+ }
+ }
+
+ JsonParser parser = new TreeTraversingParser(tree, MAPPER);
+ parser.nextToken();
+ return DESERIALIZATION_CONTEXT.readPropertyValue(parser, prop, type);
Review comment:
This sounds like a useful improvement but we'll want `@JsonSerialize` as well since we need the value to round trip to JSON and back (unless the getter is marked with `@JsonIgnore` in that case we don't need the `@JsonSerialize`).
But be aware that you'll want to ensure that all instances used via the "as" transform use the same annotation since multiple PipelineOptions can define the same property effectively and we would want to limit how inconsistent conversions can appear so this will limit the feature a little.
For example with `@JsonIgnore` https://github.com/apache/beam/blob/2ab002486345f5d97f11ccfbeb86541796c080e0/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java#L941
https://github.com/apache/beam/blob/2ab002486345f5d97f11ccfbeb86541796c080e0/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java#L1054
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
I saw the example in the test but am lost as two why converting `--myInstant=2021-07-17` actually fails. Would you be able to add an example of the failure to the tests?
Also, is this an issue with the type deserialize not being able to handle JSON string -> object conversion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-907469638
Sorry for the long wait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2ab0024) into [master](https://codecov.io/gh/apache/beam/commit/af59192ac95a564ca3d29b6385a6d1448f5a407d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (af59192) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2ab0024 differs from pull request most recent head 554b1ac. Consider uploading reports for the commit 554b1ac to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
- Coverage 83.81% 83.81% -0.01%
==========================================
Files 441 441
Lines 59773 59773
==========================================
- Hits 50101 50096 -5
- Misses 9672 9677 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.47% <0.00%> (-0.78%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.44% <0.00%> (-0.35%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.85% <0.00%> (-0.16%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.73% <0.00%> (-0.15%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...554b1ac](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690639303
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
+ } catch (JsonParseException inner) {
+ // rethrow the original exception rather the one thrown from the fallback attempt
+ throw e;
+ }
+ } else {
+ throw e;
+ }
+ }
+
+ JsonParser parser = new TreeTraversingParser(tree, MAPPER);
+ parser.nextToken();
+ return DESERIALIZATION_CONTEXT.readPropertyValue(parser, prop, type);
Review comment:
I'll double check, but I think it should already be supported. The places I've seen pass the options into the mapper directly, which will use the annotations. We only need this workaround here since the PipelineOptions class isn't directly deserialized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690634935
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
jackson expects the input to be well formed json, which means if it's a string it needs to be double quoted. You can see it in a test just by removing these catch block and running the test, it'll fail with something like:
```com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'test': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (String)"test"; line: 1, column: 5]
```
The deserialization is a different problem (which is addressed by the JsonDeserialize attribute support).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2ab0024) into [master](https://codecov.io/gh/apache/beam/commit/af59192ac95a564ca3d29b6385a6d1448f5a407d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (af59192) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2ab0024 differs from pull request most recent head 400c6ec. Consider uploading reports for the commit 400c6ec to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
- Coverage 83.81% 83.81% -0.01%
==========================================
Files 441 441
Lines 59773 59773
==========================================
- Hits 50101 50096 -5
- Misses 9672 9677 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.47% <0.00%> (-0.78%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.44% <0.00%> (-0.35%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.85% <0.00%> (-0.16%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.73% <0.00%> (-0.15%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...400c6ec](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.03%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head 16224d3. Consider uploading reports for the commit 16224d3 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.83% +0.03%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50207 +92
+ Misses 9689 9679 -10
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [18 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...16224d3](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690644227
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
ah yeah `convertValue` vs `readValue` use different code paths. afaik convert will serialize `from` to json first, then read it back, which is slightly different than the complex object case here, since the input string potentially is an actual json value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.03%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head f374523. Consider uploading reports for the commit f374523 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.83% +0.03%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50207 +92
+ Misses 9689 9679 -10
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [18 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...f374523](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r698722243
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1791,6 +1984,11 @@ private static void checkEmptyStringAllowed(
private final Map<Set<Class<? extends PipelineOptions>>, Registration<?>> combinedCache =
Maps.newConcurrentMap();
+ private final Map<Method, JsonDeserializer<Object>> deserializerCache = Maps.newConcurrentMap();
Review comment:
`computeDeserializerForMethod` should never return null here (it returns the default if there isn't one set)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2ab0024) into [master](https://codecov.io/gh/apache/beam/commit/af59192ac95a564ca3d29b6385a6d1448f5a407d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (af59192) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2ab0024 differs from pull request most recent head c45ce17. Consider uploading reports for the commit c45ce17 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
- Coverage 83.81% 83.81% -0.01%
==========================================
Files 441 441
Lines 59773 59773
==========================================
- Hits 50101 50096 -5
- Misses 9672 9677 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.47% <0.00%> (-0.78%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.44% <0.00%> (-0.35%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.85% <0.00%> (-0.16%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.73% <0.00%> (-0.15%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...c45ce17](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kw2542 commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
kw2542 commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910489301
@steveniemitz @lukecwik This commit seems to causing bunch of post commit tests to failure:
https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/172/
https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/1347/
https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/5024/
Should we revert this PR for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-908782904
Run Java_Examples_Dataflow PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.03%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head 87f2c72. Consider uploading reports for the commit 87f2c72 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.83% +0.03%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50206 +91
+ Misses 9689 9680 -9
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [18 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...87f2c72](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-910579866
looks like the failing tests are fixed in #15439 now, so we can either roll forward with that or roll back w/ the revert. I'd prefer to roll forward but I'll leave that up to you :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-900552905
# [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15338](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbb363f) into [master](https://codecov.io/gh/apache/beam/commit/12d624b90e3e62601e6c2b30b3acdff150ee1602?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12d624b) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.
> :exclamation: Current head cbb363f differs from pull request most recent head 87f2c72. Consider uploading reports for the commit 87f2c72 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15338 +/- ##
==========================================
+ Coverage 83.79% 83.82% +0.02%
==========================================
Files 440 440
Lines 59804 59886 +82
==========================================
+ Hits 50115 50197 +82
Misses 9689 9689
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `56.81% <0.00%> (-12.95%)` | :arrow_down: |
| [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | `86.45% <0.00%> (-0.99%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `75.53% <0.00%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `86.67% <0.00%> (-0.74%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <0.00%> (-0.07%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.95% <0.00%> (-0.06%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `95.16% <0.00%> (ø)` | |
| [.../python/apache\_beam/io/gcp/resource\_identifiers.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Jlc291cmNlX2lkZW50aWZpZXJzLnB5) | `100.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [16 more](https://codecov.io/gh/apache/beam/pull/15338/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80da0ec...87f2c72](https://codecov.io/gh/apache/beam/pull/15338?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-902736483
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-901409675
> hm, this says there's changes requested still but I can't actually find them in the review... @lukecwik I think I addressed all your comments.
Thats just what the UI will say forever until another review happens by the person who requested changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-902315338
I'm still tracking down one more issue w/ serialization now when serializing custom ValueProviders
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #15338:
URL: https://github.com/apache/beam/pull/15338#issuecomment-902197657
alright, all hooked up and I added some more tests to ProxyInvocationHandlerTest as well to test the round-trip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on a change in pull request #15338: [BEAM-12767] Improve PipelineOption parsing UX
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690639853
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an {@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+ SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(), annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
+ } catch (JsonParseException inner) {
+ // rethrow the original exception rather the one thrown from the fallback attempt
+ throw e;
+ }
+ } else {
+ throw e;
+ }
+ }
+
+ JsonParser parser = new TreeTraversingParser(tree, MAPPER);
+ parser.nextToken();
+ return DESERIALIZATION_CONTEXT.readPropertyValue(parser, prop, type);
Review comment:
> But be aware that you'll want to ensure that all instances used via the "as" transform use the same annotation
ack, I had saw this but didn't implement it yet pending agreement on using the jackson annotation. I'll add it in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org