You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/01/25 17:15:31 UTC
[GitHub] [beam] benWize opened a new pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
benWize opened a new pull request #16609:
URL: https://github.com/apache/beam/pull/16609
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1036641462
took a quick look, but will defer to @kileys for final review&merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize edited a comment on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize edited a comment on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1034086348
Tests executed in each runner:
- [Direct](https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Direct/lastCompletedBuild/testReport/)
- [Spark](https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Spark/4/testReport/). Some of the [disabled](https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Spark/3/testReport/junit/org.apache.beam.examples.complete/TrafficMaxLaneFlowIT/testE2ETrafficMaxLaneFlow/) tests failed with an error `Failed to serialize and deserialize property 'bigQuerySchema' with value 'GenericData{classInfo=[fields], {fields=[GenericData{classInfo=[categories, collationSpec, description, fields, maxLength, mode, name, policyTags, precision, scale, type], {name=station_id, type=STRING}}`
- [Flink](https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Flink/4/testReport/)
- [Dataflow](https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/1784/testReport/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1055731315
Run Java Examples_Flink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1065358349
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r821126227
##########
File path: examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
##########
@@ -467,7 +467,6 @@ public void processElement(ProcessContext c) {
public static void runAutocompletePipeline(Options options) throws IOException {
- options.setBigQuerySchema(FormatForBigquery.getSchema());
Review comment:
For these examples, the schema is added directly to the transform https://github.com/apache/beam/blob/46155767ecf0694d86257e9c3a8b5fd5bbc0afbb/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java#L515 using `FormatForBigquery.getSchema()`, not sure why was added to options if it wasn't used from there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r821136958
##########
File path: runners/flink/flink_runner.gradle
##########
@@ -285,8 +285,6 @@ createJavaExamplesArchetypeValidationTask(type: 'Quickstart', runner: 'FlinkLoca
task examplesIntegrationTest(type: Test) {
group = "Verification"
- // Disable gradle cache
- outputs.upToDateWhen { false }
Review comment:
I've used some of the configs to launch tests using FlinkRunner from the ValidatesRunner task, I'm not sure if for this case is it worth it to reuse previous outputs from the cache or not. I lean more towards the second.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1062026153
Run Java_Examples_Dataflow_Java11 PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r816068747
##########
File path: runners/spark/spark_runner.gradle
##########
@@ -421,10 +421,11 @@ task examplesIntegrationTest(type: Test) {
systemProperty "spark.ui.showConsoleProgress", "false"
include '**/*IT.class'
+ exclude '**/TriggerExampleIT.class' // OOM Exception
Review comment:
No, I will create one and link here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r805017360
##########
File path: examples/java/src/test/java/org/apache/beam/examples/cookbook/MaxPerKeyExamplesIT.java
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.cookbook;
+
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.client.util.BackOff;
+import com.google.api.client.util.BackOffUtils;
+import com.google.api.client.util.Sleeper;
+import com.google.api.services.bigquery.model.QueryResponse;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.extensions.gcp.util.BackOffAdapter;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.joda.time.Duration;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** An end-to-end test for {@link org.apache.beam.examples.cookbook.MaxPerKeyExamples}. */
+@RunWith(JUnit4.class)
+public class MaxPerKeyExamplesIT {
+ private MaxPerKeyExamplesIT.MaxPerKeyExamplesOptions options;
+ private final String timestamp = Long.toString(System.currentTimeMillis());
+ private final String outputDatasetId = "max_per_key_examples" + timestamp;
+ private final String outputTable = "max_per_key_examples_table";
+ private final Long defaultExpiration = 1000L * 60 * 60;
+ private String projectId;
+ private BigqueryClient bqClient;
+
+ /** Options for the MaxPerKeyExamples Integration Test. */
+ public interface MaxPerKeyExamplesOptions
+ extends TestPipelineOptions, MaxPerKeyExamples.Options {}
+
+ @Before
+ public void setupTestEnvironment() throws Exception {
+ PipelineOptionsFactory.register(TestPipelineOptions.class);
+ this.options =
+ TestPipeline.testingPipelineOptions()
+ .as(MaxPerKeyExamplesIT.MaxPerKeyExamplesOptions.class);
+ this.projectId = TestPipeline.testingPipelineOptions().as(GcpOptions.class).getProject();
+ this.bqClient = new BigqueryClient("MaxPerKeyExamplesIT");
+ this.bqClient.createNewDataset(this.projectId, this.outputDatasetId, defaultExpiration);
+ }
+
+ @After
+ public void cleanupTestEnvironment() {
+ this.bqClient.deleteDataset(this.projectId, this.outputDatasetId);
+ }
+
+ @Test
+ public void testE2EMaxPerKeyExamples() throws Exception {
+ this.options.setOutput(
+ String.format("%s:%s.%s", this.projectId, this.outputDatasetId, this.outputTable));
+ MaxPerKeyExamples.runMaxPerKeyExamples(options);
+ FluentBackoff backoffFactory =
+ FluentBackoff.DEFAULT.withMaxRetries(4).withInitialBackoff(Duration.standardSeconds(1L));
Review comment:
how long is the longest back-off period?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kileys commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
kileys commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r814216340
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java
##########
@@ -381,6 +381,13 @@ private QueryResponse queryWithRetries(
/** Creates a new dataset. */
public void createNewDataset(String projectId, String datasetId)
throws IOException, InterruptedException {
+ createNewDataset(projectId, datasetId, null);
+ }
+
+ /** Creates a new dataset with defaultTableExpiration. */
+ public void createNewDataset(
+ String projectId, String datasetId, @Nullable Long defaultTableExpiration)
Review comment:
Can you specify that the expiration is in ms?
##########
File path: runners/spark/spark_runner.gradle
##########
@@ -421,10 +421,11 @@ task examplesIntegrationTest(type: Test) {
systemProperty "spark.ui.showConsoleProgress", "false"
include '**/*IT.class'
+ exclude '**/TriggerExampleIT.class' // OOM Exception
Review comment:
Is there a Jira for these to be enabled later?
##########
File path: examples/java/src/test/java/org/apache/beam/examples/cookbook/CombinePerKeyExamplesIT.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.cookbook;
+
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.client.util.BackOff;
+import com.google.api.client.util.BackOffUtils;
+import com.google.api.client.util.Sleeper;
+import com.google.api.services.bigquery.model.QueryResponse;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.extensions.gcp.util.BackOffAdapter;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.joda.time.Duration;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** An end-to-end test for {@link org.apache.beam.examples.cookbook.CombinePerKeyExamples}. */
+@RunWith(JUnit4.class)
+public class CombinePerKeyExamplesIT {
+ private CombinePerKeyExamplesOptions options;
+ private final String timestamp = Long.toString(System.currentTimeMillis());
+ private final String outputDatasetId = "combine_per_key_examples" + timestamp;
+ private final String outputTable = "combine_per_key_examples_table";
+ private final Long defaultExpiration = 1000L * 60 * 60;
+ private String projectId;
+ private BigqueryClient bqClient;
+
+ /** Options for the CombinePerKeyExamples Integration Test. */
+ public interface CombinePerKeyExamplesOptions
+ extends TestPipelineOptions, CombinePerKeyExamples.Options {}
+
+ @Before
+ public void setupTestEnvironment() throws Exception {
+ PipelineOptionsFactory.register(TestPipelineOptions.class);
+ this.options = TestPipeline.testingPipelineOptions().as(CombinePerKeyExamplesOptions.class);
+ this.projectId = TestPipeline.testingPipelineOptions().as(GcpOptions.class).getProject();
+ this.bqClient = new BigqueryClient("CombinePerKeyExamplesIT");
+ this.bqClient.createNewDataset(this.projectId, this.outputDatasetId, defaultExpiration);
Review comment:
Is there a benefit to creating/deleting the datasets? It looks like some integration tests reuse the same dataset with a table expiration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1055841575
Run Java Examples_Direct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r817198848
##########
File path: runners/spark/spark_runner.gradle
##########
@@ -421,10 +421,11 @@ task examplesIntegrationTest(type: Test) {
systemProperty "spark.ui.showConsoleProgress", "false"
include '**/*IT.class'
+ exclude '**/TriggerExampleIT.class' // OOM Exception
Review comment:
Filled https://issues.apache.org/jira/browse/BEAM-14019 to track these
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1023670698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1055888591
Last test results for each runner:
- [Direct](https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Direct/7/)
- [Spark](https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Spark/5/)
- [Flink](https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Flink/5/)
- [Dataflow](https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2_PR/57/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1055886270
Run Java Examples on Dataflow Runner V2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kileys commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
kileys commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r817988930
##########
File path: examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
##########
@@ -467,7 +467,6 @@ public void processElement(ProcessContext c) {
public static void runAutocompletePipeline(Options options) throws IOException {
- options.setBigQuerySchema(FormatForBigquery.getSchema());
Review comment:
Why are these being removed?
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java
##########
@@ -381,6 +381,13 @@ private QueryResponse queryWithRetries(
/** Creates a new dataset. */
public void createNewDataset(String projectId, String datasetId)
throws IOException, InterruptedException {
+ createNewDataset(projectId, datasetId, null);
+ }
+
+ /** Creates a new dataset with defaultTableExpiration in ms. */
+ public void createNewDataset(
+ String projectId, String datasetId, @Nullable Long defaultTableExpiration)
Review comment:
```suggestion
String projectId, String datasetId, @Nullable Long defaultTableExpirationMs)
```
##########
File path: examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleIT.java
##########
@@ -1,100 +0,0 @@
-/*
Review comment:
Something wrong with this test?
##########
File path: runners/flink/flink_runner.gradle
##########
@@ -285,8 +285,6 @@ createJavaExamplesArchetypeValidationTask(type: 'Quickstart', runner: 'FlinkLoca
task examplesIntegrationTest(type: Test) {
group = "Verification"
- // Disable gradle cache
- outputs.upToDateWhen { false }
Review comment:
Why is this removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1055731567
Run Java Examples_Spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1055731815
Run Java Examples_Direct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1021466742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1021464977
Run Java Examples_Dataflow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1023721305
Run Java Examples on Dataflow Runner V2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1023670556
Run Java Examples_Spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1064627464
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1021536550
Run Java Examples_Flink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r817199378
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java
##########
@@ -381,6 +381,13 @@ private QueryResponse queryWithRetries(
/** Creates a new dataset. */
public void createNewDataset(String projectId, String datasetId)
throws IOException, InterruptedException {
+ createNewDataset(projectId, datasetId, null);
+ }
+
+ /** Creates a new dataset with defaultTableExpiration. */
+ public void createNewDataset(
+ String projectId, String datasetId, @Nullable Long defaultTableExpiration)
Review comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r821119851
##########
File path: examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
##########
@@ -467,7 +467,6 @@ public void processElement(ProcessContext c) {
public static void runAutocompletePipeline(Options options) throws IOException {
- options.setBigQuerySchema(FormatForBigquery.getSchema());
Review comment:
The property bigQuerySchema fails to serialize and deserialize for portable runners as part of PipelineOptions.
Stacktrace snippet
`Failed to serialize PipelineOptions
java.lang.IllegalArgumentException: Failed to serialize PipelineOptions
at org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:70)`
`Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected IOException (of type java.io.IOException): Failed to serialize and deserialize property 'bigQuerySchema' with value 'GenericData{classInfo=[fields], `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kileys merged pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
kileys merged pull request #16609:
URL: https://github.com/apache/beam/pull/16609
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r805017360
##########
File path: examples/java/src/test/java/org/apache/beam/examples/cookbook/MaxPerKeyExamplesIT.java
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.cookbook;
+
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.client.util.BackOff;
+import com.google.api.client.util.BackOffUtils;
+import com.google.api.client.util.Sleeper;
+import com.google.api.services.bigquery.model.QueryResponse;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.extensions.gcp.util.BackOffAdapter;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.joda.time.Duration;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** An end-to-end test for {@link org.apache.beam.examples.cookbook.MaxPerKeyExamples}. */
+@RunWith(JUnit4.class)
+public class MaxPerKeyExamplesIT {
+ private MaxPerKeyExamplesIT.MaxPerKeyExamplesOptions options;
+ private final String timestamp = Long.toString(System.currentTimeMillis());
+ private final String outputDatasetId = "max_per_key_examples" + timestamp;
+ private final String outputTable = "max_per_key_examples_table";
+ private final Long defaultExpiration = 1000L * 60 * 60;
+ private String projectId;
+ private BigqueryClient bqClient;
+
+ /** Options for the MaxPerKeyExamples Integration Test. */
+ public interface MaxPerKeyExamplesOptions
+ extends TestPipelineOptions, MaxPerKeyExamples.Options {}
+
+ @Before
+ public void setupTestEnvironment() throws Exception {
+ PipelineOptionsFactory.register(TestPipelineOptions.class);
+ this.options =
+ TestPipeline.testingPipelineOptions()
+ .as(MaxPerKeyExamplesIT.MaxPerKeyExamplesOptions.class);
+ this.projectId = TestPipeline.testingPipelineOptions().as(GcpOptions.class).getProject();
+ this.bqClient = new BigqueryClient("MaxPerKeyExamplesIT");
+ this.bqClient.createNewDataset(this.projectId, this.outputDatasetId, defaultExpiration);
+ }
+
+ @After
+ public void cleanupTestEnvironment() {
+ this.bqClient.deleteDataset(this.projectId, this.outputDatasetId);
+ }
+
+ @Test
+ public void testE2EMaxPerKeyExamples() throws Exception {
+ this.options.setOutput(
+ String.format("%s:%s.%s", this.projectId, this.outputDatasetId, this.outputTable));
+ MaxPerKeyExamples.runMaxPerKeyExamples(options);
+ FluentBackoff backoffFactory =
+ FluentBackoff.DEFAULT.withMaxRetries(4).withInitialBackoff(Duration.standardSeconds(1L));
Review comment:
how long is the longest back-off period with this configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1022709778
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1021536386
Run Java Examples_Direct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize removed a comment on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize removed a comment on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1021464977
Run Java Examples_Dataflow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r816069436
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java
##########
@@ -381,6 +381,13 @@ private QueryResponse queryWithRetries(
/** Creates a new dataset. */
public void createNewDataset(String projectId, String datasetId)
throws IOException, InterruptedException {
+ createNewDataset(projectId, datasetId, null);
+ }
+
+ /** Creates a new dataset with defaultTableExpiration. */
+ public void createNewDataset(
+ String projectId, String datasetId, @Nullable Long defaultTableExpiration)
Review comment:
Yes, I will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kileys commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
kileys commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1063579897
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1034079808
@tvalentyn could you help me to review this? I added some tests to check examples end to end. There are some test examples still in progress and some others are disabled because they failed for some runners, but it could be useful to have an early review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1034086348
Tests executed in each runner:
- [Direct](https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Direct/lastCompletedBuild/testReport/)
- [Spark](https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Spark/4/testReport/)
- [Flink](https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Flink/4/testReport/)
- [Dataflow](https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/1784/testReport/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r806276759
##########
File path: examples/java/src/test/java/org/apache/beam/examples/cookbook/MaxPerKeyExamplesIT.java
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.cookbook;
+
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.client.util.BackOff;
+import com.google.api.client.util.BackOffUtils;
+import com.google.api.client.util.Sleeper;
+import com.google.api.services.bigquery.model.QueryResponse;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.extensions.gcp.util.BackOffAdapter;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.joda.time.Duration;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** An end-to-end test for {@link org.apache.beam.examples.cookbook.MaxPerKeyExamples}. */
+@RunWith(JUnit4.class)
+public class MaxPerKeyExamplesIT {
+ private MaxPerKeyExamplesIT.MaxPerKeyExamplesOptions options;
+ private final String timestamp = Long.toString(System.currentTimeMillis());
+ private final String outputDatasetId = "max_per_key_examples" + timestamp;
+ private final String outputTable = "max_per_key_examples_table";
+ private final Long defaultExpiration = 1000L * 60 * 60;
+ private String projectId;
+ private BigqueryClient bqClient;
+
+ /** Options for the MaxPerKeyExamples Integration Test. */
+ public interface MaxPerKeyExamplesOptions
+ extends TestPipelineOptions, MaxPerKeyExamples.Options {}
+
+ @Before
+ public void setupTestEnvironment() throws Exception {
+ PipelineOptionsFactory.register(TestPipelineOptions.class);
+ this.options =
+ TestPipeline.testingPipelineOptions()
+ .as(MaxPerKeyExamplesIT.MaxPerKeyExamplesOptions.class);
+ this.projectId = TestPipeline.testingPipelineOptions().as(GcpOptions.class).getProject();
+ this.bqClient = new BigqueryClient("MaxPerKeyExamplesIT");
+ this.bqClient.createNewDataset(this.projectId, this.outputDatasetId, defaultExpiration);
+ }
+
+ @After
+ public void cleanupTestEnvironment() {
+ this.bqClient.deleteDataset(this.projectId, this.outputDatasetId);
+ }
+
+ @Test
+ public void testE2EMaxPerKeyExamples() throws Exception {
+ this.options.setOutput(
+ String.format("%s:%s.%s", this.projectId, this.outputDatasetId, this.outputTable));
+ MaxPerKeyExamples.runMaxPerKeyExamples(options);
+ FluentBackoff backoffFactory =
+ FluentBackoff.DEFAULT.withMaxRetries(4).withInitialBackoff(Duration.standardSeconds(1L));
Review comment:
I think is like 2 minutes max, the default exponent is 1.5 and the max retries are 4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r821130853
##########
File path: examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleIT.java
##########
@@ -1,100 +0,0 @@
-/*
Review comment:
The test for TriggerExample is not running for either runner (Flink, Spark, Direct, Dataflow), I removed it from here because I don't see a benefit of adding a non-working IT disabled test. I think further work on this example can be tracked in a different issue, like the created for other disabled examples https://issues.apache.org/jira/browse/BEAM-14019.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r821132967
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java
##########
@@ -381,6 +381,13 @@ private QueryResponse queryWithRetries(
/** Creates a new dataset. */
public void createNewDataset(String projectId, String datasetId)
throws IOException, InterruptedException {
+ createNewDataset(projectId, datasetId, null);
+ }
+
+ /** Creates a new dataset with defaultTableExpiration in ms. */
+ public void createNewDataset(
+ String projectId, String datasetId, @Nullable Long defaultTableExpiration)
Review comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r806276759
##########
File path: examples/java/src/test/java/org/apache/beam/examples/cookbook/MaxPerKeyExamplesIT.java
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.cookbook;
+
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.client.util.BackOff;
+import com.google.api.client.util.BackOffUtils;
+import com.google.api.client.util.Sleeper;
+import com.google.api.services.bigquery.model.QueryResponse;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.extensions.gcp.util.BackOffAdapter;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.joda.time.Duration;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** An end-to-end test for {@link org.apache.beam.examples.cookbook.MaxPerKeyExamples}. */
+@RunWith(JUnit4.class)
+public class MaxPerKeyExamplesIT {
+ private MaxPerKeyExamplesIT.MaxPerKeyExamplesOptions options;
+ private final String timestamp = Long.toString(System.currentTimeMillis());
+ private final String outputDatasetId = "max_per_key_examples" + timestamp;
+ private final String outputTable = "max_per_key_examples_table";
+ private final Long defaultExpiration = 1000L * 60 * 60;
+ private String projectId;
+ private BigqueryClient bqClient;
+
+ /** Options for the MaxPerKeyExamples Integration Test. */
+ public interface MaxPerKeyExamplesOptions
+ extends TestPipelineOptions, MaxPerKeyExamples.Options {}
+
+ @Before
+ public void setupTestEnvironment() throws Exception {
+ PipelineOptionsFactory.register(TestPipelineOptions.class);
+ this.options =
+ TestPipeline.testingPipelineOptions()
+ .as(MaxPerKeyExamplesIT.MaxPerKeyExamplesOptions.class);
+ this.projectId = TestPipeline.testingPipelineOptions().as(GcpOptions.class).getProject();
+ this.bqClient = new BigqueryClient("MaxPerKeyExamplesIT");
+ this.bqClient.createNewDataset(this.projectId, this.outputDatasetId, defaultExpiration);
+ }
+
+ @After
+ public void cleanupTestEnvironment() {
+ this.bqClient.deleteDataset(this.projectId, this.outputDatasetId);
+ }
+
+ @Test
+ public void testE2EMaxPerKeyExamples() throws Exception {
+ this.options.setOutput(
+ String.format("%s:%s.%s", this.projectId, this.outputDatasetId, this.outputTable));
+ MaxPerKeyExamples.runMaxPerKeyExamples(options);
+ FluentBackoff backoffFactory =
+ FluentBackoff.DEFAULT.withMaxRetries(4).withInitialBackoff(Duration.standardSeconds(1L));
Review comment:
I think is like 2 minutes max, the default exponent is 1.5 and the max retries are 4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #16609:
URL: https://github.com/apache/beam/pull/16609#discussion_r816076966
##########
File path: examples/java/src/test/java/org/apache/beam/examples/cookbook/CombinePerKeyExamplesIT.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.cookbook;
+
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.client.util.BackOff;
+import com.google.api.client.util.BackOffUtils;
+import com.google.api.client.util.Sleeper;
+import com.google.api.services.bigquery.model.QueryResponse;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.extensions.gcp.util.BackOffAdapter;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.joda.time.Duration;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** An end-to-end test for {@link org.apache.beam.examples.cookbook.CombinePerKeyExamples}. */
+@RunWith(JUnit4.class)
+public class CombinePerKeyExamplesIT {
+ private CombinePerKeyExamplesOptions options;
+ private final String timestamp = Long.toString(System.currentTimeMillis());
+ private final String outputDatasetId = "combine_per_key_examples" + timestamp;
+ private final String outputTable = "combine_per_key_examples_table";
+ private final Long defaultExpiration = 1000L * 60 * 60;
+ private String projectId;
+ private BigqueryClient bqClient;
+
+ /** Options for the CombinePerKeyExamples Integration Test. */
+ public interface CombinePerKeyExamplesOptions
+ extends TestPipelineOptions, CombinePerKeyExamples.Options {}
+
+ @Before
+ public void setupTestEnvironment() throws Exception {
+ PipelineOptionsFactory.register(TestPipelineOptions.class);
+ this.options = TestPipeline.testingPipelineOptions().as(CombinePerKeyExamplesOptions.class);
+ this.projectId = TestPipeline.testingPipelineOptions().as(GcpOptions.class).getProject();
+ this.bqClient = new BigqueryClient("CombinePerKeyExamplesIT");
+ this.bqClient.createNewDataset(this.projectId, this.outputDatasetId, defaultExpiration);
Review comment:
I have seen both scenarios, for example, this existing test
https://github.com/apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/complete/TrafficMaxLaneFlowIT.java#L63
I think creating and deleting a dataset inside the test could help to avoid side effects because of leftover data, but I don't have a strong preference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1067258043
Thank you both!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1021536658
Run Java Examples_Spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1022709593
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1021438904
Run Java Examples_Direct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1022822606
Run Java Examples_Direct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1021625433
Run Java Examples on Dataflow Runner V2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [WIP][BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1023641716
Run Java Examples_Direct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #16609: [BEAM-12572] Java examples should get continuously exercised on at least 2 runners
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #16609:
URL: https://github.com/apache/beam/pull/16609#issuecomment-1067189311
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org