You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "reuvenlax (via GitHub)" <gi...@apache.org> on 2023/05/26 22:16:20 UTC
[GitHub] [beam] reuvenlax opened a new pull request, #26919: warn if BigQuery failed rows collection is not processed
reuvenlax opened a new pull request, #26919:
URL: https://github.com/apache/beam/pull/26919
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax merged pull request #26919: warn if BigQuery failed rows collection is not processed
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax merged PR #26919:
URL: https://github.com/apache/beam/pull/26919
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on pull request #26919: warn if BigQuery failed rows collection is not processed
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #26919:
URL: https://github.com/apache/beam/pull/26919#issuecomment-1565715259
R: @shunping-google
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bvolpato commented on pull request #26919: warn if BigQuery failed rows collection is not processed
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #26919:
URL: https://github.com/apache/beam/pull/26919#issuecomment-1565639892
Actually it's not only apply, need to rename the generic type
```
23:20:19 [ant:checkstyle] [ERROR] /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Spotless_Commit/src/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingWriteTables.java:306:11: Name 'TupleTagType' must match pattern '^(((T|K|V|W|X)[0-9]*)|([A-Z][a-z][a-zA-Z]*T))$'. [MethodTypeParameterName]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26919: warn if BigQuery failed rows collection is not processed
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26919:
URL: https://github.com/apache/beam/pull/26919#issuecomment-1565715300
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bvolpato commented on pull request #26919: warn if BigQuery failed rows collection is not processed
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #26919:
URL: https://github.com/apache/beam/pull/26919#issuecomment-1565665317
Run Java_GCP_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on pull request #26919: warn if BigQuery failed rows collection is not processed
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #26919:
URL: https://github.com/apache/beam/pull/26919#issuecomment-1565714969
R: @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26919: warn if BigQuery failed rows collection is not processed
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26919:
URL: https://github.com/apache/beam/pull/26919#issuecomment-1565053532
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bvolpato commented on a diff in pull request #26919: warn if BigQuery failed rows collection is not processed
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on code in PR #26919:
URL: https://github.com/apache/beam/pull/26919#discussion_r1207505401
##########
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java:
##########
@@ -2384,7 +2389,93 @@ public void visitPrimitiveTransform(@UnknownKeyFor @NonNull @Initialized Node no
}
@Test
- public void testPubinkDynamicOverride() throws IOException {
+ public void testBigQueryDLQWarningStreamingInsertsConsumed() throws Exception {
+ testBigQueryDLQWarning(BigQueryIO.Write.Method.STREAMING_INSERTS, true);
+ }
+
+ @Test
+ public void testBigQueryDLQWarningStreamingInsertsNotConsumed() throws Exception {
+ testBigQueryDLQWarning(BigQueryIO.Write.Method.STREAMING_INSERTS, false);
+ }
+
+ @Test
+ public void testBigQueryDLQWarningStorageApiConsumed() throws Exception {
+ testBigQueryDLQWarning(BigQueryIO.Write.Method.STORAGE_WRITE_API, true);
+ }
+
+ @Test
+ public void testBigQueryDLQWarningStorageApiNotConsumed() throws Exception {
+ testBigQueryDLQWarning(BigQueryIO.Write.Method.STORAGE_WRITE_API, false);
+ }
+
+ @Test
+ public void testBigQueryDLQWarningStorageApiALOConsumed() throws Exception {
+ testBigQueryDLQWarning(BigQueryIO.Write.Method.STORAGE_API_AT_LEAST_ONCE, true);
+ }
+
+ @Test
+ public void testBigQueryDLQWarningStorageApiALONotConsumed() throws Exception {
+ testBigQueryDLQWarning(BigQueryIO.Write.Method.STORAGE_API_AT_LEAST_ONCE, false);
+ }
+
+ public void testBigQueryDLQWarning(BigQueryIO.Write.Method method, boolean processFailures)
+ throws IOException {
+ PipelineOptions options = buildPipelineOptions();
+ List<String> experiments =
+ new ArrayList<>(
+ ImmutableList.of(
+ GcpOptions.STREAMING_ENGINE_EXPERIMENT,
+ GcpOptions.WINDMILL_SERVICE_EXPERIMENT,
+ "use_runner_v2"));
Review Comment:
nit: experiments irrelevant here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org