You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2023/05/04 19:02:41 UTC

[GitHub] [beam] ahmedabu98 opened a new pull request, #26559: Add batch storage write api test addressing recent dataloss case #26503

ahmedabu98 opened a new pull request, #26559:
URL: https://github.com/apache/beam/pull/26559

   Fixes #26531
   
   The data-loss fix is in #26503. This PR creates a test to make sure this case is covered.
   
   Data-loss happened when there were multiple appends to the same stream. This would usually happen with batch writes large enough to go over the append thresholds set here: https://github.com/apache/beam/blob/250488211e5b60ae0f8221b1b3dc6288531f12ee/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryOptions.java#L149-L159
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on a diff in pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #26559:
URL: https://github.com/apache/beam/pull/26559#discussion_r1189056764


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java:
##########
@@ -2684,6 +2685,52 @@ public void testWriteToTableDecorator() throws Exception {
     p.run();
   }
 
+  @Test
+  public void testBatchStorageWriteWithMultipleAppendsPerStream() throws Exception {
+    assumeTrue(useStorageApi);

Review Comment:
   The test ran and passed for [Storage Write API](https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Commit/2551/testReport/org.apache.beam.sdk.io.gcp.bigquery/BigQueryIOWriteTest/testBatchStorageWriteWithMultipleAppendsPerStream_2_/) and [Storage write at least once](https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Commit/2551/testReport/org.apache.beam.sdk.io.gcp.bigquery/BigQueryIOWriteTest/testBatchStorageWriteWithMultipleAppendsPerStream_3_/)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1561950377

   R: @Abacn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1590010031

   test was added in #26976


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1562059091

   test is flaky


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1561964352

   Test looks good. 
   
   Discussion not blocking this PR:
   
   I am wondering shouldn't batch job always be "exactly once"? Then what is the point of using STORAGE_WRITE_AT_LEAST_ONCE in batch


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1561993462

   > Then what is the point of using STORAGE_WRITE_AT_LEAST_ONCE in batch
   
   Including `STORAGE_API_AT_LEAST_ONCE` batch tests because a different code path is taken when this method is set: https://github.com/apache/beam/blob/d91ea8e44918a7ef10ac8b33001cddab61bcedc5/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiLoads.java#L103-L110
   
   > note that this setting adds 42 unit tests
   
   Many of those tests will end up being skipped, but agree that this test class can be cleaned up to reduce the runtime


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1563434027

   Is it flaky because of the test (is there eventual consistency on the writes?) or is it flaky because of the feature?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1561974192

   note that this setting adds 42 unit tests:
   
   ![image](https://github.com/apache/beam/assets/8010435/12f1a285-5f11-434c-a0b1-e31e6f914b3a)
   
   while adding coverage, we may also needs to revisit if there are redundant tests (I suspect there are) as BigQueryIOWriteTest is already the main contributor of the run time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1563445450

   The test added in this PR is flaky. From the log it fails in processElement (not PAssert) it seems indicating issues in feature:
   
    it fails a trigger run after succeeded with a commit run: https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Phrase/174/
   
   ```
   org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Bad append: 0 + for stream 4b30ad01-673f-4c50-87ba-e9a9154af9b5 expected 1
   	at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:374)
   ...
   Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Bad append: 0 + for stream 4b30ad01-673f-4c50-87ba-e9a9154af9b5 expected 1
   	at org.apache.beam.sdk.io.gcp.bigquery.StorageApiWriteUnshardedRecords$WriteRecordsDoFn$DestinationState.lambda$flush$7(StorageApiWriteUnshardedRecords.java:578)
   	at org.apache.beam.sdk.io.gcp.bigquery.RetryManager$Operation.run(RetryManager.java:132)
   	at org.apache.beam.sdk.io.gcp.bigquery.RetryManager.run(RetryManager.java:248)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on a diff in pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #26559:
URL: https://github.com/apache/beam/pull/26559#discussion_r1189059993


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java:
##########
@@ -2684,6 +2685,52 @@ public void testWriteToTableDecorator() throws Exception {
     p.run();
   }
 
+  @Test
+  public void testBatchStorageWriteWithMultipleAppendsPerStream() throws Exception {
+    assumeTrue(useStorageApi);

Review Comment:
   Btw not sure about Java, but many of our Python tests use the parameterized equivalent, (e.g. [here](https://github.com/apache/beam/blob/f0f4707bacf4f86dee344fb91978363f3eff7b12/sdks/python/apache_beam/io/gcp/bigquery_test.py#L1470-L1474), [here](https://github.com/apache/beam/blob/f0f4707bacf4f86dee344fb91978363f3eff7b12/sdks/python/apache_beam/io/filesystem_test.py#L125), [here](https://github.com/apache/beam/blob/f0f4707bacf4f86dee344fb91978363f3eff7b12/sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py#L742))



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on a diff in pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #26559:
URL: https://github.com/apache/beam/pull/26559#discussion_r1185429757


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java:
##########
@@ -180,6 +180,7 @@ public static Iterable<Object[]> data() {
         new Object[] {false, false, false},
         new Object[] {false, false, true},
         new Object[] {true, false, false},
+        new Object[] {true, true, false},

Review Comment:
   I noticed storage write at least once mode for batch wasn't included in these tests so I added them here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on a diff in pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #26559:
URL: https://github.com/apache/beam/pull/26559#discussion_r1189064439


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java:
##########
@@ -2684,6 +2685,52 @@ public void testWriteToTableDecorator() throws Exception {
     p.run();
   }
 
+  @Test
+  public void testBatchStorageWriteWithMultipleAppendsPerStream() throws Exception {
+    assumeTrue(useStorageApi);

Review Comment:
   Although yes, the way these tests are configured with `assumeTrue()`, many of them will be skipped silently (e.g. this test got skipped for [FILE_LOADS](https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Commit/2551/testReport/org.apache.beam.sdk.io.gcp.bigquery/BigQueryIOWriteTest/testBatchStorageWriteWithMultipleAppendsPerStream_0_/)). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] github-actions[bot] commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1535264346

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1535315517

   This test fails when run against the code prior to the changes in #26503 with:
   ```
   java.lang.RuntimeException: Bad append: 0 + for stream 112e49de-0c2c-4a4e-820f-d8b96d339c83 expected 1
   ```
   
   And getting the familiar error when removing the fake BQ services and using a real table ref:
   ```
   com.google.cloud.bigquery.storage.v1.Exceptions$OffsetAlreadyExists: ALREADY_EXISTS: The offset is within stream, expected offset 1, received 0
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1555131462

   retest this please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1561364379

   `[org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest.testTriggeredFileLoadsWithTempTablesToExistingNullSchemaTable[1]](https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Phrase/167/testReport/org.apache.beam.sdk.io.gcp.bigquery/BigQueryIOWriteTest/testTriggeredFileLoadsWithTempTablesToExistingNullSchemaTable_1_/)` is a known flake


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on a diff in pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #26559:
URL: https://github.com/apache/beam/pull/26559#discussion_r1185429757


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java:
##########
@@ -180,6 +180,7 @@ public static Iterable<Object[]> data() {
         new Object[] {false, false, false},
         new Object[] {false, false, true},
         new Object[] {true, false, false},
+        new Object[] {true, true, false},

Review Comment:
   I noticed storage write at least once mode for batch wasn't included in these tests. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] bvolpato commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1535660918

   That's great, thanks @ahmedabu98 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1561974631

   Run Java_GCP_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles commented on a diff in pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #26559:
URL: https://github.com/apache/beam/pull/26559#discussion_r1188909631


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java:
##########
@@ -2684,6 +2685,52 @@ public void testWriteToTableDecorator() throws Exception {
     p.run();
   }
 
+  @Test
+  public void testBatchStorageWriteWithMultipleAppendsPerStream() throws Exception {
+    assumeTrue(useStorageApi);

Review Comment:
   Hmm I guess we'd better just leave it, but just check that it executed for now. We might want to revisit the design of the test later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1555326998

   Run Java_GCP_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on a diff in pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #26559:
URL: https://github.com/apache/beam/pull/26559#discussion_r1187303763


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java:
##########
@@ -2684,6 +2685,52 @@ public void testWriteToTableDecorator() throws Exception {
     p.run();
   }
 
+  @Test
+  public void testBatchStorageWriteWithMultipleAppendsPerStream() throws Exception {
+    assumeTrue(useStorageApi);

Review Comment:
   The parameters are initialized here and all the tests in this class rely on them: https://github.com/apache/beam/blob/f903b9c6021a87af92c9bf03c01e1a5bf6a318d6/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java#L177-L186
   
   Should we add another test that checks for existing configurations? e.g. check a configuration exists for streaming with storage api



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1555060402

   Can you iterate here and get the tests green?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #26559:
URL: https://github.com/apache/beam/pull/26559#issuecomment-1535262897

   R: @reuvenlax 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles commented on a diff in pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #26559:
URL: https://github.com/apache/beam/pull/26559#discussion_r1186439481


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java:
##########
@@ -2684,6 +2685,52 @@ public void testWriteToTableDecorator() throws Exception {
     p.run();
   }
 
+  @Test
+  public void testBatchStorageWriteWithMultipleAppendsPerStream() throws Exception {
+    assumeTrue(useStorageApi);

Review Comment:
   I do believe this can cause a test to silently never be executed if there is a misconfiguration of the parameterization. I could be wrong or out of date on this...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 closed pull request #26559: Add batch storage write api test addressing recent dataloss case #26503

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 closed pull request #26559: Add batch storage write api test addressing recent dataloss case #26503
URL: https://github.com/apache/beam/pull/26559


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org