You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "chenxuesdu (via GitHub)" <gi...@apache.org> on 2025/10/29 16:56:00 UTC
[PR] Update ChangeStreamDao to query differnet TVF for postgresSQL based on the change stream partition mode [beam]
chenxuesdu opened a new pull request, #36667:
URL: https://github.com/apache/beam/pull/36667
Add a private variable isMutableKeyRange, which will be initialized during ChangeStreamDao constructor by reading from internal spanner table.
In function changeStreamQuery, when constructing change stream query, based on the value of isMutableKeyRange, call different TVFs when the Dialect is PostgreSQL(The googleSQL part is completed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "jiangzzhu (via GitHub)" <gi...@apache.org>.
jiangzzhu commented on code in PR #36667:
URL: https://github.com/apache/beam/pull/36667#discussion_r2632893093
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadChangeStreamTest.java:
##########
@@ -140,4 +150,53 @@ public void testWithDefaultCredential() {
assertEquals(defaultCredential, changeStreamSpannerConfigWithCredential.getCredentials().get());
assertEquals(defaultCredential, metadataSpannerConfigWithCredential.getCredentials().get());
}
+
+ @Test
+ public void testFetchPartitionModeGoogleSql() {
Review Comment:
can we parameterized the tests to have different combination among googlesql/postgresql vs immuatable/mutable/empty string ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #36667:
URL: https://github.com/apache/beam/pull/36667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "chenxuesdu (via GitHub)" <gi...@apache.org>.
chenxuesdu commented on code in PR #36667:
URL: https://github.com/apache/beam/pull/36667#discussion_r2632434843
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/ChangeStreamDao.java:
##########
@@ -91,8 +107,20 @@ public ChangeStreamResultSet changeStreamQuery(
String query = "";
Statement statement;
if (this.isPostgres()) {
- query =
- "SELECT * FROM \"spanner\".\"read_json_" + changeStreamName + "\"($1, $2, $3, $4, null)";
+ // Ensure we have determined whether change stream uses mutable key range
+ boolean isMutable = isMutableKeyRangeChangeStream();
Review Comment:
Moved the logic to SpannerIO so it will only be executed once when connector starts. Please take a look. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "tianz101 (via GitHub)" <gi...@apache.org>.
tianz101 commented on code in PR #36667:
URL: https://github.com/apache/beam/pull/36667#discussion_r2479444689
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/ChangeStreamDao.java:
##########
@@ -138,4 +166,76 @@ public ChangeStreamResultSet changeStreamQuery(
private boolean isPostgres() {
return this.dialect == Dialect.POSTGRESQL;
}
+
+ // Returns the PartitionMode, fetching from Spanner on first call and caching.
+ protected PartitionMode getPartitionMode() {
+ if (this.partitionMode != PartitionMode.UNKNOWN) {
+ return this.partitionMode;
+ }
+ synchronized (this) {
+ if (this.partitionMode == PartitionMode.UNKNOWN) {
+ String fetchedPartitionMode =
+ fetchPartitionMode(this.databaseClient, this.dialect, this.changeStreamName);
+ if (fetchedPartitionMode.isEmpty()
+ || fetchedPartitionMode.equalsIgnoreCase("IMMUTABLE_KEY_RANGE")) {
+ this.partitionMode = PartitionMode.IMMUTABLE_KEY_RANGE;
+ } else {
+ this.partitionMode = PartitionMode.MUTABLE_KEY_RANGE;
+ }
+ }
+ }
+ return this.partitionMode;
+ }
+
+ // Convenience boolean method kept for compatibility
+ protected boolean isMutableKeyRangeChangeStream() {
+ return getPartitionMode() == PartitionMode.MUTABLE_KEY_RANGE;
+ }
+
+ // Returns the partition_mode option value for the given change stream.
+ private static String fetchPartitionMode(
+ DatabaseClient databaseClient, Dialect dialect, String changeStreamName) {
+ try (ReadOnlyTransaction tx = databaseClient.readOnlyTransaction()) {
+ Statement statement;
+ if (dialect == Dialect.POSTGRESQL) {
+ statement =
+ Statement.newBuilder(
+ "select option_name, option_value\n"
+ + "from information_schema.change_stream_options\n"
+ + "where change_stream_name = $1")
Review Comment:
Can we add option_name = 'partition_mode' here so we can get the parititon mode directly and no need to filter later?
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/ChangeStreamDao.java:
##########
@@ -91,8 +107,20 @@ public ChangeStreamResultSet changeStreamQuery(
String query = "";
Statement statement;
if (this.isPostgres()) {
- query =
- "SELECT * FROM \"spanner\".\"read_json_" + changeStreamName + "\"($1, $2, $3, $4, null)";
+ // Ensure we have determined whether change stream uses mutable key range
+ boolean isMutable = isMutableKeyRangeChangeStream();
Review Comment:
Is this called each time we issue the query? As our query will be constantly checkpointed and canceled and re-issued, this can be called many times.
I think we just need to query the partition mode once for the whole pipeline.
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/ChangeStreamDao.java:
##########
@@ -138,4 +166,76 @@ public ChangeStreamResultSet changeStreamQuery(
private boolean isPostgres() {
return this.dialect == Dialect.POSTGRESQL;
}
+
+ // Returns the PartitionMode, fetching from Spanner on first call and caching.
+ protected PartitionMode getPartitionMode() {
+ if (this.partitionMode != PartitionMode.UNKNOWN) {
+ return this.partitionMode;
+ }
+ synchronized (this) {
+ if (this.partitionMode == PartitionMode.UNKNOWN) {
+ String fetchedPartitionMode =
+ fetchPartitionMode(this.databaseClient, this.dialect, this.changeStreamName);
+ if (fetchedPartitionMode.isEmpty()
+ || fetchedPartitionMode.equalsIgnoreCase("IMMUTABLE_KEY_RANGE")) {
+ this.partitionMode = PartitionMode.IMMUTABLE_KEY_RANGE;
+ } else {
+ this.partitionMode = PartitionMode.MUTABLE_KEY_RANGE;
+ }
+ }
+ }
+ return this.partitionMode;
+ }
+
+ // Convenience boolean method kept for compatibility
+ protected boolean isMutableKeyRangeChangeStream() {
+ return getPartitionMode() == PartitionMode.MUTABLE_KEY_RANGE;
+ }
+
+ // Returns the partition_mode option value for the given change stream.
+ private static String fetchPartitionMode(
+ DatabaseClient databaseClient, Dialect dialect, String changeStreamName) {
+ try (ReadOnlyTransaction tx = databaseClient.readOnlyTransaction()) {
+ Statement statement;
+ if (dialect == Dialect.POSTGRESQL) {
+ statement =
+ Statement.newBuilder(
+ "select option_name, option_value\n"
+ + "from information_schema.change_stream_options\n"
+ + "where change_stream_name = $1")
+ .bind("p1")
+ .to(changeStreamName)
+ .build();
+ } else {
+ statement =
+ Statement.newBuilder(
+ "select option_name, option_value\n"
+ + "from information_schema.change_stream_options\n"
+ + "where change_stream_name = @changeStreamName")
Review Comment:
same comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "tianz101 (via GitHub)" <gi...@apache.org>.
tianz101 commented on code in PR #36667:
URL: https://github.com/apache/beam/pull/36667#discussion_r2636628669
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadChangeStreamTest.java:
##########
@@ -140,4 +150,53 @@ public void testWithDefaultCredential() {
assertEquals(defaultCredential, changeStreamSpannerConfigWithCredential.getCredentials().get());
assertEquals(defaultCredential, metadataSpannerConfigWithCredential.getCredentials().get());
}
+
+ @Test
+ public void testFetchPartitionModeGoogleSql() {
+ DatabaseClient databaseClient = Mockito.mock(DatabaseClient.class);
+ ReadOnlyTransaction transaction = Mockito.mock(ReadOnlyTransaction.class);
+ ResultSet resultSet = Mockito.mock(ResultSet.class);
+
+ when(databaseClient.readOnlyTransaction()).thenReturn(transaction);
+ when(transaction.executeQuery(any(Statement.class))).thenReturn(resultSet);
Review Comment:
If through mock, how the fetch option sql statement syntax in SpannerIO.java is tested to be correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36667:
URL: https://github.com/apache/beam/pull/36667#issuecomment-3506493436
Reminder, please take a look at this pr: @m-trieu @nielm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #36667:
URL: https://github.com/apache/beam/pull/36667#issuecomment-3516945155
waiting on author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36667:
URL: https://github.com/apache/beam/pull/36667#issuecomment-3465534396
Assigning reviewers:
R: @m-trieu for label java.
R: @nielm for label spanner.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query differnet TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #36667:
URL: https://github.com/apache/beam/pull/36667#issuecomment-3462685743
## Summary of Changes
Hello @chenxuesdu, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request enhances the Spanner change stream connector by introducing logic to adapt query behavior for PostgreSQL dialects based on the change stream's partition mode. Specifically, it allows the system to correctly query change streams that use either MUTABLE_KEY_RANGE or IMMUTABLE_KEY_RANGE partitioning by selecting the appropriate internal Spanner Table-Valued Function, ensuring proper data retrieval and compatibility.
### Highlights
* **Dynamic TVF Selection for PostgreSQL**: The ChangeStreamDao now dynamically selects between 'read_proto_bytes_' and 'read_json_' Table-Valued Functions (TVFs) for PostgreSQL change stream queries, based on the change stream's partition mode.
* **isMutableKeyRange Field**: A new private boolean field, 'isMutableKeyRange', has been introduced in ChangeStreamDao to store the detected partition mode.
* **Constructor Initialization**: The 'isMutableKeyRange' field is initialized during the ChangeStreamDao constructor by querying the 'information_schema.change_stream_options' table to determine the change stream's partition mode.
* **New Test File**: A new test file, ChangeStreamDaoTest.java, has been added to validate the logic for determining and using the 'isMutableKeyRange' property.
<details>
<summary><b>Using Gemini Code Assist</b></summary>
<br>
The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
<b>Invoking Gemini</b>
You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help | `/gemini help` | Displays a list of available commands.
<b>Customization</b>
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
<b>Limitations & Feedback</b>
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
<b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b>
</details>
[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "chenxuesdu (via GitHub)" <gi...@apache.org>.
chenxuesdu commented on code in PR #36667:
URL: https://github.com/apache/beam/pull/36667#discussion_r2632433097
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/ChangeStreamDao.java:
##########
@@ -138,4 +166,76 @@ public ChangeStreamResultSet changeStreamQuery(
private boolean isPostgres() {
return this.dialect == Dialect.POSTGRESQL;
}
+
+ // Returns the PartitionMode, fetching from Spanner on first call and caching.
+ protected PartitionMode getPartitionMode() {
+ if (this.partitionMode != PartitionMode.UNKNOWN) {
+ return this.partitionMode;
+ }
+ synchronized (this) {
+ if (this.partitionMode == PartitionMode.UNKNOWN) {
+ String fetchedPartitionMode =
+ fetchPartitionMode(this.databaseClient, this.dialect, this.changeStreamName);
+ if (fetchedPartitionMode.isEmpty()
+ || fetchedPartitionMode.equalsIgnoreCase("IMMUTABLE_KEY_RANGE")) {
+ this.partitionMode = PartitionMode.IMMUTABLE_KEY_RANGE;
+ } else {
+ this.partitionMode = PartitionMode.MUTABLE_KEY_RANGE;
+ }
+ }
+ }
+ return this.partitionMode;
+ }
+
+ // Convenience boolean method kept for compatibility
+ protected boolean isMutableKeyRangeChangeStream() {
+ return getPartitionMode() == PartitionMode.MUTABLE_KEY_RANGE;
+ }
+
+ // Returns the partition_mode option value for the given change stream.
+ private static String fetchPartitionMode(
+ DatabaseClient databaseClient, Dialect dialect, String changeStreamName) {
+ try (ReadOnlyTransaction tx = databaseClient.readOnlyTransaction()) {
+ Statement statement;
+ if (dialect == Dialect.POSTGRESQL) {
+ statement =
+ Statement.newBuilder(
+ "select option_name, option_value\n"
+ + "from information_schema.change_stream_options\n"
+ + "where change_stream_name = $1")
Review Comment:
Updated.
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/ChangeStreamDao.java:
##########
@@ -138,4 +166,76 @@ public ChangeStreamResultSet changeStreamQuery(
private boolean isPostgres() {
return this.dialect == Dialect.POSTGRESQL;
}
+
+ // Returns the PartitionMode, fetching from Spanner on first call and caching.
+ protected PartitionMode getPartitionMode() {
+ if (this.partitionMode != PartitionMode.UNKNOWN) {
+ return this.partitionMode;
+ }
+ synchronized (this) {
+ if (this.partitionMode == PartitionMode.UNKNOWN) {
+ String fetchedPartitionMode =
+ fetchPartitionMode(this.databaseClient, this.dialect, this.changeStreamName);
+ if (fetchedPartitionMode.isEmpty()
+ || fetchedPartitionMode.equalsIgnoreCase("IMMUTABLE_KEY_RANGE")) {
+ this.partitionMode = PartitionMode.IMMUTABLE_KEY_RANGE;
+ } else {
+ this.partitionMode = PartitionMode.MUTABLE_KEY_RANGE;
+ }
+ }
+ }
+ return this.partitionMode;
+ }
+
+ // Convenience boolean method kept for compatibility
+ protected boolean isMutableKeyRangeChangeStream() {
+ return getPartitionMode() == PartitionMode.MUTABLE_KEY_RANGE;
+ }
+
+ // Returns the partition_mode option value for the given change stream.
+ private static String fetchPartitionMode(
+ DatabaseClient databaseClient, Dialect dialect, String changeStreamName) {
+ try (ReadOnlyTransaction tx = databaseClient.readOnlyTransaction()) {
+ Statement statement;
+ if (dialect == Dialect.POSTGRESQL) {
+ statement =
+ Statement.newBuilder(
+ "select option_name, option_value\n"
+ + "from information_schema.change_stream_options\n"
+ + "where change_stream_name = $1")
+ .bind("p1")
+ .to(changeStreamName)
+ .build();
+ } else {
+ statement =
+ Statement.newBuilder(
+ "select option_name, option_value\n"
+ + "from information_schema.change_stream_options\n"
+ + "where change_stream_name = @changeStreamName")
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36667:
URL: https://github.com/apache/beam/pull/36667#issuecomment-3749303237
Reminder, please take a look at this pr: @m-trieu @nielm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "chenxuesdu (via GitHub)" <gi...@apache.org>.
chenxuesdu commented on code in PR #36667:
URL: https://github.com/apache/beam/pull/36667#discussion_r2700258611
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadChangeStreamTest.java:
##########
@@ -120,7 +129,8 @@ public void testSetSpannerConfigCredential() {
@Test
public void testWithDefaultCredential() {
- // Get the default credential, without setting any credentials in the pipeline options or
+ // Get the default credential, without setting any credentials in the pipeline
+ // options or
Review Comment:
Updated. Thanks.
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadChangeStreamTest.java:
##########
@@ -140,4 +150,53 @@ public void testWithDefaultCredential() {
assertEquals(defaultCredential, changeStreamSpannerConfigWithCredential.getCredentials().get());
assertEquals(defaultCredential, metadataSpannerConfigWithCredential.getCredentials().get());
}
+
+ @Test
+ public void testFetchPartitionModeGoogleSql() {
+ DatabaseClient databaseClient = Mockito.mock(DatabaseClient.class);
Review Comment:
Refactored. Please take a look.
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadChangeStreamTest.java:
##########
@@ -140,4 +150,53 @@ public void testWithDefaultCredential() {
assertEquals(defaultCredential, changeStreamSpannerConfigWithCredential.getCredentials().get());
assertEquals(defaultCredential, metadataSpannerConfigWithCredential.getCredentials().get());
}
+
+ @Test
+ public void testFetchPartitionModeGoogleSql() {
+ DatabaseClient databaseClient = Mockito.mock(DatabaseClient.class);
+ ReadOnlyTransaction transaction = Mockito.mock(ReadOnlyTransaction.class);
+ ResultSet resultSet = Mockito.mock(ResultSet.class);
+
+ when(databaseClient.readOnlyTransaction()).thenReturn(transaction);
+ when(transaction.executeQuery(any(Statement.class))).thenReturn(resultSet);
Review Comment:
Updated the test. please take a look.
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadChangeStreamTest.java:
##########
@@ -140,4 +150,53 @@ public void testWithDefaultCredential() {
assertEquals(defaultCredential, changeStreamSpannerConfigWithCredential.getCredentials().get());
assertEquals(defaultCredential, metadataSpannerConfigWithCredential.getCredentials().get());
}
+
+ @Test
+ public void testFetchPartitionModeGoogleSql() {
Review Comment:
Will do it in a separate PR.
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadChangeStreamTest.java:
##########
@@ -140,4 +150,53 @@ public void testWithDefaultCredential() {
assertEquals(defaultCredential, changeStreamSpannerConfigWithCredential.getCredentials().get());
assertEquals(defaultCredential, metadataSpannerConfigWithCredential.getCredentials().get());
}
+
+ @Test
+ public void testFetchPartitionModeGoogleSql() {
+ DatabaseClient databaseClient = Mockito.mock(DatabaseClient.class);
+ ReadOnlyTransaction transaction = Mockito.mock(ReadOnlyTransaction.class);
+ ResultSet resultSet = Mockito.mock(ResultSet.class);
+
+ when(databaseClient.readOnlyTransaction()).thenReturn(transaction);
+ when(transaction.executeQuery(any(Statement.class))).thenReturn(resultSet);
+ when(resultSet.next()).thenReturn(true).thenReturn(false);
+ when(resultSet.getString(0)).thenReturn("MUTABLE_KEY_RANGE");
+
+ assertEquals(
+ true,
+ SpannerIO.isMutableChangeStream(
+ databaseClient, Dialect.GOOGLE_STANDARD_SQL, TEST_CHANGE_STREAM));
+ }
+
+ @Test
+ public void testFetchPartitionModePostgres() {
+ DatabaseClient databaseClient = Mockito.mock(DatabaseClient.class);
+ ReadOnlyTransaction transaction = Mockito.mock(ReadOnlyTransaction.class);
+ ResultSet resultSet = Mockito.mock(ResultSet.class);
+
+ when(databaseClient.readOnlyTransaction()).thenReturn(transaction);
+ when(transaction.executeQuery(any(Statement.class))).thenReturn(resultSet);
+ when(resultSet.next()).thenReturn(true).thenReturn(false);
+ when(resultSet.getString(0)).thenReturn("IMMUTABLE_KEY_RANGE");
+
+ assertEquals(
+ false,
+ SpannerIO.isMutableChangeStream(databaseClient, Dialect.POSTGRESQL, TEST_CHANGE_STREAM));
+ }
+
+ @Test
+ public void testFetchPartitionModeNotFound() {
+ DatabaseClient databaseClient = Mockito.mock(DatabaseClient.class);
+ ReadOnlyTransaction transaction = Mockito.mock(ReadOnlyTransaction.class);
+ ResultSet resultSet = Mockito.mock(ResultSet.class);
+
+ when(databaseClient.readOnlyTransaction()).thenReturn(transaction);
+ when(transaction.executeQuery(any(Statement.class))).thenReturn(resultSet);
+ when(resultSet.next()).thenReturn(false);
+
+ assertEquals(
+ false,
+ SpannerIO.isMutableChangeStream(
+ databaseClient, Dialect.GOOGLE_STANDARD_SQL, TEST_CHANGE_STREAM));
+ }
}
Review Comment:
Updated the test. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "chenxuesdu (via GitHub)" <gi...@apache.org>.
chenxuesdu commented on code in PR #36667:
URL: https://github.com/apache/beam/pull/36667#discussion_r2723158597
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadChangeStreamTest.java:
##########
@@ -19,21 +19,39 @@
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
import com.google.auth.Credentials;
import com.google.cloud.Timestamp;
+import com.google.cloud.spanner.DatabaseClient;
+import com.google.cloud.spanner.Dialect;
import com.google.cloud.spanner.Options.RpcPriority;
+import com.google.cloud.spanner.ReadOnlyTransaction;
+import com.google.cloud.spanner.ResultSet;
+import com.google.cloud.spanner.Statement;
+import java.util.Arrays;
+import java.util.Collection;
import org.apache.beam.sdk.extensions.gcp.auth.TestCredential;
import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.io.gcp.spanner.changestreams.MetadataSpannerConfigFactory;
import org.apache.beam.sdk.testing.TestPipeline;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
+import org.junit.runners.Parameterized.Parameters;
+import org.mockito.ArgumentCaptor;
-@RunWith(JUnit4.class)
+@RunWith(Enclosed.class)
Review Comment:
Enclosed is used to support parameterized test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "tianz101 (via GitHub)" <gi...@apache.org>.
tianz101 commented on code in PR #36667:
URL: https://github.com/apache/beam/pull/36667#discussion_r2636399940
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadChangeStreamTest.java:
##########
@@ -120,7 +129,8 @@ public void testSetSpannerConfigCredential() {
@Test
public void testWithDefaultCredential() {
- // Get the default credential, without setting any credentials in the pipeline options or
+ // Get the default credential, without setting any credentials in the pipeline
+ // options or
Review Comment:
The following 2 lines of comments can be combined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "tianz101 (via GitHub)" <gi...@apache.org>.
tianz101 commented on code in PR #36667:
URL: https://github.com/apache/beam/pull/36667#discussion_r2636622865
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadChangeStreamTest.java:
##########
@@ -140,4 +150,53 @@ public void testWithDefaultCredential() {
assertEquals(defaultCredential, changeStreamSpannerConfigWithCredential.getCredentials().get());
assertEquals(defaultCredential, metadataSpannerConfigWithCredential.getCredentials().get());
}
+
+ @Test
+ public void testFetchPartitionModeGoogleSql() {
+ DatabaseClient databaseClient = Mockito.mock(DatabaseClient.class);
Review Comment:
These lines are the same for all tests below, can they be refactored to avoid repeatition? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36667:
URL: https://github.com/apache/beam/pull/36667#issuecomment-3768059109
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @Abacn for label java.
R: @nielm for label spanner.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "tianz101 (via GitHub)" <gi...@apache.org>.
tianz101 commented on code in PR #36667:
URL: https://github.com/apache/beam/pull/36667#discussion_r2636629364
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadChangeStreamTest.java:
##########
@@ -140,4 +150,53 @@ public void testWithDefaultCredential() {
assertEquals(defaultCredential, changeStreamSpannerConfigWithCredential.getCredentials().get());
assertEquals(defaultCredential, metadataSpannerConfigWithCredential.getCredentials().get());
}
+
+ @Test
+ public void testFetchPartitionModeGoogleSql() {
+ DatabaseClient databaseClient = Mockito.mock(DatabaseClient.class);
+ ReadOnlyTransaction transaction = Mockito.mock(ReadOnlyTransaction.class);
+ ResultSet resultSet = Mockito.mock(ResultSet.class);
+
+ when(databaseClient.readOnlyTransaction()).thenReturn(transaction);
+ when(transaction.executeQuery(any(Statement.class))).thenReturn(resultSet);
+ when(resultSet.next()).thenReturn(true).thenReturn(false);
+ when(resultSet.getString(0)).thenReturn("MUTABLE_KEY_RANGE");
+
+ assertEquals(
+ true,
+ SpannerIO.isMutableChangeStream(
+ databaseClient, Dialect.GOOGLE_STANDARD_SQL, TEST_CHANGE_STREAM));
+ }
+
+ @Test
+ public void testFetchPartitionModePostgres() {
+ DatabaseClient databaseClient = Mockito.mock(DatabaseClient.class);
+ ReadOnlyTransaction transaction = Mockito.mock(ReadOnlyTransaction.class);
+ ResultSet resultSet = Mockito.mock(ResultSet.class);
+
+ when(databaseClient.readOnlyTransaction()).thenReturn(transaction);
+ when(transaction.executeQuery(any(Statement.class))).thenReturn(resultSet);
+ when(resultSet.next()).thenReturn(true).thenReturn(false);
+ when(resultSet.getString(0)).thenReturn("IMMUTABLE_KEY_RANGE");
+
+ assertEquals(
+ false,
+ SpannerIO.isMutableChangeStream(databaseClient, Dialect.POSTGRESQL, TEST_CHANGE_STREAM));
+ }
+
+ @Test
+ public void testFetchPartitionModeNotFound() {
+ DatabaseClient databaseClient = Mockito.mock(DatabaseClient.class);
+ ReadOnlyTransaction transaction = Mockito.mock(ReadOnlyTransaction.class);
+ ResultSet resultSet = Mockito.mock(ResultSet.class);
+
+ when(databaseClient.readOnlyTransaction()).thenReturn(transaction);
+ when(transaction.executeQuery(any(Statement.class))).thenReturn(resultSet);
+ when(resultSet.next()).thenReturn(false);
+
+ assertEquals(
+ false,
+ SpannerIO.isMutableChangeStream(
+ databaseClient, Dialect.GOOGLE_STANDARD_SQL, TEST_CHANGE_STREAM));
+ }
}
Review Comment:
Also should the the failure or throw exception case of fetch option sql statement be tested as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode [beam]
Posted by "tianz101 (via GitHub)" <gi...@apache.org>.
tianz101 commented on code in PR #36667:
URL: https://github.com/apache/beam/pull/36667#discussion_r2723144822
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadChangeStreamTest.java:
##########
@@ -19,21 +19,39 @@
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
import com.google.auth.Credentials;
import com.google.cloud.Timestamp;
+import com.google.cloud.spanner.DatabaseClient;
+import com.google.cloud.spanner.Dialect;
import com.google.cloud.spanner.Options.RpcPriority;
+import com.google.cloud.spanner.ReadOnlyTransaction;
+import com.google.cloud.spanner.ResultSet;
+import com.google.cloud.spanner.Statement;
+import java.util.Arrays;
+import java.util.Collection;
import org.apache.beam.sdk.extensions.gcp.auth.TestCredential;
import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.io.gcp.spanner.changestreams.MetadataSpannerConfigFactory;
import org.apache.beam.sdk.testing.TestPipeline;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
+import org.junit.runners.Parameterized.Parameters;
+import org.mockito.ArgumentCaptor;
-@RunWith(JUnit4.class)
+@RunWith(Enclosed.class)
Review Comment:
why this one is Enclosed.class? How is it different from JUnit4.class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org