You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "bvolpato (via GitHub)" <gi...@apache.org> on 2023/04/28 16:18:26 UTC
[GitHub] [beam] bvolpato opened a new pull request, #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
bvolpato opened a new pull request, #26474:
URL: https://github.com/apache/beam/pull/26474
Cherry-picking https://github.com/apache/beam/pull/25824 to the release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26474:
URL: https://github.com/apache/beam/pull/26474#discussion_r1180751916
##########
sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcSchemaIOProvider.java:
##########
@@ -202,11 +204,14 @@ protected JdbcIO.DataSourceConfiguration getDataSourceConfiguration() {
dataSourceConfiguration = dataSourceConfiguration.withConnectionInitSqls(initSqls);
}
- if (config.getSchema().hasField("maxConnections")) {
- @Nullable Integer maxConnections = config.getInt32("maxConnections");
- if (maxConnections != null) {
- dataSourceConfiguration = dataSourceConfiguration.withMaxConnections(maxConnections);
- }
+ @Nullable Integer maxConnections = config.getInt32("maxConnections");
Review Comment:
I see, the actual breaking change happens here. This if clause is removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #26474:
URL: https://github.com/apache/beam/pull/26474#issuecomment-1528804671
Run Python 3.11 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26474:
URL: https://github.com/apache/beam/pull/26474#issuecomment-1528520220
Run Python 3.11 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26474:
URL: https://github.com/apache/beam/pull/26474#discussion_r1180743271
##########
sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcSchemaIOProvider.java:
##########
@@ -67,6 +67,8 @@ public Schema configurationSchema() {
.addNullableField("fetchSize", FieldType.INT16)
.addNullableField("outputParallelization", FieldType.BOOLEAN)
.addNullableField("autosharding", FieldType.BOOLEAN)
+ .addNullableField("maxConnections", FieldType.INT16)
Review Comment:
This is a breaking change for xlang schema transform, see #25062
in general we may need a better way to make this kind of change not breaking xlang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bvolpato commented on pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #26474:
URL: https://github.com/apache/beam/pull/26474#issuecomment-1527795012
R: @jrmccluskey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26474:
URL: https://github.com/apache/beam/pull/26474#issuecomment-1527984966
Possibly related: https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Python311/105/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #26474:
URL: https://github.com/apache/beam/pull/26474#issuecomment-1527828445
## [Codecov](https://codecov.io/gh/apache/beam/pull/26474?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#26474](https://codecov.io/gh/apache/beam/pull/26474?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0762aad) into [release-2.47.0](https://codecov.io/gh/apache/beam/commit/4d1f1f19086d80bb809f76ab8cd7d409bafe122e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4d1f1f1) will **decrease** coverage by `0.62%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## release-2.47.0 #26474 +/- ##
==================================================
- Coverage 71.17% 70.55% -0.62%
==================================================
Files 787 787
Lines 103294 103311 +17
==================================================
- Hits 73518 72896 -622
- Misses 28287 28926 +639
Partials 1489 1489
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `78.94% <ø> (-0.93%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/26474?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/26474?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `79.16% <ø> (ø)` | |
... and [29 files with indirect coverage changes](https://codecov.io/gh/apache/beam/pull/26474/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26474:
URL: https://github.com/apache/beam/pull/26474#issuecomment-1527985707
Run Python 3.11 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26474:
URL: https://github.com/apache/beam/pull/26474#discussion_r1180822556
##########
sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcSchemaIOProvider.java:
##########
@@ -67,6 +67,8 @@ public Schema configurationSchema() {
.addNullableField("fetchSize", FieldType.INT16)
.addNullableField("outputParallelization", FieldType.BOOLEAN)
.addNullableField("autosharding", FieldType.BOOLEAN)
+ .addNullableField("maxConnections", FieldType.INT16)
Review Comment:
opened #26480
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #26474:
URL: https://github.com/apache/beam/pull/26474#issuecomment-1528804687
Run Python_Integration PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey merged pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey merged PR #26474:
URL: https://github.com/apache/beam/pull/26474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26474:
URL: https://github.com/apache/beam/pull/26474#issuecomment-1527795917
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #26474:
URL: https://github.com/apache/beam/pull/26474#issuecomment-1527895001
Run Python_Integration PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26474:
URL: https://github.com/apache/beam/pull/26474#discussion_r1180820688
##########
sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcSchemaIOProvider.java:
##########
@@ -67,6 +67,8 @@ public Schema configurationSchema() {
.addNullableField("fetchSize", FieldType.INT16)
.addNullableField("outputParallelization", FieldType.BOOLEAN)
.addNullableField("autosharding", FieldType.BOOLEAN)
+ .addNullableField("maxConnections", FieldType.INT16)
Review Comment:
I see, I think I find the cause. Here maxConnections is "FieldType.INT16" but in jdbc.py it is "typing.Optional[int]". This configuration gets encoded by ShortCoder then decode with VarIntCoder and data corruption happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bvolpato commented on pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #26474:
URL: https://github.com/apache/beam/pull/26474#issuecomment-1528455165
Updated with patch provided by @Abacn at https://github.com/apache/beam/pull/25062. Thanks Yi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #26474:
URL: https://github.com/apache/beam/pull/26474#issuecomment-1528804950
I'm not anticipating this PR significantly impacting the Python tests I'm re-running, will merge after those have any result
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26474:
URL: https://github.com/apache/beam/pull/26474#issuecomment-1528821939
Dataflow tests are flaky recently due to no worker available in us-central1 region.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on pull request #26474: Cherry-pick: Add driverJars parameter to JdbcIO. (#25824)
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #26474:
URL: https://github.com/apache/beam/pull/26474#issuecomment-1528823274
That would explain the flakey failures I've been seeing on streaming integration tests. Thank you for the context!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org