You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "clairemcginty (via GitHub)" <gi...@apache.org> on 2024/12/12 19:55:37 UTC
[PR] Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… [beam]
clairemcginty opened a new pull request, #33368:
URL: https://github.com/apache/beam/pull/33368
Rationale:
I would like to use [gcs-connector](https://github.com/GoogleCloudDataproc/hadoop-connectors/tree/master/gcs) 3.x, which supports the new Parquet VectorIO feature. However, gcs-connector 3.x also drops Java 8 and targets Java 11, which blocks us from upgrading it directly in Beam, since Beam is still targeting 8 (see https://github.com/apache/beam/pull/31898#issuecomment-2229819968).
Additionally, as a Beam user, I can't just upgrade gcs-connector on my end, due to breaking changes in how `GoogleCloudStorageImpl` is instantiated: in 2.x it has [public constructors](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.26/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageImpl.java#L302-L394), but in 3.x it [drops the public constructors and enforces a Builder pattern](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.0.4/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageImpl.java#L2496-L2517).
Therefore, when running on gcs-connector 3.x, my pipeline throws a NoSuchMethodError from `GcsUtil` when it tries to invoke the 2.x constructor: https://github.com/apache/beam/blob/v2.61.0/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java#L727
This PR adds a pipeline option for a GoogleCloudStorage Provider, so that users who want to use gcs-connector 3.x can be unblocked from doing so. It defaults to invoking the gcs-connector 2.x public constructor, but 3.x users can override it to use the Builder.
------------------------
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support gcs-connector 3.x in GcsUtil [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2590200779
catch only the checked reflection Exceptions should fix the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2540013743
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support gcs-connector 3.x in GcsUtil [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2590137895
looks like there was a spotbugs error - looking into it now..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support gcs-connector 3.x in GcsUtil [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2590195476
yeah, can be found in "SpotBugs Results": https://github.com/apache/beam/actions/runs/12768890245?pr=33368
```
Code | Warning
-- | --
REC | Exception is caught when Exception is not thrown in org.apache.beam.sdk.extensions.gcp.util.GcsUtil.createGoogleCloudStorage(GoogleCloudStorageOptions, Storage, Credentials)
| Bug type REC_CATCH_EXCEPTION (click for details)In class org.apache.beam.sdk.extensions.gcp.util.GcsUtilIn method org.apache.beam.sdk.extensions.gcp.util.GcsUtil.createGoogleCloudStorage(GoogleCloudStorageOptions, Storage, Credentials)At GcsUtil.java:[line 780]
Code Warning
REC Exception is caught when Exception is not thrown in org.apache.beam.sdk.extensions.gcp.util.GcsUtil.createGoogleCloudStorage(GoogleCloudStorageOptions, Storage, Credentials)
[Bug type REC_CATCH_EXCEPTION (click for details)](https://github.com/apache/beam/pull/33368#REC_CATCH_EXCEPTION)
In class org.apache.beam.sdk.extensions.gcp.util.GcsUtil
In method org.apache.beam.sdk.extensions.gcp.util.GcsUtil.createGoogleCloudStorage(GoogleCloudStorageOptions, Storage, Credentials)
At GcsUtil.java:[line 780]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support gcs-connector 3.x in GcsUtil [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on code in PR #33368:
URL: https://github.com/apache/beam/pull/33368#discussion_r1915192473
##########
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java:
##########
@@ -739,7 +741,47 @@ public WritableByteChannel create(GcsPath path, CreateOptions options) throws IO
GoogleCloudStorage createGoogleCloudStorage(
GoogleCloudStorageOptions options, Storage storage, Credentials credentials) {
- return new GoogleCloudStorageImpl(options, storage, credentials);
+ try {
+ return new GoogleCloudStorageImpl(options, storage, credentials);
+ } catch (NoSuchMethodError e) {
+ // gcs-connector 3.x drops the direct constructor and exclusively uses Builder
+ // TODO eliminate reflection once Beam drops Java 8 support and upgrades to gcsio 3.x
+ try {
+ final Method builderMethod = GoogleCloudStorageImpl.class.getMethod("builder");
+ Object builder = builderMethod.invoke(null);
+ final Class<?> builderClass =
+ Class.forName(
+ "com.google.cloud.hadoop.gcsio.AutoBuilder_GoogleCloudStorageImpl_Builder");
+
+ final Method setOptionsMethod =
+ builderClass.getMethod("setOptions", GoogleCloudStorageOptions.class);
+ setOptionsMethod.setAccessible(true);
+ builder = setOptionsMethod.invoke(builder, options);
+
+ final Method setHttpTransportMethod =
+ builderClass.getMethod("setHttpTransport", HttpTransport.class);
+ setHttpTransportMethod.setAccessible(true);
+ builder =
+ setHttpTransportMethod.invoke(builder, storage.getRequestFactory().getTransport());
+
+ final Method setCredentialsMethod =
+ builderClass.getMethod("setCredentials", Credentials.class);
+ setCredentialsMethod.setAccessible(true);
+ builder = setCredentialsMethod.invoke(builder, credentials);
+
+ final Method setHttpRequestInitializerMethod =
+ builderClass.getMethod("setHttpRequestInitializer", HttpRequestInitializer.class);
+ setHttpRequestInitializerMethod.setAccessible(true);
+ builder = setHttpRequestInitializerMethod.invoke(builder, httpRequestInitializer);
+
+ final Method buildMethod = builderClass.getMethod("build");
+ buildMethod.setAccessible(true);
+ return (GoogleCloudStorage) buildMethod.invoke(builder);
+ } catch (Exception reflectionError) {
+ throw new RuntimeException(
+ "Failed to construct GoogleCloudStorageImpl from gcsio 3.x Builder", e);
Review Comment:
Thanks for spotting that! Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2563869824
> looks like it is coded in test to prevent the exposure of unwanted classes. Refactor in a way that does not leak these may fix
thanks for looking into it @Abacn ! hmm, this seems challenging to refactor since the exposure is coming from `interface com.google.cloud.hadoop.gcsio.GoogleCloudStorage` itself... and there's no superclass/subclass that would be appropriate to substitute in here. I guess instead of adding a provider for a `GoogleCloudStorage` instance, we could add a provider for a `BiFunction<StorageResourceId, CreateObjectOptions, WritableByteChannel>` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support gcs-connector 3.x in GcsUtil [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on code in PR #33368:
URL: https://github.com/apache/beam/pull/33368#discussion_r1911124212
##########
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java:
##########
@@ -739,7 +741,42 @@ public WritableByteChannel create(GcsPath path, CreateOptions options) throws IO
GoogleCloudStorage createGoogleCloudStorage(
GoogleCloudStorageOptions options, Storage storage, Credentials credentials) {
- return new GoogleCloudStorageImpl(options, storage, credentials);
+ try {
+ // Attempt to construct gcs-connector 3.x-style GoogleCloudStorage, which is created
+ // exclusively via Builder method; this can be replaced once Java 8 is dropped and
Review Comment:
good idea- updated 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support gcs-connector 3.x in GcsUtil [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2590028309
that makes sense @Abacn ! Updated so that we try to the 2.x constructor first and attempt the 3.x Builder only if that throws a `NoSuchMethodError`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on code in PR #33368:
URL: https://github.com/apache/beam/pull/33368#discussion_r1882842440
##########
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.java:
##########
@@ -208,4 +222,13 @@ public PathValidator create(PipelineOptions options) {
.build();
}
}
+
+ @FunctionalInterface
+ public interface GoogleCloudStorageProvider {
+ GoogleCloudStorage get(
+ GoogleCloudStorageOptions options,
+ Storage storage,
+ Credentials credentials,
+ HttpRequestInitializer httpRequestInitializer);
Review Comment:
These 4 params should cover both the [2.x constructor](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.26/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageImpl.java#L302-L394) and the [3.x Builder](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.0.4/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageImpl.java#L2496-L2517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2558603917
> understand, thanks!
thanks @Abacn ! should I update CHANGES.md for the `2.62.0` release or the next one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2545689821
> Hi, thanks for the investigation. Is the builder constructor also supported on 2.x ? If so we can just change to use it in all case and no need extra options exposed to user
Unfortunately it isn't :/ There is no way to construct a `GoogleCloudStorageImpl` that works for both gcs-connector 2.x and 3.x (at least as far as I've been able to find).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support gcs-connector 3.x in GcsUtil [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #33368:
URL: https://github.com/apache/beam/pull/33368#discussion_r1911109001
##########
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java:
##########
@@ -739,7 +741,42 @@ public WritableByteChannel create(GcsPath path, CreateOptions options) throws IO
GoogleCloudStorage createGoogleCloudStorage(
GoogleCloudStorageOptions options, Storage storage, Credentials credentials) {
- return new GoogleCloudStorageImpl(options, storage, credentials);
+ try {
+ // Attempt to construct gcs-connector 3.x-style GoogleCloudStorage, which is created
+ // exclusively via Builder method; this can be replaced once Java 8 is dropped and
Review Comment:
could add a // TODO tag like
// TODO eliminate reflection once Beam drops Java 8 support and upgrade to use gcsio 3.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support gcs-connector 3.x in GcsUtil [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2588278435
Sorry for another late request
I've been a little bit worry about relying on the `try {} catch {}` to determine the code path, which happened to bring out performance regression in #33575
since using Reflection is already a hack, I do not see a good solution to this either. But could we try {return new GoogleCloudStorageImpl(...)} first? So in the default dependency versions (gcs 2.x), it won't throw Exception (even though it is handled) after this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support gcs-connector 3.x in GcsUtil [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #33368:
URL: https://github.com/apache/beam/pull/33368#discussion_r1915019335
##########
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java:
##########
@@ -739,7 +741,47 @@ public WritableByteChannel create(GcsPath path, CreateOptions options) throws IO
GoogleCloudStorage createGoogleCloudStorage(
GoogleCloudStorageOptions options, Storage storage, Credentials credentials) {
- return new GoogleCloudStorageImpl(options, storage, credentials);
+ try {
+ return new GoogleCloudStorageImpl(options, storage, credentials);
+ } catch (NoSuchMethodError e) {
+ // gcs-connector 3.x drops the direct constructor and exclusively uses Builder
+ // TODO eliminate reflection once Beam drops Java 8 support and upgrades to gcsio 3.x
+ try {
+ final Method builderMethod = GoogleCloudStorageImpl.class.getMethod("builder");
+ Object builder = builderMethod.invoke(null);
+ final Class<?> builderClass =
+ Class.forName(
+ "com.google.cloud.hadoop.gcsio.AutoBuilder_GoogleCloudStorageImpl_Builder");
+
+ final Method setOptionsMethod =
+ builderClass.getMethod("setOptions", GoogleCloudStorageOptions.class);
+ setOptionsMethod.setAccessible(true);
+ builder = setOptionsMethod.invoke(builder, options);
+
+ final Method setHttpTransportMethod =
+ builderClass.getMethod("setHttpTransport", HttpTransport.class);
+ setHttpTransportMethod.setAccessible(true);
+ builder =
+ setHttpTransportMethod.invoke(builder, storage.getRequestFactory().getTransport());
+
+ final Method setCredentialsMethod =
+ builderClass.getMethod("setCredentials", Credentials.class);
+ setCredentialsMethod.setAccessible(true);
+ builder = setCredentialsMethod.invoke(builder, credentials);
+
+ final Method setHttpRequestInitializerMethod =
+ builderClass.getMethod("setHttpRequestInitializer", HttpRequestInitializer.class);
+ setHttpRequestInitializerMethod.setAccessible(true);
+ builder = setHttpRequestInitializerMethod.invoke(builder, httpRequestInitializer);
+
+ final Method buildMethod = builderClass.getMethod("build");
+ buildMethod.setAccessible(true);
+ return (GoogleCloudStorage) buildMethod.invoke(builder);
+ } catch (Exception reflectionError) {
+ throw new RuntimeException(
+ "Failed to construct GoogleCloudStorageImpl from gcsio 3.x Builder", e);
Review Comment:
```suggestion
"Failed to construct GoogleCloudStorageImpl from gcsio 3.x Builder", reflectionError);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2583095096
> a last resort would be using reflection to handle both constructor or builder. Say, use a try to encapsulate the constictor call, when using 3.x dependency ut will throw an error, catch that Exception and construct the instance using Reflection. Is that possible to do?
I think that makes sense... so we'd keep the official Beam dep on 2.x. but try to reflectively construct a 3.x instance first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2559601868
java precommit tineout for several rerun. It's passing on HEAD, could you please taking a look if it is related to this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2582994173
a last resort would be using reflection to handle both constructor or builder. Say, use a try to encapsulate the constictor call, when using 3.x dependency ut will throw an error, catch that Exception and construct the instance using Reflection. Is that possible to do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2540105486
cc @Abacn - wdyt of this workaround for gcs-connector 3.x?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2563692445
after multiple rerun there is indeed a related test failure:
https://github.com/apache/beam/runs/34888346578
```
testGcpCoreApiSurface (org.apache.beam.sdk.extensions.gcp.GcpCoreApiSurfaceTest) failed
java.lang.AssertionError:
Expected: API surface to include only:
...
but: The following disallowed classes appeared on the API surface:
class com.google.protobuf.AbstractMessage exposed via:
...
interface org.apache.beam.sdk.extensions.gcp.options.GcsOptions$GoogleCloudStorageProvider
```
looks like it is coded in test to prevent the exposure of unwanted classes. Refactor in a way that does not leak these may fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support gcs-connector 3.x in GcsUtil [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2587469562
there is an [integration failure](https://github.com/apache/beam/actions/runs/12716146382/job/35449939696?pr=33368) but it doesn't look related:
```
org.apache.beam.sdk.io.gcp.bigquery.StorageApiSinkSchemaUpdateIT > testExactlyOnceWithAutoSchemaUpdate[1] FAILED
org.apache.beam.sdk.Pipeline$PipelineExecutionException at StorageApiSinkSchemaUpdateIT.java:431
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException at GoogleJsonResponseException.java:146
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2542558687
Hi, thanks for the investigation. Is the builder constructor also supported on 2.x ? If so we can just change to use it in all case and no need extra options exposed to user
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2582927125
> > looks like it is coded in test to prevent the exposure of unwanted classes. Refactor in a way that does not leak these may fix
>
> hmm, this seems challenging to refactor since the exposure is coming from `interface com.google.cloud.hadoop.gcsio.GoogleCloudStorage` itself... and there's no superclass/subclass that would be appropriate to substitute in here. I guess instead of adding a provider for a `GoogleCloudStorage` instance, we could add a provider for a `BiFunction<StorageResourceId, CreateObjectOptions, WritableByteChannel>` ? Although that would leak more implementation details to the user.
hey @Abacn ! just wanted to ping this thread before the PR goes stale :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support gcs-connector 3.x in GcsUtil [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on PR #33368:
URL: https://github.com/apache/beam/pull/33368#issuecomment-2583760962
seems to work! tested it on 2.x and 3.x 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support gcs-connector 3.x in GcsUtil [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #33368:
URL: https://github.com/apache/beam/pull/33368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org