You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Udi Meiri (JIRA)" <ji...@apache.org> on 2018/10/03 01:22:00 UTC
[jira] [Commented] (BEAM-5614) Using gs:// paths without first
doing a "gcloud auth" gives an unhelpful error message
[ https://issues.apache.org/jira/browse/BEAM-5614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636321#comment-16636321 ]
Udi Meiri commented on BEAM-5614:
---------------------------------
Full output:
WARNING: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or
"API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/.
Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:145)
at org.apache.beam.examples.WordCount.runWordCount(WordCount.java:176)
at org.apache.beam.examples.WordCount.main(WordCount.java:192)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214)
... 5 more
Caused by: java.lang.IllegalArgumentException: DataflowRunner requires gcpTempLocation, but failed to retrieve a value from PipelineOptions
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:243)
... 10 more
Caused by: java.lang.IllegalArgumentException: Error constructing default value for gcpTempLocation: tempLocation is not a valid GCS path, gs://XXXXXXX/staging/.
at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:255)
at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:232)
at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:592)
at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:533)
at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:158)
at com.sun.proxy.$Proxy15.getGcpTempLocation(Unknown Source)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:241)
... 10 more
Caused by: java.lang.RuntimeException: Unable to verify that GCS bucket gs://XXXXXXX exists.
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible(GcsPathValidator.java:86)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported(GcsPathValidator.java:53)
at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:252)
... 16 more
Caused by: com.google.api.client.http.HttpResponseException: 400 Bad Request
{
"error" : "invalid_grant",
"error_description" : "Bad Request"
}
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1072)
at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:227)
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:179)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:165)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
at com.google.cloud.hadoop.util.ChainingHttpRequestInitializer.initialize(ChainingHttpRequestInitializer.java:52)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.ResilientOperation$AbstractGoogleClientRequestExecutor.call(ResilientOperation.java:163)
at com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:64)
at org.apache.beam.sdk.util.GcsUtil.getBucket(GcsUtil.java:474)
at org.apache.beam.sdk.util.GcsUtil.bucketAccessible(GcsUtil.java:461)
at org.apache.beam.sdk.util.GcsUtil.bucketAccessible(GcsUtil.java:434)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible(GcsPathValidator.java:83)
... 18 more
> Using gs:// paths without first doing a "gcloud auth" gives an unhelpful error message
> --------------------------------------------------------------------------------------
>
> Key: BEAM-5614
> URL: https://issues.apache.org/jira/browse/BEAM-5614
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Reporter: Udi Meiri
> Assignee: Chamikara Jayalath
> Priority: Major
>
> Users see an error like:
> java.lang.IllegalArgumentException: Error constructing default value for gcpTempLocation: tempLocation is not a valid GCS path, gs://bucket/path/.
> Also reported here: https://stackoverflow.com/questions/43026371/apache-beam-minimalwordcount-example-with-dataflow-runner-on-eclipse
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)