Using JClouds Blobstore with Google Dataproc

[Alexis Brenon <al...@adyoulike.com>]

Hi,

I'm trying to use jclouds blobstore API from a Spark job executed on Google
Dataproc.
Locally, I am able to pass credentials from the
`GOOGLE_APPLICATION_CREDENTIALS` environment variable.
However, on a Dataproc cluster, this variable does not exist (I don't know
exactly how Google Client library authenticates itself in this case). I
tried to instantiate a Blobstore, without providing any credentials, but
this doesn't work:

scala> val bs = ContextBuilder.newBuilder(new
GoogleCloudStorageApiMetadata()).build(classOf[BlobStoreContext]).getBlobStore
java.util.NoSuchElementException: property google-cloud-storage.identity
not present in properties: [jclouds.idempotent-methods,
jclouds.user-threads, jclouds.max-session-failures, oauth.endpoint,
jclouds.google-cloud-storage.operation-complete-timeout,
jclouds.google-cloud-storage.operation-complete-interval,
jclouds.oauth.jws-alg, jclouds.max-connection-reuse, jclouds.endpoint,
jclouds.connection-close-header, jclouds.scheduler-threads,
jclouds.build-version, jclouds.iso3166-codes, jclouds.retries-delay-start,
jclouds.so-timeout, jclouds.max-connections-per-host,
jclouds.max-connections-per-context, jclouds.strip-expect-header,
jclouds.api, jclouds.user-agent, jclouds.max-parallel-deletes,
jclouds.oauth.audience, jclouds.api-version, jclouds.payloads.pretty-print,
jclouds.output-socket-buffer-size, jclouds.connection-timeout,
jclouds.provider, jclouds.session-interval]
  at
org.jclouds.ContextBuilder.searchPropertiesForProviderScopedProperty(ContextBuilder.java:294)
  at org.jclouds.ContextBuilder.resolveProperties(ContextBuilder.java:414)
  at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:317)
  at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:615)
  at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:595)
  at org.jclouds.ContextBuilder.build(ContextBuilder.java:588)
  ... 49 elided

Has anybody managed to instantiate a GCS blobstore object in a Dataproc
cluster?

Kind regards,
Alexis.

-- 
[image: Adyoulike] <https://www.adyoulike.com/>
Alexis BRENON
Lead Data
<alexis.brenon> alexis.brenon@adyoulike.com
<https://twitter.com/adyoulike>  <https://www.facebook.com/adyoulike/>
<https://www.instagram.com/adyoulike/>
<https://www.linkedin.com/company/adyoulike/>
<https://adyoulike.com/>