How to use a custom filesystem provider?

[Jean-Philippe Martin <jp...@google.com.INVALID>]

The full source for my example is available on github
<https://github.com/jean-philippe-martin/SparkRepro>.

I'm using maven to depend on gcloud-java-nio
<https://mvnrepository.com/artifact/com.google.cloud/gcloud-java-nio/0.2.5>,
which provides a Java FileSystem for Google Cloud Storage, via "gs://"
URLs. My Spark project uses maven-shade-plugin to create one big jar with
all the source in it.

The big jar correctly includes a
META-INF/services/java.nio.file.spi.FileSystemProviderfile, containing the
correct name for the class (
com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider). I
checked and that class is also correctly included in the jar file.

The program uses FileSystemProvider.installedProviders() to list the
filesystem providers it finds. "gs" should be listed (and it is if I run
the same function in a non-Spark context), but when running with Spark on
Dataproc, that provider's gone.

I'd like to know: *How can I use a custom filesystem in my Spark program*?
(asked earlier
<http://stackoverflow.com/questions/39500445/filesystem-provider-disappearing-in-spark>
in Stackoverflow but I didn't get any traction there)

Re: How to use a custom filesystem provider?

[Jean-Philippe Martin <jp...@google.com.INVALID>]

>
> There's a bit of confusion setting in here; the FileSystem implementations
> spark uses are subclasses of org.apache.hadoop.fs.FileSystem; the nio
> class with the same name is different.
> grab the google cloud storage connector and put it on your classpath


I was using the gs:// filesystem as an example. I should have mentioned
that I'm aware of the workaround for that one.

I'm not asking how to read from Google Cloud Storage from Spark.

What I'm interested in is Java's built-in extension mechanism for its
"Path" objects, aka custom filesystem providers
<http://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/filesystemprovider.html>.
What if I want to use my own different custom filesystem provider?
Something that allow me to take a funky-looking string like "foo://bar/baz"
and open it like a regular file, even though this results in a TCP
connection to the bar server and ask it to give me the "baz" file out of
its holographic quantum entangled storage (or other unspecified future
technology that can provide file-like objects).

Re: How to use a custom filesystem provider?

[Steve Loughran <st...@hortonworks.com>]

On 21 Sep 2016, at 20:10, Jean-Philippe Martin <jp...@google.com.INVALID>> wrote:


The full source for my example is available on github<https://github.com/jean-philippe-martin/SparkRepro>.

I'm using maven to depend on gcloud-java-nio<https://mvnrepository.com/artifact/com.google.cloud/gcloud-java-nio/0.2.5>, which provides a Java FileSystem for Google Cloud Storage, via "gs://" URLs. My Spark project uses maven-shade-plugin to create one big jar with all the source in it.

The big jar correctly includes a META-INF/services/java.nio.file.spi.FileSystemProviderfile, containing the correct name for the class (com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider). I checked and that class is also correctly included in the jar file.

The program uses FileSystemProvider.installedProviders() to list the filesystem providers it finds. "gs" should be listed (and it is if I run the same function in a non-Spark context), but when running with Spark on Dataproc, that provider's gone.

I'd like to know: How can I use a custom filesystem in my Spark program?



There's a bit of confusion setting in here; the FileSystem implementations spark uses are subclasses of org.apache.hadoop.fs.FileSystem; the nio class with the same name is different.

grab the google cloud storage connector and put it on your classpath

https://cloud.google.com/hadoop/google-cloud-storage-connector
https://github.com/GoogleCloudPlatform/bigdata-interop