Re: HDFS adapter

[Patrick Stuedi <ps...@gmail.com>]

Hi Laurent

Yes crail://localhost:9060/mypath would be the path you want to use in your
Spark program to access Crail. Did you check if your Crail deployment is up
and running and can be accessed using the crail hdfs client (./bin/crail
fs)? If the crail hdfs client works access via Spark should be a matter of
setting the extra class path.

Please also take a look at

https://incubator-crail.readthedocs.io/en/latest/config.html#configuration

for examples how to configure Crail on a single machine or on a cluster.

-Patrick


On Mon, Apr 19, 2021 at 6:18 PM Laurent d'Orazio <
laurent.dorazio@univ-rennes1.fr> wrote:

> Dear all,
> My deepest apologies for a probably naive question :(
> I want to test Crail instead of HDFS with a basic Spark test (in Python)
> reading a very small (3 lines of 3 columns) cvs file (textFile) and saving
> it in another file (with saveAsTextFile).
> I have thus changed the path from hdfs://localhost:9000/<mypath> to
> hdfs://localhost:9060/<mypath>
> I have the enclosed error.
> Please note that I tried using crail://localhost:9060/<mypath>, but in
> that case I have a Java.io.IOException: No FileSystem for Scheme : crail.
> Can you let me know what is my mistake?
> Thanks in advance.
> Best,
> Laurent
>