You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Akanksha Sharma B <ak...@ericsson.com> on 2018/05/24 07:24:25 UTC

writing to maprfs?

Hi All,


I have a pipeline; it writes to hdfs, using "org.apache.beam.sdk.io.hdfs" package.

I was hoping that it will work with maprfs as well. However I have been debugging for some days, with no success.


I do not provide hdfsConfiguration from command line, and instead use the configuration file /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml (please find it attached with the mail)

However I get the stack trace ("No FileSystem for scheme: maprfs"). Please find the detailed trace attached. I made sure that I have /opt/mapr/lib/maprfs-5.2.2-mapr.jar on classpath. I use spark submit to run the pipeline.


If you have already tried writing to maprfs from beam, ccould you please help?


Regards,

Akanksha

Re: writing to maprfs?

Posted by Alexey Romanenko <ar...@gmail.com>.
Hello Akanksha,

Thank you for reporting this! It looks that website misses the details how to use โ€œorg.apache.beam.sdk.io.hdfsโ€ with different FSs, so I created a Jira to improve this:
https://issues.apache.org/jira/browse/BEAM-4397 <https://issues.apache.org/jira/browse/BEAM-4397>

WBR,
Alexey


> On 24 May 2018, at 10:36, Akanksha Sharma B <ak...@ericsson.com> wrote:
> 
> Hi,
> 
> Answering my own question ๐Ÿ˜Š
> 
> writing to maprfs worked after I added following property to core-site.xml:-
> 
>     <property>
>         <name>fs.maprfs.impl</name>
>         <value>com.mapr.fs.MapRFileSystem</value>
>     </property>
> 
> 
> Regards,
> Akanksha
> 
> From: Akanksha Sharma B
> Sent: Thursday, May 24, 2018 9:24:25 AM
> To: user@beam.apache.org
> Subject: writing to maprfs?
>  
> Hi All,
> 
> I have a pipeline; it writes to hdfs, using "org.apache.beam.sdk.io.hdfs" package.
> I was hoping that it will work with maprfs as well. However I have been debugging for some days, with no success.
> 
> I do not provide hdfsConfiguration from command line, and instead use the configuration file /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml (please find it attached with the mail)
> However I get the stack trace ("No FileSystem for scheme: maprfs"). Please find the detailed trace attached. I made sure that I have/opt/mapr/lib/maprfs-5.2.2-mapr.jar on classpath. I use spark submit to run the pipeline.
> 
> If you have already tried writing to maprfs from beam, ccould you please help?
> 
> Regards,
> Akanksha

Re: writing to maprfs?

Posted by Akanksha Sharma B <ak...@ericsson.com>.
Hi,


Answering my own question ๐Ÿ˜Š


writing to maprfs worked after I added following property to core-site.xml:-


    <property>
        <name>fs.maprfs.impl</name>
        <value>com.mapr.fs.MapRFileSystem</value>
    </property>



Regards,

Akanksha


________________________________
From: Akanksha Sharma B
Sent: Thursday, May 24, 2018 9:24:25 AM
To: user@beam.apache.org
Subject: writing to maprfs?


Hi All,


I have a pipeline; it writes to hdfs, using "org.apache.beam.sdk.io.hdfs" package.

I was hoping that it will work with maprfs as well. However I have been debugging for some days, with no success.


I do not provide hdfsConfiguration from command line, and instead use the configuration file /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml (please find it attached with the mail)

However I get the stack trace ("No FileSystem for scheme: maprfs"). Please find the detailed trace attached. I made sure that I have /opt/mapr/lib/maprfs-5.2.2-mapr.jar on classpath. I use spark submit to run the pipeline.


If you have already tried writing to maprfs from beam, ccould you please help?


Regards,

Akanksha