You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Olsen, Katherine" <KO...@scitor.com> on 2015/09/17 19:17:15 UTC
HDFS and Marathon
Hi,
I'm attempting to deploy a NiFi instance on Mesos using Apache Marathon. Currently whenever the NiFi instance is re-started, a new Mesos container is pulled down and serves as the run time environment. This is all good except that the new container has an empty flow file repository. My preferred solution to avoid this is to use a location on the distributed file system (HDFS) as the flow file repository location. However, when I attempted to change the configuration file setting in nifi.properties to write to HDFS as follows, it simply created a folder named hdfs:// with associated sub-directories inside the local Mesos container that is running NiFi.
nifi.flowfile.repository.directory = hdfs://ip_address:8020/nifi/flowfile_repository
Is it currently possible to get Nifi to be respectful of the hdfs naming convention and save its flow file repository to a distributed file system like HDFS? I'm using NiFi version 0.2.1, Mesos version 0.23.0, and Marathon version 0.10.0.
Thanks,
Kathy
CONFIDENTIALITY NOTICE (v1.0): This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain proprietary, confidential or privileged information or otherwise be protected by law. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please notify the sender and destroy all copies and the original message.
Re: HDFS and Marathon
Posted by Sean Busbey <bu...@cloudera.com>.
related: fuse over webhdfs: https://github.com/aw-altiscale/webhdfs
On Thu, Sep 17, 2015 at 8:04 PM, Tony Kurc <tr...@gmail.com> wrote:
> has anyone tried using fuse [1] to accomplish something like what Kathy was
> asking?
>
> [1] https://wiki.apache.org/hadoop/MountableHDFS
>
> On Thu, Sep 17, 2015 at 1:26 PM, Joe Witt <jo...@gmail.com> wrote:
>
>> Kathy,
>>
>> In short, no. But this is a good idea. Our content repository
>> implementations today expect access to a 'local' file system. That
>> local file system could in theory be something like an NFS mount but
>> we'd need to implement it differently to work with HDFS. This would
>> be a great JIRA to create if you're interested.
>>
>> Thanks
>> Joe
>>
>> On Thu, Sep 17, 2015 at 10:17 AM, Olsen, Katherine <KO...@scitor.com>
>> wrote:
>> > Hi,
>> >
>> >
>> >
>> > I'm attempting to deploy a NiFi instance on Mesos using Apache
>> Marathon. Currently whenever the NiFi instance is re-started, a new Mesos
>> container is pulled down and serves as the run time environment. This is
>> all good except that the new container has an empty flow file repository.
>> My preferred solution to avoid this is to use a location on the distributed
>> file system (HDFS) as the flow file repository location. However, when I
>> attempted to change the configuration file setting in nifi.properties to
>> write to HDFS as follows, it simply created a folder named hdfs:// with
>> associated sub-directories inside the local Mesos container that is running
>> NiFi.
>> >
>> >
>> >
>> > nifi.flowfile.repository.directory =
>> hdfs://ip_address:8020/nifi/flowfile_repository
>> >
>> >
>> >
>> > Is it currently possible to get Nifi to be respectful of the hdfs naming
>> convention and save its flow file repository to a distributed file system
>> like HDFS? I'm using NiFi version 0.2.1, Mesos version 0.23.0, and
>> Marathon version 0.10.0.
>> >
>> >
>> >
>> > Thanks,
>> >
>> > Kathy
>> >
>> > CONFIDENTIALITY NOTICE (v1.0): This e-mail message, including any
>> attachments, is for the sole use of the intended recipient(s) and may
>> contain proprietary, confidential or privileged information or otherwise be
>> protected by law. Any unauthorized review, use, disclosure or distribution
>> is prohibited. If you are not the intended recipient, please notify the
>> sender and destroy all copies and the original message.
>> >
>> >
>>
--
Sean
Re: HDFS and Marathon
Posted by Tony Kurc <tr...@gmail.com>.
has anyone tried using fuse [1] to accomplish something like what Kathy was
asking?
[1] https://wiki.apache.org/hadoop/MountableHDFS
On Thu, Sep 17, 2015 at 1:26 PM, Joe Witt <jo...@gmail.com> wrote:
> Kathy,
>
> In short, no. But this is a good idea. Our content repository
> implementations today expect access to a 'local' file system. That
> local file system could in theory be something like an NFS mount but
> we'd need to implement it differently to work with HDFS. This would
> be a great JIRA to create if you're interested.
>
> Thanks
> Joe
>
> On Thu, Sep 17, 2015 at 10:17 AM, Olsen, Katherine <KO...@scitor.com>
> wrote:
> > Hi,
> >
> >
> >
> > I'm attempting to deploy a NiFi instance on Mesos using Apache
> Marathon. Currently whenever the NiFi instance is re-started, a new Mesos
> container is pulled down and serves as the run time environment. This is
> all good except that the new container has an empty flow file repository.
> My preferred solution to avoid this is to use a location on the distributed
> file system (HDFS) as the flow file repository location. However, when I
> attempted to change the configuration file setting in nifi.properties to
> write to HDFS as follows, it simply created a folder named hdfs:// with
> associated sub-directories inside the local Mesos container that is running
> NiFi.
> >
> >
> >
> > nifi.flowfile.repository.directory =
> hdfs://ip_address:8020/nifi/flowfile_repository
> >
> >
> >
> > Is it currently possible to get Nifi to be respectful of the hdfs naming
> convention and save its flow file repository to a distributed file system
> like HDFS? I'm using NiFi version 0.2.1, Mesos version 0.23.0, and
> Marathon version 0.10.0.
> >
> >
> >
> > Thanks,
> >
> > Kathy
> >
> > CONFIDENTIALITY NOTICE (v1.0): This e-mail message, including any
> attachments, is for the sole use of the intended recipient(s) and may
> contain proprietary, confidential or privileged information or otherwise be
> protected by law. Any unauthorized review, use, disclosure or distribution
> is prohibited. If you are not the intended recipient, please notify the
> sender and destroy all copies and the original message.
> >
> >
>
Re: HDFS and Marathon
Posted by Joe Witt <jo...@gmail.com>.
Kathy,
In short, no. But this is a good idea. Our content repository
implementations today expect access to a 'local' file system. That
local file system could in theory be something like an NFS mount but
we'd need to implement it differently to work with HDFS. This would
be a great JIRA to create if you're interested.
Thanks
Joe
On Thu, Sep 17, 2015 at 10:17 AM, Olsen, Katherine <KO...@scitor.com> wrote:
> Hi,
>
>
>
> I'm attempting to deploy a NiFi instance on Mesos using Apache Marathon. Currently whenever the NiFi instance is re-started, a new Mesos container is pulled down and serves as the run time environment. This is all good except that the new container has an empty flow file repository. My preferred solution to avoid this is to use a location on the distributed file system (HDFS) as the flow file repository location. However, when I attempted to change the configuration file setting in nifi.properties to write to HDFS as follows, it simply created a folder named hdfs:// with associated sub-directories inside the local Mesos container that is running NiFi.
>
>
>
> nifi.flowfile.repository.directory = hdfs://ip_address:8020/nifi/flowfile_repository
>
>
>
> Is it currently possible to get Nifi to be respectful of the hdfs naming convention and save its flow file repository to a distributed file system like HDFS? I'm using NiFi version 0.2.1, Mesos version 0.23.0, and Marathon version 0.10.0.
>
>
>
> Thanks,
>
> Kathy
>
> CONFIDENTIALITY NOTICE (v1.0): This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain proprietary, confidential or privileged information or otherwise be protected by law. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please notify the sender and destroy all copies and the original message.
>
>