You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Developer Developer <de...@gmail.com> on 2008/03/08 19:36:56 UTC
Setting nutch/hadopp multi node environment on a SAN device.
Hello Friends,
The tutorial @ http://wiki.apache.org/nutch/NutchHadoopTutorial says
don't use DFS on an NFS mount (this would be pretty stupid anyway)..
I am setting up multi node nutch/hadoop environment with lots of storage
available on a SAN device. I tried running nutch/hadoop with DFS file system
pointed to a directory on the SAN drive which was mounted as NFS . It failed
( expected ?).
How do I make it work ? In other words how do I make nutch/hadoop node
stores its content on a SAN drive directly ? Is it possible ? If not what
is the work around ? Do I have to make lots of local hard drive available
for crawling and then copy it to the NFS drive when the crawl completes ?
That would be very stupid solution !
Thanks !
Re: Setting nutch/hadopp multi node environment on a SAN device.
Posted by Developer Developer <de...@gmail.com>.
Hi folks,
any more comments from hadoop experts ?
On Sun, Mar 9, 2008 at 7:56 AM, Developer Developer <de...@gmail.com>
wrote:
> it is nfs mount
>
>
> On Sat, Mar 8, 2008 at 9:07 PM, Dennis Kubes <ku...@apache.org> wrote:
>
> > How is the san accessed, as a network drive, a special protocol?
> >
> > Dennis
> >
> > Developer Developer wrote:
> > > Any comments?
> > >
> > > On Sat, Mar 8, 2008 at 1:36 PM, Developer Developer <
> > devquestions@gmail.com>
> > > wrote:
> > >
> > >> Hello Friends,
> > >>
> > >>
> > >> The tutorial @ http://wiki.apache.org/nutch/NutchHadoopTutorial says
> > >> don't use DFS on an NFS mount (this would be pretty stupid anyway)..
> > >>
> > >> I am setting up multi node nutch/hadoop environment with lots of
> > storage
> > >> available on a SAN device. I tried running nutch/hadoop with DFS file
> > system
> > >> pointed to a directory on the SAN drive which was mounted as NFS . It
> > failed
> > >> ( expected ?).
> > >>
> > >> How do I make it work ? In other words how do I make nutch/hadoop
> > node
> > >> stores its content on a SAN drive directly ? Is it possible ? If not
> > what
> > >> is the work around ? Do I have to make lots of local hard drive
> > available
> > >> for crawling and then copy it to the NFS drive when the crawl
> > completes ?
> > >> That would be very stupid solution !
> > >>
> > >> Thanks !
> > >>
> > >>
> > >>
> > >
> >
>
>
Re: Setting nutch/hadopp multi node environment on a SAN device.
Posted by Developer Developer <de...@gmail.com>.
it is nfs mount
On Sat, Mar 8, 2008 at 9:07 PM, Dennis Kubes <ku...@apache.org> wrote:
> How is the san accessed, as a network drive, a special protocol?
>
> Dennis
>
> Developer Developer wrote:
> > Any comments?
> >
> > On Sat, Mar 8, 2008 at 1:36 PM, Developer Developer <
> devquestions@gmail.com>
> > wrote:
> >
> >> Hello Friends,
> >>
> >>
> >> The tutorial @ http://wiki.apache.org/nutch/NutchHadoopTutorial says
> >> don't use DFS on an NFS mount (this would be pretty stupid anyway)..
> >>
> >> I am setting up multi node nutch/hadoop environment with lots of
> storage
> >> available on a SAN device. I tried running nutch/hadoop with DFS file
> system
> >> pointed to a directory on the SAN drive which was mounted as NFS . It
> failed
> >> ( expected ?).
> >>
> >> How do I make it work ? In other words how do I make nutch/hadoop node
> >> stores its content on a SAN drive directly ? Is it possible ? If not
> what
> >> is the work around ? Do I have to make lots of local hard drive
> available
> >> for crawling and then copy it to the NFS drive when the crawl
> completes ?
> >> That would be very stupid solution !
> >>
> >> Thanks !
> >>
> >>
> >>
> >
>
Re: Setting nutch/hadopp multi node environment on a SAN device.
Posted by Dennis Kubes <ku...@apache.org>.
How is the san accessed, as a network drive, a special protocol?
Dennis
Developer Developer wrote:
> Any comments?
>
> On Sat, Mar 8, 2008 at 1:36 PM, Developer Developer <de...@gmail.com>
> wrote:
>
>> Hello Friends,
>>
>>
>> The tutorial @ http://wiki.apache.org/nutch/NutchHadoopTutorial says
>> don't use DFS on an NFS mount (this would be pretty stupid anyway)..
>>
>> I am setting up multi node nutch/hadoop environment with lots of storage
>> available on a SAN device. I tried running nutch/hadoop with DFS file system
>> pointed to a directory on the SAN drive which was mounted as NFS . It failed
>> ( expected ?).
>>
>> How do I make it work ? In other words how do I make nutch/hadoop node
>> stores its content on a SAN drive directly ? Is it possible ? If not what
>> is the work around ? Do I have to make lots of local hard drive available
>> for crawling and then copy it to the NFS drive when the crawl completes ?
>> That would be very stupid solution !
>>
>> Thanks !
>>
>>
>>
>
Re: Setting nutch/hadopp multi node environment on a SAN device.
Posted by Developer Developer <de...@gmail.com>.
Any comments?
On Sat, Mar 8, 2008 at 1:36 PM, Developer Developer <de...@gmail.com>
wrote:
> Hello Friends,
>
>
> The tutorial @ http://wiki.apache.org/nutch/NutchHadoopTutorial says
> don't use DFS on an NFS mount (this would be pretty stupid anyway)..
>
> I am setting up multi node nutch/hadoop environment with lots of storage
> available on a SAN device. I tried running nutch/hadoop with DFS file system
> pointed to a directory on the SAN drive which was mounted as NFS . It failed
> ( expected ?).
>
> How do I make it work ? In other words how do I make nutch/hadoop node
> stores its content on a SAN drive directly ? Is it possible ? If not what
> is the work around ? Do I have to make lots of local hard drive available
> for crawling and then copy it to the NFS drive when the crawl completes ?
> That would be very stupid solution !
>
> Thanks !
>
>
>