Re: filemgr ingestion

[Chris Mattmann <ma...@apache.org>]

Hi Etienne,

Thanks for your query. The default protocol used by the
RemoteDataTransfer is XML-RPC. It's a call-back mechanism
that I baked up myself with a configurable chunk size parameter.

My suggestion is rather than use that DataTransfer, I would leverage:

(1) a distributed filesystem like HadoopFS, Tachyon (from BDAS), GlusterFS,
or even NFS that logically mounts a set of local shared nothing disks as a
virtual global mount

(2) use the LocalDataTransfer in the File Manager, with #1 in place

The main reason for this is that it allows the people who work on
distributed
and reliable filesystems that solve those problems in a great way, it
allows 
us in the OODT community to take advantage of that work without having to
write
all the complex functionality ourselves. That way we just have
light-weight insulated
pieces/components in OODT that rely on the workhorses that do things right.

HTH!

Cheers,
Chris

------------------------
Chris Mattmann
chris.mattmann@gmail.com




-----Original Message-----
From: Etienne Koen <ko...@gmail.com>
Date: Wednesday, September 10, 2014 1:57 AM
To: Shakeh Khudikyan <Sh...@jpl.nasa.gov>
Cc: Chris Mattmann <ch...@gmail.com>
Subject: filemgr ingestion

>Hi Shakeh,
>What is the default protocol that filemgr uses for RemoteDataTransfer
>ingestion?
>
>Also, is the a way to specify the protocol?
>
>Cheers
>Etienne
>

Re: filemgr ingestion

["Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>]

No problem Etienne, and yes that's a fair statement! :)

Cheers man.

Chris


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++






-----Original Message-----
From: Etienne Koen <ko...@gmail.com>
Date: Wednesday, September 10, 2014 9:37 AM
To: Chris Mattmann <ma...@apache.org>
Cc: Shakeh Khudikyan <Sh...@jpl.nasa.gov>,
"dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: Re: filemgr ingestion

>Thanks for clearing that out Chris,
>I think that the capability of the filemgr should be considered as a test
>on its own and does not actually fall into the category which we are
>currently benchmarking for the baseline WAN/LAN performance of the tools.
>
>There should be a separate test section for archiving...
>Thanks for the help!
>Etienne 
>On Sep 10, 2014 6:15 PM, "Chris Mattmann" <ma...@apache.org> wrote:
>
>Hi Etienne,
>
>Thanks for your query. The default protocol used by the
>RemoteDataTransfer is XML-RPC. It's a call-back mechanism
>that I baked up myself with a configurable chunk size parameter.
>
>My suggestion is rather than use that DataTransfer, I would leverage:
>
>(1) a distributed filesystem like HadoopFS, Tachyon (from BDAS),
>GlusterFS,
>or even NFS that logically mounts a set of local shared nothing disks as a
>virtual global mount
>
>(2) use the LocalDataTransfer in the File Manager, with #1 in place
>
>The main reason for this is that it allows the people who work on
>distributed
>and reliable filesystems that solve those problems in a great way, it
>allows
>us in the OODT community to take advantage of that work without having to
>write
>all the complex functionality ourselves. That way we just have
>light-weight insulated
>pieces/components in OODT that rely on the workhorses that do things
>right.
>
>HTH!
>
>Cheers,
>Chris
>
>------------------------
>Chris Mattmann
>chris.mattmann@gmail.com
>
>
>
>
>-----Original Message-----
>From: Etienne Koen <ko...@gmail.com>
>Date: Wednesday, September 10, 2014 1:57 AM
>To: Shakeh Khudikyan <Sh...@jpl.nasa.gov>
>Cc: Chris Mattmann <ch...@gmail.com>
>Subject: filemgr ingestion
>
>>Hi Shakeh,
>>What is the default protocol that filemgr uses for RemoteDataTransfer
>>ingestion?
>>
>>Also, is the a way to specify the protocol?
>>
>>Cheers
>>Etienne
>>
>
>
>
>
>

Re: filemgr ingestion

[Etienne Koen <ko...@gmail.com>]

Thanks for clearing that out Chris,

I think that the capability of the filemgr should be considered as a test
on its own and does not actually fall into the category which we are
currently benchmarking for the baseline WAN/LAN performance of the tools.

There should be a separate test section for archiving...

Thanks for the help!

Etienne
On Sep 10, 2014 6:15 PM, "Chris Mattmann" <ma...@apache.org> wrote:

> Hi Etienne,
>
> Thanks for your query. The default protocol used by the
> RemoteDataTransfer is XML-RPC. It's a call-back mechanism
> that I baked up myself with a configurable chunk size parameter.
>
> My suggestion is rather than use that DataTransfer, I would leverage:
>
> (1) a distributed filesystem like HadoopFS, Tachyon (from BDAS), GlusterFS,
> or even NFS that logically mounts a set of local shared nothing disks as a
> virtual global mount
>
> (2) use the LocalDataTransfer in the File Manager, with #1 in place
>
> The main reason for this is that it allows the people who work on
> distributed
> and reliable filesystems that solve those problems in a great way, it
> allows
> us in the OODT community to take advantage of that work without having to
> write
> all the complex functionality ourselves. That way we just have
> light-weight insulated
> pieces/components in OODT that rely on the workhorses that do things right.
>
> HTH!
>
> Cheers,
> Chris
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Etienne Koen <ko...@gmail.com>
> Date: Wednesday, September 10, 2014 1:57 AM
> To: Shakeh Khudikyan <Sh...@jpl.nasa.gov>
> Cc: Chris Mattmann <ch...@gmail.com>
> Subject: filemgr ingestion
>
> >Hi Shakeh,
> >What is the default protocol that filemgr uses for RemoteDataTransfer
> >ingestion?
> >
> >Also, is the a way to specify the protocol?
> >
> >Cheers
> >Etienne
> >
>
>
>