You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by "Mallder, Valerie" <Va...@jhuapl.edu> on 2015/09/29 18:51:02 UTC
file staging question
Hi All,
I have a question related to the file staging feature in a PGE Task. Basically, how are you using this feature and what exactly does it do for you?
Does it just move some files from one place to another? If so, does the destination directory have to exist or does it get created? And, does the destination directory structure need to be the same as the source directory structure?
When does the 'staging' take place? Does it execute before the <exe> section of the PGE?
Thanks,
Val
P.S. I have read every line of ever file in OODT that contains the word 'staging' and it is still not clear to me what this feature actually does or how I can configure my PGE config file to use it. But, I did find some totally awesome documentation in the curator J But, it made it sound like the directory you set up as your 'staging' directory is the directory that files get ingested from. Whereas, I was under the impression that the directory you specify as your <output> directory is the directory that files get ingested from. So, it was confusing. L Wonderfully thorough documentation, just confusing for my case.
Valerie A. Mallder
New Horizons Deputy Mission System Engineer
The Johns Hopkins University/Applied Physics Laboratory
11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
240-228-7846 (Office) 410-504-2233 (Blackberry)
RE: file staging question
Posted by "Mallder, Valerie" <Va...@jhuapl.edu>.
Awesome. Thanks Chris. I'm actually already using the cp command in my exe block to do something similar. I had been wondering if file staging was a better (or more elegant) way to do what I needed to do. But, I will just continue doing what I'm doing :) Thanks!
Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory
> -----Original Message-----
> From: Chris Mattmann [mailto:chris.mattmann@gmail.com]
> Sent: Tuesday, September 29, 2015 12:59 PM
> To: dev@oodt.apache.org
> Subject: Re: file staging question
>
> No problem Val.
>
> The fileStaging feature was sort of half baked, however it does work and here is
> the basic concept behind it.
> If you have files in your CAS-PGE job that need to be copied from controlled
> access storage and staged into the job directory for the running CAS-PGE job, the
> fileStager is the way to do it. You give it a FM query that defines what files to
> select, and then it uses the File Manager configured data transferer to copy the
> file(s) from controlled access storage to the PGE job working dir.
>
> Another way to accomplish this is simply in your CAS-PGE exe block and cmd
> blocks is to add a cp or remote staging command (rsync, scp, etc.) that
> accomplishes the same thing.
>
> Cheers,
> Chris
>
> —
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
>
>
> -----Original Message-----
> From: "Mallder, Valerie" <Va...@jhuapl.edu>
> Reply-To: <de...@oodt.apache.org>
> Date: Tuesday, September 29, 2015 at 9:51 AM
> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: file staging question
>
> >Hi All,
> >
> >I have a question related to the file staging feature in a PGE Task.
> >Basically, how are you using this feature and what exactly does it do
> >for you?
> >
> >Does it just move some files from one place to another? If so, does
> >the destination directory have to exist or does it get created? And,
> >does the destination directory structure need to be the same as the
> >source directory structure?
> >
> >When does the 'staging' take place? Does it execute before the <exe>
> >section of the PGE?
> >
> >
> >Thanks,
> >Val
> >
> >P.S. I have read every line of ever file in OODT that contains the word
> >'staging' and it is still not clear to me what this feature actually does
> >or how I can configure my PGE config file to use it. But, I did find
> >some totally awesome documentation in the curator J But, it made it
> >sound like the directory you set up as your 'staging' directory is the
> >directory that files get ingested from. Whereas, I was under the
> >impression that the directory you specify as your <output> directory is
> >the directory that files get ingested from. So, it was confusing. L
> >Wonderfully thorough documentation, just confusing for my case.
> >
> >
> >Valerie A. Mallder
> >
> >New Horizons Deputy Mission System Engineer The Johns Hopkins
> >University/Applied Physics Laboratory
> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
> >240-228-7846 (Office) 410-504-2233 (Blackberry)
> >
>
Re: file staging question
Posted by Chris Mattmann <ch...@gmail.com>.
No problem Val.
The fileStaging feature was sort of half baked, however
it does work and here is the basic concept behind it.
If you have files in your CAS-PGE job that need to be
copied from controlled access storage and staged into the
job directory for the running CAS-PGE job, the fileStager
is the way to do it. You give it a FM query that defines
what files to select, and then it uses the File Manager
configured data transferer to copy the file(s) from controlled
access storage to the PGE job working dir.
Another way to accomplish this is simply in your CAS-PGE
exe block and cmd blocks is to add a cp or remote staging
command (rsync, scp, etc.) that accomplishes the same thing.
Cheers,
Chris
—
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: "Mallder, Valerie" <Va...@jhuapl.edu>
Reply-To: <de...@oodt.apache.org>
Date: Tuesday, September 29, 2015 at 9:51 AM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: file staging question
>Hi All,
>
>I have a question related to the file staging feature in a PGE Task.
>Basically, how are you using this feature and what exactly does it do for
>you?
>
>Does it just move some files from one place to another? If so, does the
>destination directory have to exist or does it get created? And, does
>the destination directory structure need to be the same as the source
>directory structure?
>
>When does the 'staging' take place? Does it execute before the <exe>
>section of the PGE?
>
>
>Thanks,
>Val
>
>P.S. I have read every line of ever file in OODT that contains the word
>'staging' and it is still not clear to me what this feature actually does
>or how I can configure my PGE config file to use it. But, I did find
>some totally awesome documentation in the curator J But, it made it
>sound like the directory you set up as your 'staging' directory is the
>directory that files get ingested from. Whereas, I was under the
>impression that the directory you specify as your <output> directory is
>the directory that files get ingested from. So, it was confusing. L
>Wonderfully thorough documentation, just confusing for my case.
>
>
>Valerie A. Mallder
>
>New Horizons Deputy Mission System Engineer
>The Johns Hopkins University/Applied Physics Laboratory
>11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
>240-228-7846 (Office) 410-504-2233 (Blackberry)
>