You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by "Verma, Rishi (388J)" <Ri...@jpl.nasa.gov> on 2012/02/23 01:17:32 UTC
Overriding FileManager's transfer of a product upon ingestion
Hello all,
Wanted to ask a question about how can one ingest a product into CAS FileManager via command-line without having the product transferred by FileManager.
For example, the below command is used to ingest a product into a local FileManager:
./filemgr-client --url $FILEMGR_URL --operation --ingestProduct --productName `basename $PROD_PATH` --productStructure Flat --productTypeName $PRODUCT_TYPE --metadataFile file://$PROD_PATH.met --refs file://$PROD_PATH
The "--refs" argument specifies where the product is located on disk. However, after running the above ingest command, the product is transferred (by default) to $FILEMGR_HOME/files.
So two questions:
[1] How can we tell FileManager NOT to copy the file, but to instead leave the file where it is and only catalog the metadata?
[2] How could one override the default transfer location of $FILEMGR_HOME/files to a different location?
Thanks!
Rishi
Re: Overriding FileManager's transfer of a product upon ingestion
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Rishi,
On Feb 23, 2012, at 3:06 PM, Verma, Rishi (388J) wrote:
> Hi Chris,
>
> Appreciate the help!
>
> But, what value would I enter for repositoryPath (within product type
> policy) to ensure an ingested product's file location reference uses only
> the value specified by the '--refs' argument?
>
> I ask because there is interference between product type policy's
> repositoryPath variable and the argument attached to '--refs' when
> ingesting a product.
>
> Example:
> * '--refs' is passed the value: file:///usr/local/data/rawfiles/file1.raw
> * 'repositoryPath' within policy has value: file:///usr/local/data/rawfiles
> * Product's FileLocation reference ends up with value:
> '/usr/local/data/rawfiles/file1.raw/file1.raw
>
> As you can see, the product's final file location reference is invalid.
It's actually not invalid. It's controlled by the Versioner scheme, and I think
the one that ships by default (the Basic Versioner) uses the following
schema:
/ProductTypeRepoPath/Filename/Filename
To place a file in the archive.
Of course, this can be customized by changing the versioner.
For more information on Versioning, check out this SMC-IT 2009 paper
that describes some of the rationale behind them:
http://sunset.usc.edu/~mattmann/pubs/SMCIT09.pdf
The basic idea is that a Versioner is uniquely tied to a ProductType
1-to-1 mapping as shown here:
http://oodt.apache.org/components/maven/filemgr/development/developer.html
Versioners define the scheme for laying out a Product's files (defined by
its References) on a storage device. There are several versioners that
come with the File Manager, and a lot of times, projects write their own
(in many cases, trivially extending MetadataBaseFileVersioner and providing
a filePathSpec).
>
> It doesn't seem possible to leave 'repositoryPath' blank, so do you have
> any suggestions on what to do in a situation like this?
Use the InPlaceVersioner :), here:
http://s.apache.org/sj
Cheers,
Chris
>
> On 2/22/12 8:14 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov> wrote:
>
>> Hi Rishi,
>>
>> Use the --clientTransfer option and use the InPlaceDataTransferer. For
>> #2,
>> update the policy (repositoryPath) in the default product type policy to
>> not store
>> to $HOME/files.
>>
>> HTH!
>>
>> Cheers,
>> Chris
>>
>> On Feb 22, 2012, at 5:17 PM, Verma, Rishi (388J) wrote:
>>
>>> Hello all,
>>>
>>> Wanted to ask a question about how can one ingest a product into CAS
>>> FileManager via command-line without having the product transferred by
>>> FileManager.
>>>
>>> For example, the below command is used to ingest a product into a local
>>> FileManager:
>>> ./filemgr-client --url $FILEMGR_URL --operation --ingestProduct
>>> --productName `basename $PROD_PATH` --productStructure Flat
>>> --productTypeName $PRODUCT_TYPE --metadataFile file://$PROD_PATH.met
>>> --refs file://$PROD_PATH
>>>
>>> The "--refs" argument specifies where the product is located on disk.
>>> However, after running the above ingest command, the product is
>>> transferred (by default) to $FILEMGR_HOME/files.
>>>
>>> So two questions:
>>> [1] How can we tell FileManager NOT to copy the file, but to instead
>>> leave the file where it is and only catalog the metadata?
>>> [2] How could one override the default transfer location of
>>> $FILEMGR_HOME/files to a different location?
>>>
>>> Thanks!
>>> Rishi
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Overriding FileManager's transfer of a product upon ingestion
Posted by "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>.
Write a versioner that gives you your desired behavior. The versioner is the one creating the reference. Its a fairly simple extension point and was meant to satisfy these types of use cases.
>From the sound of it your versioner would simply copy the reference. Since you would still be using the in place transferer nothing would be moved.
--Paul
Sent from my iPhone
On Feb 23, 2012, at 7:06 PM, "Verma, Rishi (388J)" <Ri...@jpl.nasa.gov> wrote:
> Hi Chris,
>
> Appreciate the help!
>
> But, what value would I enter for repositoryPath (within product type
> policy) to ensure an ingested product's file location reference uses only
> the value specified by the '--refs' argument?
>
> I ask because there is interference between product type policy's
> repositoryPath variable and the argument attached to '--refs' when
> ingesting a product.
>
> Example:
> * '--refs' is passed the value: file:///usr/local/data/rawfiles/file1.raw
> * 'repositoryPath' within policy has value: file:///usr/local/data/rawfiles
> * Product's FileLocation reference ends up with value:
> '/usr/local/data/rawfiles/file1.raw/file1.raw
>
> As you can see, the product's final file location reference is invalid.
>
> It doesn't seem possible to leave 'repositoryPath' blank, so do you have
> any suggestions on what to do in a situation like this?
>
> Thanks.
> rishi
>
> On 2/22/12 8:14 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov> wrote:
>
>> Hi Rishi,
>>
>> Use the --clientTransfer option and use the InPlaceDataTransferer. For
>> #2,
>> update the policy (repositoryPath) in the default product type policy to
>> not store
>> to $HOME/files.
>>
>> HTH!
>>
>> Cheers,
>> Chris
>>
>> On Feb 22, 2012, at 5:17 PM, Verma, Rishi (388J) wrote:
>>
>>> Hello all,
>>>
>>> Wanted to ask a question about how can one ingest a product into CAS
>>> FileManager via command-line without having the product transferred by
>>> FileManager.
>>>
>>> For example, the below command is used to ingest a product into a local
>>> FileManager:
>>> ./filemgr-client --url $FILEMGR_URL --operation --ingestProduct
>>> --productName `basename $PROD_PATH` --productStructure Flat
>>> --productTypeName $PRODUCT_TYPE --metadataFile file://$PROD_PATH.met
>>> --refs file://$PROD_PATH
>>>
>>> The "--refs" argument specifies where the product is located on disk.
>>> However, after running the above ingest command, the product is
>>> transferred (by default) to $FILEMGR_HOME/files.
>>>
>>> So two questions:
>>> [1] How can we tell FileManager NOT to copy the file, but to instead
>>> leave the file where it is and only catalog the metadata?
>>> [2] How could one override the default transfer location of
>>> $FILEMGR_HOME/files to a different location?
>>>
>>> Thanks!
>>> Rishi
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
Re: Overriding FileManager's transfer of a product upon ingestion
Posted by "Verma, Rishi (388J)" <Ri...@jpl.nasa.gov>.
Hi Chris,
Appreciate the help!
But, what value would I enter for repositoryPath (within product type
policy) to ensure an ingested product's file location reference uses only
the value specified by the '--refs' argument?
I ask because there is interference between product type policy's
repositoryPath variable and the argument attached to '--refs' when
ingesting a product.
Example:
* '--refs' is passed the value: file:///usr/local/data/rawfiles/file1.raw
* 'repositoryPath' within policy has value: file:///usr/local/data/rawfiles
* Product's FileLocation reference ends up with value:
'/usr/local/data/rawfiles/file1.raw/file1.raw
As you can see, the product's final file location reference is invalid.
It doesn't seem possible to leave 'repositoryPath' blank, so do you have
any suggestions on what to do in a situation like this?
Thanks.
rishi
On 2/22/12 8:14 PM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hi Rishi,
>
>Use the --clientTransfer option and use the InPlaceDataTransferer. For
>#2,
>update the policy (repositoryPath) in the default product type policy to
>not store
>to $HOME/files.
>
>HTH!
>
>Cheers,
>Chris
>
>On Feb 22, 2012, at 5:17 PM, Verma, Rishi (388J) wrote:
>
>> Hello all,
>>
>> Wanted to ask a question about how can one ingest a product into CAS
>>FileManager via command-line without having the product transferred by
>>FileManager.
>>
>> For example, the below command is used to ingest a product into a local
>>FileManager:
>> ./filemgr-client --url $FILEMGR_URL --operation --ingestProduct
>>--productName `basename $PROD_PATH` --productStructure Flat
>>--productTypeName $PRODUCT_TYPE --metadataFile file://$PROD_PATH.met
>>--refs file://$PROD_PATH
>>
>> The "--refs" argument specifies where the product is located on disk.
>>However, after running the above ingest command, the product is
>>transferred (by default) to $FILEMGR_HOME/files.
>>
>> So two questions:
>> [1] How can we tell FileManager NOT to copy the file, but to instead
>>leave the file where it is and only catalog the metadata?
>> [2] How could one override the default transfer location of
>>$FILEMGR_HOME/files to a different location?
>>
>> Thanks!
>> Rishi
>
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Overriding FileManager's transfer of a product upon ingestion
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Rishi,
Use the --clientTransfer option and use the InPlaceDataTransferer. For #2,
update the policy (repositoryPath) in the default product type policy to not store
to $HOME/files.
HTH!
Cheers,
Chris
On Feb 22, 2012, at 5:17 PM, Verma, Rishi (388J) wrote:
> Hello all,
>
> Wanted to ask a question about how can one ingest a product into CAS FileManager via command-line without having the product transferred by FileManager.
>
> For example, the below command is used to ingest a product into a local FileManager:
> ./filemgr-client --url $FILEMGR_URL --operation --ingestProduct --productName `basename $PROD_PATH` --productStructure Flat --productTypeName $PRODUCT_TYPE --metadataFile file://$PROD_PATH.met --refs file://$PROD_PATH
>
> The "--refs" argument specifies where the product is located on disk. However, after running the above ingest command, the product is transferred (by default) to $FILEMGR_HOME/files.
>
> So two questions:
> [1] How can we tell FileManager NOT to copy the file, but to instead leave the file where it is and only catalog the metadata?
> [2] How could one override the default transfer location of $FILEMGR_HOME/files to a different location?
>
> Thanks!
> Rishi
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++