You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oodt.apache.org by YunHee Kang <yu...@gmail.com> on 2012/08/01 17:28:52 UTC
Question about metadata specification for Filemgr and Pushpull
Hi Chris and Brian,
I am reading source codes for handling "Catalog exception" related the
runtime error described below.
org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
Failure writing request
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(XmlRpcFileManagerClient.java:606)
at org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.java:284)
at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlreadyInDatabase(FileRetrievalSystem.java:254)
at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDownloadQueue(FileRetrievalSystem.java:463)
at org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:138)
at org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
at java.lang.Thread.run(Thread.java:662)
I think that the exception was caused by the wrong value of the
parameter productName of the method hasProduct() in the following
codelet of XmlRpcFileManagerClient.java:
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(XmlRpcFileManagerClient.java:606)
public boolean hasProduct(String productName) throws CatalogException {
Vector<Object> argList = new Vector<Object>();
argList.add(productName);
boolean hasProduct = false;
try {
hasProduct = ((Boolean) client.execute("filemgr.hasProduct",
argList)).booleanValue();
} catch (XmlRpcException e) {
throw new CatalogException(e.getMessage());
} catch (IOException e) {
throw new CatalogException(e.getMessage());
}
return hasProduct;
}
I inspected that the element "mine-type" described below was added
in the file mimestypes.xml that is one of pushpull property files.
<mime-type type="product/tes">
<_comment>ProductType=MyTesProductType</_comment>
<glob
pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5"
isregex="true"/>
</mime-type>
I would like to know what the element "mine-type" means.
I am wondering how I can check the value of productName in the
properties of Filemgr and Pushpull.
I am sorry to bother you again.
Thanks,
Yunhee
Re: Question about metadata specification for Filemgr and Pushpull
Posted by Brian Foster <ho...@me.com>.
Hey YunHee,
Okay, found another bug... lol... you're on a roll dude!
I've attached the fix to issue:
https://issues.apache.org/jira/browse/OODT-481
Just patch the fix into the trunk pushpull and download with communication to filemgr should work now.
-brian
On Aug 1, 2012, at 8:28 AM, YunHee Kang wrote:
> Hi Chris and Brian,
>
> I am reading source codes for handling "Catalog exception" related the
> runtime error described below.
> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> Failure writing request
> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(XmlRpcFileManagerClient.java:606)
> at org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.java:284)
> at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlreadyInDatabase(FileRetrievalSystem.java:254)
> at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDownloadQueue(FileRetrievalSystem.java:463)
> at org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:138)
> at org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
> at java.lang.Thread.run(Thread.java:662)
>
> I think that the exception was caused by the wrong value of the
> parameter productName of the method hasProduct() in the following
> codelet of XmlRpcFileManagerClient.java:
> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(XmlRpcFileManagerClient.java:606)
> public boolean hasProduct(String productName) throws CatalogException {
> Vector<Object> argList = new Vector<Object>();
> argList.add(productName);
>
> boolean hasProduct = false;
>
> try {
> hasProduct = ((Boolean) client.execute("filemgr.hasProduct",
> argList)).booleanValue();
> } catch (XmlRpcException e) {
> throw new CatalogException(e.getMessage());
> } catch (IOException e) {
> throw new CatalogException(e.getMessage());
> }
> return hasProduct;
> }
>
> I inspected that the element "mine-type" described below was added
> in the file mimestypes.xml that is one of pushpull property files.
> <mime-type type="product/tes">
> <_comment>ProductType=MyTesProductType</_comment>
> <glob
> pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5"
> isregex="true"/>
> </mime-type>
>
> I would like to know what the element "mine-type" means.
> I am wondering how I can check the value of productName in the
> properties of Filemgr and Pushpull.
>
> I am sorry to bother you again.
>
> Thanks,
> Yunhee
Re: Question about metadata specification for Filemgr and Pushpull
Posted by YunHee Kang <yu...@gmail.com>.
Hi Brian,
I overlooked the function of the CAS crawler Framework to ingest
download files into the CAS File Manager. I appreciate your
explanation about the relationship between Pushpull framework and
Crawler Framework. To be honest, I was confused why you only modified
two source programs in the patch. Now it helps to clearfy my
thinking.
Thanks,
Yunhee
2012/8/3 Brian Foster <ho...@me.com>:
> Hey YunHee,
>
> This is expected behavoir now... pushpull only downloads the files... it
> talks to the filemgr to try to determine if it already has a file before it
> redownloads it... if you want automated file ingest, you must set up a
> crawler to crawl your pushpull staging directory... there is documentation
> for that as well... http://oodt.apache.org/components/maven/crawler/user/
> ... the command line information is out of date there possibly, however just
> run: ./crawler_launcher --help ... the help menus are pretty involved... you
> can also learn how to customize your command-line options here:
> https://svn.apache.org/repos/asf/oodt/trunk/cli/README.txt
>
> -brian
>
>
> On Aug 02, 2012, at 10:08 AM, Kang YunHee <yu...@gmail.com> wrote:
>
> Hi Brain,
>
> I applied the patch(OODT-481.2012-08-01.txt) to the source of pushpull
> framework, rebuilt it and swapped its jar out in my deployment according
> to the following steps:
>
> patch -p0 < OODT-481.2012-08-01.txt
> mvn clean
> mvn install
> cd target
> cp cas-pushpull-0.5-SNAPSHOT.jar ~/oodt-0.5/cas-pushpull/lib/
>
>
>
>
> After I ran the pushpull script, I found out there is no "Catalog
> exception" as you can see:
> Aug 3, 2012 1:31:25 AM
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem
> addToDownloadQueue
> WARNING: Skipping file {parent = 'null', path =
> '/TES/TL2CO2N.005/2004.09.20/TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5',
> isDir = 'false'} because it is already in staging area
> PageSize: 8 PageLoc: 2
> FileList size: 2
> PageSize: 8 PageLoc: 952
> FileList size: 952
> Aug 3, 2012 1:35:38 AM
> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
> INFO: Disconnecting protocol
> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
> Aug 3, 2012 1:35:38 AM org.apache.oodt.cas.pushpull.daemon.Daemon$1 run
> INFO: Daemon with ID = 90121 on RMI registry port 9012 is shutting down
>
>
> But I am not sure that the file downloaded is ingested by my filemgr.
> I think if it was ingested by my filemgr, it would be in the repository
> directory.
> However I did not find it in my repository directory:
>
> Let me know how to check the status of cataloging mentioned above.
>
> Thanks,
> Yunhee
>
> On 8/2/12 6:29 AM, "Brian Foster" <ho...@me.com> wrote:
>
>>
>>hey YunHee,
>>
>>I've submitted the patch, so you can instead of having to patch the code
>>you can just resync your pushpull code and rebuild it and swap the jar
>>out in your deployment
>>
>>-brian
>>
>>On Aug 1, 2012, at 8:28 AM, YunHee Kang wrote:
>>
>>> Hi Chris and Brian,
>>>
>>> I am reading source codes for handling "Catalog exception" related the
>>> runtime error described below.
>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>> Failure writing request
>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(Xml
>>>RpcFileManagerClient.java:606)
>>> at
>>>org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.jav
>>>a:284)
>>> at
>>>org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlread
>>>yInDatabase(FileRetrievalSystem.java:254)
>>> at
>>>org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDow
>>>nloadQueue(FileRetrievalSystem.java:463)
>>> at
>>>org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFil
>>>e(RemoteCrawler.java:138)
>>> at
>>>org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles
>>>(RetrievalSetup.java:109)
>>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> I think that the exception was caused by the wrong value of the
>>> parameter productName of the method hasProduct() in the following
>>> codelet of XmlRpcFileManagerClient.java:
>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(Xml
>>>RpcFileManagerClient.java:606)
>>> public boolean hasProduct(String productName) throws CatalogException {
>>> Vector<Object> argList = new Vector<Object>();
>>> argList.add(productName);
>>>
>>> boolean hasProduct = false;
>>>
>>> try {
>>> hasProduct = ((Boolean) client.execute("filemgr.hasProduct",
>>> argList)).booleanValue();
>>> } catch (XmlRpcException e) {
>>> throw new CatalogException(e.getMessage());
>>> } catch (IOException e) {
>>> throw new CatalogException(e.getMessage());
>>> }
>>> return hasProduct;
>>> }
>>>
>>> I inspected that the element "mine-type" described below was added
>>> in the file mimestypes.xml that is one of pushpull property files.
>>> <mime-type type="product/tes">
>>> <_comment>ProductType=MyTesProductType</_comment>
>>> <glob
>>> pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5"
>>> isregex="true"/>
>>> </mime-type>
>>>
>>> I would like to know what the element "mine-type" means.
>>> I am wondering how I can check the value of productName in the
>>> properties of Filemgr and Pushpull.
>>>
>>> I am sorry to bother you again.
>>>
>>> Thanks,
>>> Yunhee
>>
>
>
Re: Question about metadata specification for Filemgr and Pushpull
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
+1 to Brian's comments below.
Cheers,
Chris
On Aug 2, 2012, at 11:59 AM, Brian Foster wrote:
> Hey YunHee,
>
> This is expected behavoir now... pushpull only downloads the files... it talks to the filemgr to try to determine if it already has a file before it redownloads it... if you want automated file ingest, you must set up a crawler to crawl your pushpull staging directory... there is documentation for that as well... http://oodt.apache.org/components/maven/crawler/user/ ... the command line information is out of date there possibly, however just run: ./crawler_launcher --help ... the help menus are pretty involved... you can also learn how to customize your command-line options here: https://svn.apache.org/repos/asf/oodt/trunk/cli/README.txt
>
> -brian
>
> On Aug 02, 2012, at 10:08 AM, Kang YunHee <yu...@gmail.com> wrote:
>
>> Hi Brain,
>>
>> I applied the patch(OODT-481.2012-08-01.txt) to the source of pushpull
>> framework, rebuilt it and swapped its jar out in my deployment according
>> to the following steps:
>>
>> patch -p0 < OODT-481.2012-08-01.txt
>> mvn clean
>> mvn install
>> cd target
>> cp cas-pushpull-0.5-SNAPSHOT.jar ~/oodt-0.5/cas-pushpull/lib/
>>
>>
>>
>>
>> After I ran the pushpull script, I found out there is no "Catalog
>> exception" as you can see:
>> Aug 3, 2012 1:31:25 AM
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem
>> addToDownloadQueue
>> WARNING: Skipping file {parent = 'null', path =
>> '/TES/TL2CO2N.005/2004.09.20/TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5',
>> isDir = 'false'} because it is already in staging area
>> PageSize: 8 PageLoc: 2
>> FileList size: 2
>> PageSize: 8 PageLoc: 952
>> FileList size: 952
>> Aug 3, 2012 1:35:38 AM
>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
>> INFO: Disconnecting protocol
>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
>> Aug 3, 2012 1:35:38 AM org.apache.oodt.cas.pushpull.daemon.Daemon$1 run
>> INFO: Daemon with ID = 90121 on RMI registry port 9012 is shutting down
>>
>>
>> But I am not sure that the file downloaded is ingested by my filemgr.
>> I think if it was ingested by my filemgr, it would be in the repository
>> directory.
>> However I did not find it in my repository directory:
>>
>> Let me know how to check the status of cataloging mentioned above.
>>
>> Thanks,
>> Yunhee
>>
>> On 8/2/12 6:29 AM, "Brian Foster" <ho...@me.com> wrote:
>>
>> >
>> >hey YunHee,
>> >
>> >I've submitted the patch, so you can instead of having to patch the code
>> >you can just resync your pushpull code and rebuild it and swap the jar
>> >out in your deployment
>> >
>> >-brian
>> >
>> >On Aug 1, 2012, at 8:28 AM, YunHee Kang wrote:
>> >
>> >> Hi Chris and Brian,
>> >>
>> >> I am reading source codes for handling "Catalog exception" related the
>> >> runtime error described below.
>> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> >> Failure writing request
>> >> at
>> >>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(Xml
>> >>RpcFileManagerClient.java:606)
>> >> at
>> >>org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.jav
>> >>a:284)
>> >> at
>> >>org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlread
>> >>yInDatabase(FileRetrievalSystem.java:254)
>> >> at
>> >>org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDow
>> >>nloadQueue(FileRetrievalSystem.java:463)
>> >> at
>> >>org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFil
>> >>e(RemoteCrawler.java:138)
>> >> at
>> >>org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles
>> >>(RetrievalSetup.java:109)
>> >> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>> >> at java.lang.Thread.run(Thread.java:662)
>> >>
>> >> I think that the exception was caused by the wrong value of the
>> >> parameter productName of the method hasProduct() in the following
>> >> codelet of XmlRpcFileManagerClient.java:
>> >> at
>> >>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(Xml
>> >>RpcFileManagerClient.java:606)
>> >> public boolean hasProduct(String productName) throws CatalogException {
>> >> Vector<Object> argList = new Vector<Object>();
>> >> argList.add(productName);
>> >>
>> >> boolean hasProduct = false;
>> >>
>> >> try {
>> >> hasProduct = ((Boolean) client.execute("filemgr.hasProduct",
>> >> argList)).booleanValue();
>> >> } catch (XmlRpcException e) {
>> >> throw new CatalogException(e.getMessage());
>> >> } catch (IOException e) {
>> >> throw new CatalogException(e.getMessage());
>> >> }
>> >> return hasProduct;
>> >> }
>> >>
>> >> I inspected that the element "mine-type" described below was added
>> >> in the file mimestypes.xml that is one of pushpull property files.
>> >> <mime-type type="product/tes">
>> >> <_comment>ProductType=MyTesProductType</_comment>
>> >> <glob
>> >> pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5"
>> >> isregex="true"/>
>> >> </mime-type>
>> >>
>> >> I would like to know what the element "mine-type" means.
>> >> I am wondering how I can check the value of productName in the
>> >> properties of Filemgr and Pushpull.
>> >>
>> >> I am sorry to bother you again.
>> >>
>> >> Thanks,
>> >> Yunhee
>> >
>>
>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Question about metadata specification for Filemgr and Pushpull
Posted by Brian Foster <ho...@me.com>.
Hey YunHee,
This is expected behavoir now... pushpull only downloads the files... it talks to the filemgr to try to determine if it already has a file before it redownloads it... if you want automated file ingest, you must set up a crawler to crawl your pushpull staging directory... there is documentation for that as well...�http://oodt.apache.org/components/maven/crawler/user/�... the command line information is out of date there possibly, however just run: ./crawler_launcher --help ... the help menus are pretty involved... you can also learn how to customize your command-line options here:�https://svn.apache.org/repos/asf/oodt/trunk/cli/README.txt
-brian
On Aug 02, 2012, at 10:08 AM, Kang YunHee <yu...@gmail.com> wrote:
Hi Brain,
I applied the patch(OODT-481.2012-08-01.txt) to the source of pushpull
framework, rebuilt it and swapped its jar out in my deployment according
to the following steps:
patch -p0 < OODT-481.2012-08-01.txt
mvn clean
mvn install
cd target
cp cas-pushpull-0.5-SNAPSHOT.jar ~/oodt-0.5/cas-pushpull/lib/
After I ran the pushpull script, I found out there is no "Catalog
exception" as you can see:
Aug 3, 2012 1:31:25 AM
org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem
addToDownloadQueue
WARNING: Skipping file {parent = 'null', path =
'/TES/TL2CO2N.005/2004.09.20/TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5',
isDir = 'false'} because it is already in staging area
PageSize: 8 PageLoc: 2
FileList size: 2
PageSize: 8 PageLoc: 952
FileList size: 952
Aug 3, 2012 1:35:38 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
INFO: Disconnecting protocol
org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
Aug 3, 2012 1:35:38 AM org.apache.oodt.cas.pushpull.daemon.Daemon$1 run
INFO: Daemon with ID = 90121 on RMI registry port 9012 is shutting down
But I am not sure that the file downloaded is ingested by my filemgr.
I think if it was ingested by my filemgr, it would be in the repository
directory.
However I did not find it in my repository directory:
Let me know how to check the status of cataloging mentioned above.
Thanks,
Yunhee
On 8/2/12 6:29 AM, "Brian Foster" <ho...@me.com> wrote:
>
>hey YunHee,
>
>I've submitted the patch, so you can instead of having to patch the code
>you can just resync your pushpull code and rebuild it and swap the jar
>out in your deployment
>
>-brian
>
>On Aug 1, 2012, at 8:28 AM, YunHee Kang wrote:
>
>> Hi Chris and Brian,
>>
>> I am reading source codes for handling "Catalog exception" related the
>> runtime error described below.
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Failure writing request
>> at
>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(Xml
>>RpcFileManagerClient.java:606)
>> at
>>org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.jav
>>a:284)
>> at
>>org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlread
>>yInDatabase(FileRetrievalSystem.java:254)
>> at
>>org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDow
>>nloadQueue(FileRetrievalSystem.java:463)
>> at
>>org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFil
>>e(RemoteCrawler.java:138)
>> at
>>org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles
>>(RetrievalSetup.java:109)
>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> I think that the exception was caused by the wrong value of the
>> parameter productName of the method hasProduct() in the following
>> codelet of XmlRpcFileManagerClient.java:
>> at
>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(Xml
>>RpcFileManagerClient.java:606)
>> public boolean hasProduct(String productName) throws CatalogException {
>> Vector<Object> argList = new Vector<Object>();
>> argList.add(productName);
>>
>> boolean hasProduct = false;
>>
>> try {
>> hasProduct = ((Boolean) client.execute("filemgr.hasProduct",
>> argList)).booleanValue();
>> } catch (XmlRpcException e) {
>> throw new CatalogException(e.getMessage());
>> } catch (IOException e) {
>> throw new CatalogException(e.getMessage());
>> }
>> return hasProduct;
>> }
>>
>> I inspected that the element "mine-type" described below was added
>> in the file mimestypes.xml that is one of pushpull property files.
>> <mime-type type="product/tes">
>> <_comment>ProductType=MyTesProductType</_comment>
>> <glob
>> pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5"
>> isregex="true"/>
>> </mime-type>
>>
>> I would like to know what the element "mine-type" means.
>> I am wondering how I can check the value of productName in the
>> properties of Filemgr and Pushpull.
>>
>> I am sorry to bother you again.
>>
>> Thanks,
>> Yunhee
>
Re: Question about metadata specification for Filemgr and Pushpull
Posted by Kang YunHee <yu...@gmail.com>.
Hi Brain,
I applied the patch(OODT-481.2012-08-01.txt) to the source of pushpull
framework, rebuilt it and swapped its jar out in my deployment according
to the following steps:
patch -p0 < OODT-481.2012-08-01.txt
mvn clean
mvn install
cd target
cp cas-pushpull-0.5-SNAPSHOT.jar ~/oodt-0.5/cas-pushpull/lib/
After I ran the pushpull script, I found out there is no "Catalog
exception" as you can see:
Aug 3, 2012 1:31:25 AM
org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem
addToDownloadQueue
WARNING: Skipping file {parent = 'null', path =
'/TES/TL2CO2N.005/2004.09.20/TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5',
isDir = 'false'} because it is already in staging area
PageSize: 8 PageLoc: 2
FileList size: 2
PageSize: 8 PageLoc: 952
FileList size: 952
Aug 3, 2012 1:35:38 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
INFO: Disconnecting protocol
org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
Aug 3, 2012 1:35:38 AM org.apache.oodt.cas.pushpull.daemon.Daemon$1 run
INFO: Daemon with ID = 90121 on RMI registry port 9012 is shutting down
But I am not sure that the file downloaded is ingested by my filemgr.
I think if it was ingested by my filemgr, it would be in the repository
directory.
However I did not find it in my repository directory:
Let me know how to check the status of cataloging mentioned above.
Thanks,
Yunhee
On 8/2/12 6:29 AM, "Brian Foster" <ho...@me.com> wrote:
>
>hey YunHee,
>
>I've submitted the patch, so you can instead of having to patch the code
>you can just resync your pushpull code and rebuild it and swap the jar
>out in your deployment
>
>-brian
>
>On Aug 1, 2012, at 8:28 AM, YunHee Kang wrote:
>
>> Hi Chris and Brian,
>>
>> I am reading source codes for handling "Catalog exception" related the
>> runtime error described below.
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Failure writing request
>> at
>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(Xml
>>RpcFileManagerClient.java:606)
>> at
>>org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.jav
>>a:284)
>> at
>>org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlread
>>yInDatabase(FileRetrievalSystem.java:254)
>> at
>>org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDow
>>nloadQueue(FileRetrievalSystem.java:463)
>> at
>>org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFil
>>e(RemoteCrawler.java:138)
>> at
>>org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles
>>(RetrievalSetup.java:109)
>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> I think that the exception was caused by the wrong value of the
>> parameter productName of the method hasProduct() in the following
>> codelet of XmlRpcFileManagerClient.java:
>> at
>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(Xml
>>RpcFileManagerClient.java:606)
>> public boolean hasProduct(String productName) throws CatalogException {
>> Vector<Object> argList = new Vector<Object>();
>> argList.add(productName);
>>
>> boolean hasProduct = false;
>>
>> try {
>> hasProduct = ((Boolean) client.execute("filemgr.hasProduct",
>> argList)).booleanValue();
>> } catch (XmlRpcException e) {
>> throw new CatalogException(e.getMessage());
>> } catch (IOException e) {
>> throw new CatalogException(e.getMessage());
>> }
>> return hasProduct;
>> }
>>
>> I inspected that the element "mine-type" described below was added
>> in the file mimestypes.xml that is one of pushpull property files.
>> <mime-type type="product/tes">
>> <_comment>ProductType=MyTesProductType</_comment>
>> <glob
>> pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5"
>> isregex="true"/>
>> </mime-type>
>>
>> I would like to know what the element "mine-type" means.
>> I am wondering how I can check the value of productName in the
>> properties of Filemgr and Pushpull.
>>
>> I am sorry to bother you again.
>>
>> Thanks,
>> Yunhee
>
Re: Question about metadata specification for Filemgr and Pushpull
Posted by Brian Foster <ho...@me.com>.
hey YunHee,
I've submitted the patch, so you can instead of having to patch the code you can just resync your pushpull code and rebuild it and swap the jar out in your deployment
-brian
On Aug 1, 2012, at 8:28 AM, YunHee Kang wrote:
> Hi Chris and Brian,
>
> I am reading source codes for handling "Catalog exception" related the
> runtime error described below.
> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> Failure writing request
> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(XmlRpcFileManagerClient.java:606)
> at org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.java:284)
> at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlreadyInDatabase(FileRetrievalSystem.java:254)
> at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDownloadQueue(FileRetrievalSystem.java:463)
> at org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:138)
> at org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
> at java.lang.Thread.run(Thread.java:662)
>
> I think that the exception was caused by the wrong value of the
> parameter productName of the method hasProduct() in the following
> codelet of XmlRpcFileManagerClient.java:
> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(XmlRpcFileManagerClient.java:606)
> public boolean hasProduct(String productName) throws CatalogException {
> Vector<Object> argList = new Vector<Object>();
> argList.add(productName);
>
> boolean hasProduct = false;
>
> try {
> hasProduct = ((Boolean) client.execute("filemgr.hasProduct",
> argList)).booleanValue();
> } catch (XmlRpcException e) {
> throw new CatalogException(e.getMessage());
> } catch (IOException e) {
> throw new CatalogException(e.getMessage());
> }
> return hasProduct;
> }
>
> I inspected that the element "mine-type" described below was added
> in the file mimestypes.xml that is one of pushpull property files.
> <mime-type type="product/tes">
> <_comment>ProductType=MyTesProductType</_comment>
> <glob
> pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5"
> isregex="true"/>
> </mime-type>
>
> I would like to know what the element "mine-type" means.
> I am wondering how I can check the value of productName in the
> properties of Filemgr and Pushpull.
>
> I am sorry to bother you again.
>
> Thanks,
> Yunhee