You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Mengying Wang <wa...@usc.edu> on 2014/09/19 21:22:57 UTC
No data files are downloaded using Apache OODT Pushpull
Dear Everyone,
I am trying to use the Apache OODT Pushpull to download some remote data
files to my local machine. According to the Apache OODT Pushpull user guide
(https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide),
I have completed all steps. However, when I run the command ./pushpull, no
data files are downloaded to my staging area after 15 minutes. I have used
FileZilla to login to the remove server and confirmed that there are some
data files. So I am not sure what is wrong with my configurations? Could
you please help me out? Many thanks!
Following are my running screenshot and configuration details:
$ ./pushpull
TRANSFER: org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
^C
*[CAS_PP_HOME]/etc/push_pull_framework.properties*
line
21 #external configuration files
22
org.apache.oodt.cas.pushpull.config.external.properties.files=[CAS_PP_HOME]/etc/default.properties
35 # ingester filemgr url
36 org.apache.oodt.cas.filemgr.url=http://localhost:9000
61 #protocolfactory specification for protocol types
62
org.apache.oodt.cas.pushpull.config.protocolfactory.info.files=[CAS_PP_HOME]/policy/ProtocolFactoryInfo.xml
69 #parser to retrievalmethod map
70
org.apache.oodt.cas.pushpull.config.parser.info.files=[CAS_PP_HOME]/policy/ParserToRetrievalMethodMap.xml
71
72 #unique metadata element info
73
org.apache.oodt.cas.pushpull.config.type.detection.file=[CAS_PP_HOME]/policy/mimetypes.xml
74
75 #directory below which all data file will be downloaded to
76
org.apache.oodt.cas.pushpull.data.files.base.staging.area=[CAS_PP_HOME]/staging
*[CAS_PP_HOME]/policy/mimetypes.xml*
<mime-type type="product/rtg_sst">
<sub-class-of type="ncep/model"/>
<_comment><![CDATA[ProductType=RTG_SST &&
UniqueElement=RetrievedFromLoc]]></_comment>
<glob pattern="rtg_sst_grb_0\.5\.\d{8}" isregex="true"/>
<glob pattern="sst\.\d{8}_rtgssthr_grb_0\.5\.grib2" isregex="true"/>
</mime-type>
*[CAS_PP_HOME]/etc/examples/ExternalSourcesFiles/ExternalSources.xml*
<sources>
<source host="polar.ncep.noaa.gov">
<login type="ftp" alias="PolarNcepNoaaFtp">
<username>anonymous</username>
<password>user@host.com</password>
</login>
</source>
</sources>
*[CAS_PP_HOME]/etc/examples/RemoteSpecsFiles/RemoteSpecs.xml*
<remoteSpecs>
<aliasSpecs>
<aliasSpec
file="[CAS_PP_HOME]/etc/examples/ExternalSources/ExternalSources.xml"/>
</aliasSpecs>
<daemons>
<daemon alias="PolarNcepNoaaFtp" active="no">
<runInfo firstRunDateTime="2007-11-27T00:00:00Z" period="3m"
epsilon="20s" runOnReboot="yes"/>
<propInfo
dir="/Users/AngelaWang/Documents/programs/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles">
<propFiles regExp="RTG_SST\.xml"
parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser"/>
</propInfo>
<dataInfo stagingArea="RtgSstFtp" queryElement="Filename"/>
</daemon>
</daemons>
</remoteSpecs>
*[CAS_PP_HOME]/etc/examples/DirStructXmlParserFiles/RTG_SST.xml*
<root>
<dirstruct starting_path="/history/sst">
<nodirs/>
<file name="rtg_sst_grb_0\.5\.\d{8}"/>
</dirstruct>
</root>
Thanks again!
Best,
Angela Wang
Re: No data files are downloaded using Apache OODT Pushpull
Posted by Chris Mattmann <ma...@apache.org>.
Hi Angela,
I didn't get your screenshot, it was likely stripped. Can you use
something like min.us and then paste the link?
Cheers,
Chris
-----Original Message-----
From: Mengying Wang <wa...@usc.edu>
Reply-To: <de...@oodt.apache.org>
Date: Friday, September 19, 2014 12:22 PM
To: <de...@oodt.apache.org>
Subject: No data files are downloaded using Apache OODT Pushpull
>Dear Everyone,
>
>I am trying to use the Apache OODT Pushpull to download some remote data
>files to my local machine. According to the Apache OODT Pushpull user
>guide
>(https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guid
>e),
>I have completed all steps. However, when I run the command ./pushpull, no
>data files are downloaded to my staging area after 15 minutes. I have used
>FileZilla to login to the remove server and confirmed that there are some
>data files. So I am not sure what is wrong with my configurations? Could
>you please help me out? Many thanks!
>
>Following are my running screenshot and configuration details:
>
>$ ./pushpull
>
>TRANSFER:
>org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
>
>^C
>
>*[CAS_PP_HOME]/etc/push_pull_framework.properties*
>line
>21 #external configuration files
>22
>org.apache.oodt.cas.pushpull.config.external.properties.files=[CAS_PP_HOME
>]/etc/default.properties
>
>35 # ingester filemgr url
>36 org.apache.oodt.cas.filemgr.url=http://localhost:9000
>
>61 #protocolfactory specification for protocol types
>62
>org.apache.oodt.cas.pushpull.config.protocolfactory.info.files=[CAS_PP_HOM
>E]/policy/ProtocolFactoryInfo.xml
>
>69 #parser to retrievalmethod map
>70
>org.apache.oodt.cas.pushpull.config.parser.info.files=[CAS_PP_HOME]/policy
>/ParserToRetrievalMethodMap.xml
>71
>72 #unique metadata element info
>73
>org.apache.oodt.cas.pushpull.config.type.detection.file=[CAS_PP_HOME]/poli
>cy/mimetypes.xml
>74
>75 #directory below which all data file will be downloaded to
>76
>org.apache.oodt.cas.pushpull.data.files.base.staging.area=[CAS_PP_HOME]/st
>aging
>
>
>*[CAS_PP_HOME]/policy/mimetypes.xml*
><mime-type type="product/rtg_sst">
> <sub-class-of type="ncep/model"/>
> <_comment><![CDATA[ProductType=RTG_SST &&
>UniqueElement=RetrievedFromLoc]]></_comment>
> <glob pattern="rtg_sst_grb_0\.5\.\d{8}" isregex="true"/>
> <glob pattern="sst\.\d{8}_rtgssthr_grb_0\.5\.grib2" isregex="true"/>
></mime-type>
>
>*[CAS_PP_HOME]/etc/examples/ExternalSourcesFiles/ExternalSources.xml*
><sources>
> <source host="polar.ncep.noaa.gov">
> <login type="ftp" alias="PolarNcepNoaaFtp">
> <username>anonymous</username>
> <password>user@host.com</password>
> </login>
> </source>
></sources>
>
>*[CAS_PP_HOME]/etc/examples/RemoteSpecsFiles/RemoteSpecs.xml*
><remoteSpecs>
> <aliasSpecs>
> <aliasSpec
>file="[CAS_PP_HOME]/etc/examples/ExternalSources/ExternalSources.xml"/>
> </aliasSpecs>
>
> <daemons>
> <daemon alias="PolarNcepNoaaFtp" active="no">
> <runInfo firstRunDateTime="2007-11-27T00:00:00Z" period="3m"
>epsilon="20s" runOnReboot="yes"/>
> <propInfo
>dir="/Users/AngelaWang/Documents/programs/oodt/cas-pushpull/etc/examples/D
>irStructXmlParserFiles">
> <propFiles regExp="RTG_SST\.xml"
>parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXml
>Parser"/>
> </propInfo>
> <dataInfo stagingArea="RtgSstFtp" queryElement="Filename"/>
> </daemon>
> </daemons>
></remoteSpecs>
>
>*[CAS_PP_HOME]/etc/examples/DirStructXmlParserFiles/RTG_SST.xml*
><root>
> <dirstruct starting_path="/history/sst">
> <nodirs/>
> <file name="rtg_sst_grb_0\.5\.\d{8}"/>
> </dirstruct>
></root>
>
>Thanks again!
>
>Best,
>Angela Wang