You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by "Mistry, Chintu (GSFC-586.0)[SGT INC]" <ch...@nasa.gov> on 2013/03/28 13:47:12 UTC
pushPull component problem
Hi,
I have been playing around with push pull for last two days and trying to download some data from remote location without success.
I would appreciate if you guys can shed some light on what is going on and why isn't push pull find files to download even though its clearly there at remote location.
Here is my configuration. I am trying to download /pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine and it lists all the files as well. However for some reason it does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I am doing wrong. I have tried almost all combinations and still could not get it to work.
In my /etc/pushpull.properties file, I have types=false so that it downloads the file even if its not specified in mime-types.xml file.
org.apache.oodt.cas.pushpull.allow.only.defined.types=false
ExternalSources.xml
<sources>
<source host="eclipse.ncdc.noaa.gov">
<login type="ftp" alias="eclipseClass">
<username>anonymous</username>
<password>user@host.com</password>
</login>
</source>
</sources>
RemoteSpecs.xml
<daemon alias="eclipseClass" active="yes">
<runInfo firstRunDateTime="2013-03-25T00:00:00Z" period="1m" runOnReboot="yes"/>
<propInfo dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
<propFiles regExp="TEST_AVHRR_ONLY\.xml" parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser"/>
</propInfo>
<dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no" queryElement="Filename"/>
</daemon>
TEST_AVHRR_ONLY.xml
<root>
<dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
<nodirs/>
<!-- <file name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
<file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
</dirstruct>
</root>
Regards
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
Re: pushPull component problem
Posted by "Verma, Rishi (388J)" <Ri...@jpl.nasa.gov>.
Hey Chintu,
I took a look at your output file, and it seems you must still be using a RegEx somewhere?
PushPull seems to be scanning through a bunch of files, not only one:
---
FileList size: 86
Filename : [avhrr-only-v2.20130101.nc.gz]
Filename : [avhrr-only-v2.20130102.nc.gz]
Filename : [avhrr-only-v2.20130103.nc.gz]
Filename : [avhrr-only-v2.20130104.nc.gz]
Filename : [avhrr-only-v2.20130105.nc.gz]
---
Can you try the following? Specify an exact filename within TEST_AVHRR_ONLY.xml (you may have already done this), and specify the exact same filename within mime-types.xml. Make sure types=true wtihin pushpull.properties and that the alias for your product type matches the type name within mime-types.xml. You should see PushPull trying to scan for only one file.
Another thing you can try is to recreating the tutorial example from [1] and slowly adjusting it to your desired FTP server and product details. I recently set up PushPull using this method, and got it working.
Rishi
--
[1] https://cwiki.apache.org/OODT/oodt-push-pull-user-guide.html
On Mar 28, 2013, at 7:12 AM, Mistry, Chintu (GSFC-586.0)[SGT INC] wrote:
I have attached verbose log. I have added extra print statements and compiled to see what is returned from remote site. Hope this helps.
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>>
Date: Thursday, March 28, 2013 10:05 AM
To: Chintu Mistry <ch...@nasa.gov>>, "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>>
Subject: Re: pushPull component problem
Hey Chintu,
Thanks for sending this — really helped verify it's something else.
Can you capture the log of push pull, with verbose logging turned on, and maybe do a pastebin?
Cheers,
Chris
From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>>
Date: Thursday, March 28, 2013 7:03 AM
To: jpluser <ch...@jpl.nasa.gov>>, "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>>
Subject: Re: pushPull component problem
Something else is wrong. I know that the regex is fine. I don't even have regex, it is straight up file name. It should match.
Thanks for looking into this.
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>>
Date: Thursday, March 28, 2013 9:01 AM
To: Chintu Mistry <ch...@nasa.gov>>, "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>>
Subject: Re: pushPull component problem
Hey Chintu,
Sorry that you have been having trouble and thanks for pasting the relevant portions
of push pull config to diagnose this.
I think you have an issue in your RemoteFileSpecs file (TEST_AVHRR_ONLY.xml) --
one quick way to debug this is to use the tool that Brian Foster wrote:
(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
$ cd bin
$ java –Djava.ext.dirs=../lib org.apache.oodt.cas.pushpull.util.ExpressionValidator
It's a simple GUI that will allow you to put in regular expressions from your push pull
RmoteFileSpecs files, and then put in file names from the server to see if they match.
Can you run the above and let me know if it's matching? If not, we've found your issue.
If it is matching, we'll move on to the next set of things to check.
Thanks!
Cheers,
Chris
From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>>
Date: Thursday, March 28, 2013 5:47 AM
To: "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: jpluser <ch...@jpl.nasa.gov>>, "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>>
Subject: pushPull component problem
Hi,
I have been playing around with push pull for last two days and trying to download some data from remote location without success.
I would appreciate if you guys can shed some light on what is going on and why isn't push pull find files to download even though its clearly there at remote location.
Here is my configuration. I am trying to download /pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz file from eclipse.ncdc.noaa.gov<http://eclipse.ncdc.noaa.gov>. It logs-in to the server fine and it lists all the files as well. However for some reason it does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I am doing wrong. I have tried almost all combinations and still could not get it to work.
In my /etc/pushpull.properties file, I have types=false so that it downloads the file even if its not specified in mime-types.xml file.
org.apache.oodt.cas.pushpull.allow.only.defined.types=false
ExternalSources.xml
<sources>
<source host="eclipse.ncdc.noaa.gov<http://eclipse.ncdc.noaa.gov>">
<login type="ftp" alias="eclipseClass">
<username>anonymous</username>
<pa...@host.com></password>
</login>
</source>
</sources>
RemoteSpecs.xml
<daemon alias="eclipseClass" active="yes">
<runInfo firstRunDateTime="2013-03-25T00:00:00Z" period="1m" runOnReboot="yes"/>
<propInfo dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
<propFiles regExp="TEST_AVHRR_ONLY\.xml" parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser"/>
</propInfo>
<dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no" queryElement="Filename"/>
</daemon>
TEST_AVHRR_ONLY.xml
<root>
<dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
<nodirs/>
<!-- <file name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
<file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
</dirstruct>
</root>
Regards
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
<typescript.txt>
Re: pushPull component problem
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Chintu,
Looking at jVFTP doesn't look like it barfed, looks like it ran,
but just didn't match any files. How about a regex that matches all files,
or something else similar?
The other option I would do is try and run one of the sample configs
(e.g., for MODIS.xml)
and see if those work?
Also try Rishi's suggestions too and let me know what you think.
Thanks.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
Date: Thursday, March 28, 2013 9:28 AM
To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
<de...@oodt.apache.org>
Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
Subject: Re: pushPull component problem
>Sorry about not explaining properly. When I said "failed to run", I meant
>class not found. Did not really look deep enough what the problem is.
>Apparently I did not include all jar files properly.
>
>Anyway, I managed to run both ftp client (of course I did change
>ProtocolFactor.xml file). But still no luck. Attaching files with output.
>
>Ftp4CheFtp => Completely barfed
>jvFTP => runs fine but same output as the other plugins.
>
>
>
>Regards
>--
>Chintu Mistry
>NASA Goddard Space Flight Center
>Bldg L40B, Room S776
>Office: 240 684 0477
>Mobile: 770 310 1047
>
>
>
>
>
>
>On 3/28/13 12:13 PM, "Mattmann, Chris A" <ch...@jpl.nasa.gov>
>wrote:
>
>>Hey Chintu,
>>
>>Thanks -- sorry to be asking all these questions, but what does "failed
>>to
>>run" mean?
>>
>>Did you update the protocol plugins config to ref the new plugins and
>>their FQCN (fully
>>qualified class names?) Do you have a log file for that?
>>
>>Cheers,
>>Chris
>>
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>Chris Mattmann, Ph.D.
>>Senior Computer Scientist
>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>Office: 171-266B, Mailstop: 171-246
>>Email: chris.a.mattmann@nasa.gov
>>WWW: http://sunset.usc.edu/~mattmann/
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>Adjunct Assistant Professor, Computer Science Department
>>University of Southern California, Los Angeles, CA 90089 USA
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>-----Original Message-----
>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
>>Date: Thursday, March 28, 2013 9:12 AM
>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>><de...@oodt.apache.org>
>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>Subject: Re: pushPull component problem
>>
>>>Since we are not using OODT 0.6, it did not compile straight away.
>>>However, I did manage to compile it with "0.5" version of
>>>cas-protocol-api. But when I was running it, It failed to run.
>>>
>>>--
>>>Chintu Mistry
>>>NASA Goddard Space Flight Center
>>>Bldg L40B, Room S776
>>>Office: 240 684 0477
>>>Mobile: 770 310 1047
>>>
>>>
>>>
>>>
>>>
>>>
>>>On 3/28/13 12:04 PM, "Mattmann, Chris A" <ch...@jpl.nasa.gov>
>>>wrote:
>>>
>>>>Hey Chintu,
>>>>
>>>>Thanks for identifying this!
>>>>
>>>>Have you tried the alternative plugins? They use a diff set of FTP
>>>>protocol
>>>>plugins (that unfortunately are not ALv2 licensed or Category A
>>>>approved,
>>>>so we can't ship with Apache OODT).
>>>>
>>>>They may do better at handling that symlink than the default version of
>>>>commons-ftp
>>>>we're using.
>>>>
>>>>Let me know. We'll investigate.
>>>>
>>>>Cheers,
>>>>Chris
>>>>
>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>Chris Mattmann, Ph.D.
>>>>Senior Computer Scientist
>>>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>Office: 171-266B, Mailstop: 171-246
>>>>Email: chris.a.mattmann@nasa.gov
>>>>WWW: http://sunset.usc.edu/~mattmann/
>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>Adjunct Assistant Professor, Computer Science Department
>>>>University of Southern California, Los Angeles, CA 90089 USA
>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>>>
>>>>
>>>>
>>>>-----Original Message-----
>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>><ch...@nasa.gov>
>>>>Date: Thursday, March 28, 2013 9:02 AM
>>>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>>>><de...@oodt.apache.org>
>>>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>>>Subject: Re: pushPull component problem
>>>>
>>>>>Chris,
>>>>>
>>>>>I figured out the problem. Don't know the solution.
>>>>>
>>>>>So the problem is the symbolic links on FTP server.
>>>>>
>>>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR is actually
>>>>>/san1/oisst/NetCDF/2013/AVHRR
>>>>>Where pub -> san1
>>>>> And OI-daily-v2 -> oisst
>>>>>
>>>>>So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it
>>>>>works.
>>>>>Otherwise it does not work.
>>>>>
>>>>>Another example is : ftp://aftp.cmdl.noaa.gov
>>>>>
>>>>>/data/radiation/surfrad/dra/2013
>>>>>->/data/radiation/surfrad/Desert_Rock_NV/2013
>>>>>
>>>>>
>>>>>
>>>>>I don't know how to fix this. Well, not exactly sure where to look. It
>>>>>appears that the filename filters are being applied based on the path
>>>>>specified in DirStructParserFiles instead of just the filename.
>>>>>Because
>>>>>I
>>>>>can see the correct path when I print RemoteFile (path =
>>>>>'/san1/oisst/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.g
>>>>>z
>>>>>'
>>>>>)
>>>>>on STDOUT.
>>>>>
>>>>>
>>>>>Hopefully you guys can fix this fast or let me know where to look.
>>>>>
>>>>>Regards
>>>>>--
>>>>>Chintu Mistry
>>>>>NASA Goddard Space Flight Center
>>>>>Bldg L40B, Room S776
>>>>>Office: 240 684 0477
>>>>>Mobile: 770 310 1047
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>On 3/28/13 10:15 AM, "Mattmann, Chris A"
>>>>><ch...@jpl.nasa.gov>
>>>>>wrote:
>>>>>
>>>>>>Hey Chintu,
>>>>>>
>>>>>>This totally helped, thanks.
>>>>>>
>>>>>>Can you take a look at the plugins here?
>>>>>>
>>>>>>https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html
>>>>>>
>>>>>>
>>>>>>Try installing those, and let me know if either of those FTP
>>>>>>plugins works better and actually downloads the filesŠ
>>>>>>
>>>>>>In the meanwhile I'll look at the logs and see what I can come up
>>>>>>with.
>>>>>>
>>>>>>Cheers,
>>>>>>Chris
>>>>>>
>>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>Chris Mattmann, Ph.D.
>>>>>>Senior Computer Scientist
>>>>>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>Office: 171-266B, Mailstop: 171-246
>>>>>>Email: chris.a.mattmann@nasa.gov
>>>>>>WWW: http://sunset.usc.edu/~mattmann/
>>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>Adjunct Assistant Professor, Computer Science Department
>>>>>>University of Southern California, Los Angeles, CA 90089 USA
>>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>-----Original Message-----
>>>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>>>><ch...@nasa.gov>
>>>>>>Date: Thursday, March 28, 2013 7:12 AM
>>>>>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>>>>>><de...@oodt.apache.org>
>>>>>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>>>>>Subject: Re: pushPull component problem
>>>>>>
>>>>>>>I have attached verbose log. I have added extra print statements and
>>>>>>>compiled to see what is returned from remote site. Hope this helps.
>>>>>>>
>>>>>>>
>>>>>>>--
>>>>>>>Chintu Mistry
>>>>>>>NASA Goddard Space Flight Center
>>>>>>>Bldg L40B, Room S776
>>>>>>>Office: 240 684 0477
>>>>>>>Mobile: 770 310 1047
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>>>>>>>Date: Thursday, March 28, 2013 10:05 AM
>>>>>>>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
>>>>>>><de...@oodt.apache.org>
>>>>>>>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>>>>>>>Subject: Re: pushPull component problem
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>Hey Chintu,
>>>>>>>
>>>>>>>
>>>>>>>Thanks for sending this ‹ really helped verify it's something else.
>>>>>>>
>>>>>>>
>>>>>>>Can you capture the log of push pull, with verbose logging turned
>>>>>>>on,
>>>>>>>and
>>>>>>>maybe do a pastebin?
>>>>>>>
>>>>>>>
>>>>>>>Cheers,
>>>>>>>Chris
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>>>>><ch...@nasa.gov>
>>>>>>>Date: Thursday, March 28, 2013 7:03 AM
>>>>>>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>>>>>>><de...@oodt.apache.org>
>>>>>>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>>>>>>Subject: Re: pushPull component problem
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Something else is wrong. I know that the regex is fine. I don't
>>>>>>>>even
>>>>>>>>have regex, it is straight up file name. It should match.
>>>>>>>>
>>>>>>>>
>>>>>>>>Thanks for looking into this.
>>>>>>>>--
>>>>>>>>Chintu Mistry
>>>>>>>>NASA Goddard Space Flight Center
>>>>>>>>Bldg L40B, Room S776
>>>>>>>>Office: 240 684 0477
>>>>>>>>Mobile: 770 310 1047
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>>>>>>>>Date: Thursday, March 28, 2013 9:01 AM
>>>>>>>>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
>>>>>>>><de...@oodt.apache.org>
>>>>>>>>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>>>>>>>>Subject: Re: pushPull component problem
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Hey Chintu,
>>>>>>>>
>>>>>>>>
>>>>>>>>Sorry that you have been having trouble and thanks for pasting the
>>>>>>>>relevant portions
>>>>>>>>of push pull config to diagnose this.
>>>>>>>>
>>>>>>>>
>>>>>>>>I think you have an issue in your RemoteFileSpecs file
>>>>>>>>(TEST_AVHRR_ONLY.xml) --
>>>>>>>>one quick way to debug this is to use the tool that Brian Foster
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
>>>>>>>>$ cd bin
>>>>>>>>$ java Djava.ext.dirs=../lib
>>>>>>>>org.apache.oodt.cas.pushpull.util.ExpressionValidator
>>>>>>>>
>>>>>>>>
>>>>>>>>It's a simple GUI that will allow you to put in regular expressions
>>>>>>>>from
>>>>>>>>your push pull
>>>>>>>>RmoteFileSpecs files, and then put in file names from the server to
>>>>>>>>see
>>>>>>>>if they match.
>>>>>>>>
>>>>>>>>
>>>>>>>>Can you run the above and let me know if it's matching? If not,
>>>>>>>>we've
>>>>>>>>found your issue.
>>>>>>>>If it is matching, we'll move on to the next set of things to
>>>>>>>>check.
>>>>>>>>
>>>>>>>>
>>>>>>>>Thanks!
>>>>>>>>
>>>>>>>>
>>>>>>>>Cheers,
>>>>>>>>Chris
>>>>>>>>
>>>>>>>>
>>>>>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>>>>>><ch...@nasa.gov>
>>>>>>>>Date: Thursday, March 28, 2013 5:47 AM
>>>>>>>>To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>>>>>>>Cc: jpluser <ch...@jpl.nasa.gov>, "Ramirez, Paul M
>>>>>>>>(388J)"
>>>>>>>><pa...@jpl.nasa.gov>
>>>>>>>>Subject: pushPull component problem
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>Hi,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>I have been playing around with push pull for last two days and
>>>>>>>>>trying
>>>>>>>>>to download some data from remote location without success.
>>>>>>>>>I would appreciate if you guys can shed some light on what is
>>>>>>>>>going
>>>>>>>>>on
>>>>>>>>>and why isn't push pull find files to download even though its
>>>>>>>>>clearly
>>>>>>>>>there at remote location.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Here is my configuration. I am trying to download
>>>>>>>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_prelimin
>>>>>>>>>a
>>>>>>>>>r
>>>>>>>>>y
>>>>>>>>>.
>>>>>>>>>n
>>>>>>>>>c
>>>>>>>>>.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine
>>>>>>>>>and
>>>>>>>>>it lists all the files as well. However for some reason it
>>>>>>>>> does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what
>>>>>>>>>I
>>>>>>>>>am
>>>>>>>>>doing wrong. I have tried almost all combinations and still could
>>>>>>>>>not
>>>>>>>>>get it to work.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>In my /etc/pushpull.properties file, I have types=false so that it
>>>>>>>>>downloads the file even if its not specified in mime-types.xml
>>>>>>>>>file.
>>>>>>>>>org.apache.oodt.cas.pushpull.allow.only.defined.types=false
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>ExternalSources.xml
>>>>>>>>><sources>
>>>>>>>>> <source host="eclipse.ncdc.noaa.gov">
>>>>>>>>> <login type="ftp" alias="eclipseClass">
>>>>>>>>> <username>anonymous</username>
>>>>>>>>> <password>user@host.com</password>
>>>>>>>>> </login>
>>>>>>>>> </source>
>>>>>>>>></sources>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>RemoteSpecs.xml
>>>>>>>>> <daemon alias="eclipseClass" active="yes">
>>>>>>>>> <runInfo firstRunDateTime="2013-03-25T00:00:00Z"
>>>>>>>>>period="1m" runOnReboot="yes"/>
>>>>>>>>> <propInfo
>>>>>>>>>dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
>>>>>>>>> <propFiles regExp="TEST_AVHRR_ONLY\.xml"
>>>>>>>>>parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirS
>>>>>>>>>t
>>>>>>>>>r
>>>>>>>>>u
>>>>>>>>>c
>>>>>>>>>t
>>>>>>>>>X
>>>>>>>>>mlParser"/>
>>>>>>>>> </propInfo>
>>>>>>>>> <dataInfo stagingArea="AVHRR-ONLY"
>>>>>>>>>deleteFromServer="no"
>>>>>>>>>queryElement="Filename"/>
>>>>>>>>> </daemon>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>TEST_AVHRR_ONLY.xml
>>>>>>>>><root>
>>>>>>>>> <dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
>>>>>>>>> <nodirs/>
>>>>>>>>> <!-- <file
>>>>>>>>>name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
>>>>>>>>> <file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
>>>>>>>>> </dirstruct>
>>>>>>>>></root>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Regards
>>>>>>>>>--
>>>>>>>>>Chintu Mistry
>>>>>>>>>NASA Goddard Space Flight Center
>>>>>>>>>Bldg L40B, Room S776
>>>>>>>>>Office: 240 684 0477
>>>>>>>>>Mobile: 770 310 1047
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: pushPull component problem
Posted by "Mistry, Chintu (GSFC-586.0)[SGT INC]" <ch...@nasa.gov>.
Sorry about not explaining properly. When I said "failed to run", I meant
class not found. Did not really look deep enough what the problem is.
Apparently I did not include all jar files properly.
Anyway, I managed to run both ftp client (of course I did change
ProtocolFactor.xml file). But still no luck. Attaching files with output.
Ftp4CheFtp => Completely barfed
jvFTP => runs fine but same output as the other plugins.
Regards
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
On 3/28/13 12:13 PM, "Mattmann, Chris A" <ch...@jpl.nasa.gov>
wrote:
>Hey Chintu,
>
>Thanks -- sorry to be asking all these questions, but what does "failed to
>run" mean?
>
>Did you update the protocol plugins config to ref the new plugins and
>their FQCN (fully
>qualified class names?) Do you have a log file for that?
>
>Cheers,
>Chris
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>-----Original Message-----
>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
>Date: Thursday, March 28, 2013 9:12 AM
>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
><de...@oodt.apache.org>
>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>Subject: Re: pushPull component problem
>
>>Since we are not using OODT 0.6, it did not compile straight away.
>>However, I did manage to compile it with "0.5" version of
>>cas-protocol-api. But when I was running it, It failed to run.
>>
>>--
>>Chintu Mistry
>>NASA Goddard Space Flight Center
>>Bldg L40B, Room S776
>>Office: 240 684 0477
>>Mobile: 770 310 1047
>>
>>
>>
>>
>>
>>
>>On 3/28/13 12:04 PM, "Mattmann, Chris A" <ch...@jpl.nasa.gov>
>>wrote:
>>
>>>Hey Chintu,
>>>
>>>Thanks for identifying this!
>>>
>>>Have you tried the alternative plugins? They use a diff set of FTP
>>>protocol
>>>plugins (that unfortunately are not ALv2 licensed or Category A
>>>approved,
>>>so we can't ship with Apache OODT).
>>>
>>>They may do better at handling that symlink than the default version of
>>>commons-ftp
>>>we're using.
>>>
>>>Let me know. We'll investigate.
>>>
>>>Cheers,
>>>Chris
>>>
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>Chris Mattmann, Ph.D.
>>>Senior Computer Scientist
>>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>Office: 171-266B, Mailstop: 171-246
>>>Email: chris.a.mattmann@nasa.gov
>>>WWW: http://sunset.usc.edu/~mattmann/
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>Adjunct Assistant Professor, Computer Science Department
>>>University of Southern California, Los Angeles, CA 90089 USA
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>>
>>>
>>>
>>>-----Original Message-----
>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>><ch...@nasa.gov>
>>>Date: Thursday, March 28, 2013 9:02 AM
>>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>>><de...@oodt.apache.org>
>>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>>Subject: Re: pushPull component problem
>>>
>>>>Chris,
>>>>
>>>>I figured out the problem. Don't know the solution.
>>>>
>>>>So the problem is the symbolic links on FTP server.
>>>>
>>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR is actually
>>>>/san1/oisst/NetCDF/2013/AVHRR
>>>>Where pub -> san1
>>>> And OI-daily-v2 -> oisst
>>>>
>>>>So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it
>>>>works.
>>>>Otherwise it does not work.
>>>>
>>>>Another example is : ftp://aftp.cmdl.noaa.gov
>>>>
>>>>/data/radiation/surfrad/dra/2013
>>>>->/data/radiation/surfrad/Desert_Rock_NV/2013
>>>>
>>>>
>>>>
>>>>I don't know how to fix this. Well, not exactly sure where to look. It
>>>>appears that the filename filters are being applied based on the path
>>>>specified in DirStructParserFiles instead of just the filename. Because
>>>>I
>>>>can see the correct path when I print RemoteFile (path =
>>>>'/san1/oisst/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz
>>>>'
>>>>)
>>>>on STDOUT.
>>>>
>>>>
>>>>Hopefully you guys can fix this fast or let me know where to look.
>>>>
>>>>Regards
>>>>--
>>>>Chintu Mistry
>>>>NASA Goddard Space Flight Center
>>>>Bldg L40B, Room S776
>>>>Office: 240 684 0477
>>>>Mobile: 770 310 1047
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>On 3/28/13 10:15 AM, "Mattmann, Chris A"
>>>><ch...@jpl.nasa.gov>
>>>>wrote:
>>>>
>>>>>Hey Chintu,
>>>>>
>>>>>This totally helped, thanks.
>>>>>
>>>>>Can you take a look at the plugins here?
>>>>>
>>>>>https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html
>>>>>
>>>>>
>>>>>Try installing those, and let me know if either of those FTP
>>>>>plugins works better and actually downloads the filesŠ
>>>>>
>>>>>In the meanwhile I'll look at the logs and see what I can come up
>>>>>with.
>>>>>
>>>>>Cheers,
>>>>>Chris
>>>>>
>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>Chris Mattmann, Ph.D.
>>>>>Senior Computer Scientist
>>>>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>Office: 171-266B, Mailstop: 171-246
>>>>>Email: chris.a.mattmann@nasa.gov
>>>>>WWW: http://sunset.usc.edu/~mattmann/
>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>Adjunct Assistant Professor, Computer Science Department
>>>>>University of Southern California, Los Angeles, CA 90089 USA
>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>-----Original Message-----
>>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>>><ch...@nasa.gov>
>>>>>Date: Thursday, March 28, 2013 7:12 AM
>>>>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>>>>><de...@oodt.apache.org>
>>>>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>>>>Subject: Re: pushPull component problem
>>>>>
>>>>>>I have attached verbose log. I have added extra print statements and
>>>>>>compiled to see what is returned from remote site. Hope this helps.
>>>>>>
>>>>>>
>>>>>>--
>>>>>>Chintu Mistry
>>>>>>NASA Goddard Space Flight Center
>>>>>>Bldg L40B, Room S776
>>>>>>Office: 240 684 0477
>>>>>>Mobile: 770 310 1047
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>>>>>>Date: Thursday, March 28, 2013 10:05 AM
>>>>>>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
>>>>>><de...@oodt.apache.org>
>>>>>>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>>>>>>Subject: Re: pushPull component problem
>>>>>>
>>>>>>
>>>>>>
>>>>>>Hey Chintu,
>>>>>>
>>>>>>
>>>>>>Thanks for sending this ‹ really helped verify it's something else.
>>>>>>
>>>>>>
>>>>>>Can you capture the log of push pull, with verbose logging turned on,
>>>>>>and
>>>>>>maybe do a pastebin?
>>>>>>
>>>>>>
>>>>>>Cheers,
>>>>>>Chris
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>>>><ch...@nasa.gov>
>>>>>>Date: Thursday, March 28, 2013 7:03 AM
>>>>>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>>>>>><de...@oodt.apache.org>
>>>>>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>>>>>Subject: Re: pushPull component problem
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>Something else is wrong. I know that the regex is fine. I don't even
>>>>>>>have regex, it is straight up file name. It should match.
>>>>>>>
>>>>>>>
>>>>>>>Thanks for looking into this.
>>>>>>>--
>>>>>>>Chintu Mistry
>>>>>>>NASA Goddard Space Flight Center
>>>>>>>Bldg L40B, Room S776
>>>>>>>Office: 240 684 0477
>>>>>>>Mobile: 770 310 1047
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>>>>>>>Date: Thursday, March 28, 2013 9:01 AM
>>>>>>>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
>>>>>>><de...@oodt.apache.org>
>>>>>>>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>>>>>>>Subject: Re: pushPull component problem
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>Hey Chintu,
>>>>>>>
>>>>>>>
>>>>>>>Sorry that you have been having trouble and thanks for pasting the
>>>>>>>relevant portions
>>>>>>>of push pull config to diagnose this.
>>>>>>>
>>>>>>>
>>>>>>>I think you have an issue in your RemoteFileSpecs file
>>>>>>>(TEST_AVHRR_ONLY.xml) --
>>>>>>>one quick way to debug this is to use the tool that Brian Foster
>>>>>>>wrote:
>>>>>>>
>>>>>>>
>>>>>>>(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
>>>>>>>$ cd bin
>>>>>>>$ java Djava.ext.dirs=../lib
>>>>>>>org.apache.oodt.cas.pushpull.util.ExpressionValidator
>>>>>>>
>>>>>>>
>>>>>>>It's a simple GUI that will allow you to put in regular expressions
>>>>>>>from
>>>>>>>your push pull
>>>>>>>RmoteFileSpecs files, and then put in file names from the server to
>>>>>>>see
>>>>>>>if they match.
>>>>>>>
>>>>>>>
>>>>>>>Can you run the above and let me know if it's matching? If not,
>>>>>>>we've
>>>>>>>found your issue.
>>>>>>>If it is matching, we'll move on to the next set of things to check.
>>>>>>>
>>>>>>>
>>>>>>>Thanks!
>>>>>>>
>>>>>>>
>>>>>>>Cheers,
>>>>>>>Chris
>>>>>>>
>>>>>>>
>>>>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>>>>><ch...@nasa.gov>
>>>>>>>Date: Thursday, March 28, 2013 5:47 AM
>>>>>>>To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>>>>>>Cc: jpluser <ch...@jpl.nasa.gov>, "Ramirez, Paul M
>>>>>>>(388J)"
>>>>>>><pa...@jpl.nasa.gov>
>>>>>>>Subject: pushPull component problem
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>I have been playing around with push pull for last two days and
>>>>>>>>trying
>>>>>>>>to download some data from remote location without success.
>>>>>>>>I would appreciate if you guys can shed some light on what is going
>>>>>>>>on
>>>>>>>>and why isn't push pull find files to download even though its
>>>>>>>>clearly
>>>>>>>>there at remote location.
>>>>>>>>
>>>>>>>>
>>>>>>>>Here is my configuration. I am trying to download
>>>>>>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_prelimina
>>>>>>>>r
>>>>>>>>y
>>>>>>>>.
>>>>>>>>n
>>>>>>>>c
>>>>>>>>.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine
>>>>>>>>and
>>>>>>>>it lists all the files as well. However for some reason it
>>>>>>>> does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I
>>>>>>>>am
>>>>>>>>doing wrong. I have tried almost all combinations and still could
>>>>>>>>not
>>>>>>>>get it to work.
>>>>>>>>
>>>>>>>>
>>>>>>>>In my /etc/pushpull.properties file, I have types=false so that it
>>>>>>>>downloads the file even if its not specified in mime-types.xml
>>>>>>>>file.
>>>>>>>>org.apache.oodt.cas.pushpull.allow.only.defined.types=false
>>>>>>>>
>>>>>>>>
>>>>>>>>ExternalSources.xml
>>>>>>>><sources>
>>>>>>>> <source host="eclipse.ncdc.noaa.gov">
>>>>>>>> <login type="ftp" alias="eclipseClass">
>>>>>>>> <username>anonymous</username>
>>>>>>>> <password>user@host.com</password>
>>>>>>>> </login>
>>>>>>>> </source>
>>>>>>>></sources>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>RemoteSpecs.xml
>>>>>>>> <daemon alias="eclipseClass" active="yes">
>>>>>>>> <runInfo firstRunDateTime="2013-03-25T00:00:00Z"
>>>>>>>>period="1m" runOnReboot="yes"/>
>>>>>>>> <propInfo
>>>>>>>>dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
>>>>>>>> <propFiles regExp="TEST_AVHRR_ONLY\.xml"
>>>>>>>>parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirSt
>>>>>>>>r
>>>>>>>>u
>>>>>>>>c
>>>>>>>>t
>>>>>>>>X
>>>>>>>>mlParser"/>
>>>>>>>> </propInfo>
>>>>>>>> <dataInfo stagingArea="AVHRR-ONLY"
>>>>>>>>deleteFromServer="no"
>>>>>>>>queryElement="Filename"/>
>>>>>>>> </daemon>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>TEST_AVHRR_ONLY.xml
>>>>>>>><root>
>>>>>>>> <dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
>>>>>>>> <nodirs/>
>>>>>>>> <!-- <file
>>>>>>>>name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
>>>>>>>> <file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
>>>>>>>> </dirstruct>
>>>>>>>></root>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Regards
>>>>>>>>--
>>>>>>>>Chintu Mistry
>>>>>>>>NASA Goddard Space Flight Center
>>>>>>>>Bldg L40B, Room S776
>>>>>>>>Office: 240 684 0477
>>>>>>>>Mobile: 770 310 1047
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: pushPull component problem
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Chintu,
Thanks -- sorry to be asking all these questions, but what does "failed to
run" mean?
Did you update the protocol plugins config to ref the new plugins and
their FQCN (fully
qualified class names?) Do you have a log file for that?
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
Date: Thursday, March 28, 2013 9:12 AM
To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
<de...@oodt.apache.org>
Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
Subject: Re: pushPull component problem
>Since we are not using OODT 0.6, it did not compile straight away.
>However, I did manage to compile it with "0.5" version of
>cas-protocol-api. But when I was running it, It failed to run.
>
>--
>Chintu Mistry
>NASA Goddard Space Flight Center
>Bldg L40B, Room S776
>Office: 240 684 0477
>Mobile: 770 310 1047
>
>
>
>
>
>
>On 3/28/13 12:04 PM, "Mattmann, Chris A" <ch...@jpl.nasa.gov>
>wrote:
>
>>Hey Chintu,
>>
>>Thanks for identifying this!
>>
>>Have you tried the alternative plugins? They use a diff set of FTP
>>protocol
>>plugins (that unfortunately are not ALv2 licensed or Category A approved,
>>so we can't ship with Apache OODT).
>>
>>They may do better at handling that symlink than the default version of
>>commons-ftp
>>we're using.
>>
>>Let me know. We'll investigate.
>>
>>Cheers,
>>Chris
>>
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>Chris Mattmann, Ph.D.
>>Senior Computer Scientist
>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>Office: 171-266B, Mailstop: 171-246
>>Email: chris.a.mattmann@nasa.gov
>>WWW: http://sunset.usc.edu/~mattmann/
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>Adjunct Assistant Professor, Computer Science Department
>>University of Southern California, Los Angeles, CA 90089 USA
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>-----Original Message-----
>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
>>Date: Thursday, March 28, 2013 9:02 AM
>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>><de...@oodt.apache.org>
>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>Subject: Re: pushPull component problem
>>
>>>Chris,
>>>
>>>I figured out the problem. Don't know the solution.
>>>
>>>So the problem is the symbolic links on FTP server.
>>>
>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR is actually
>>>/san1/oisst/NetCDF/2013/AVHRR
>>>Where pub -> san1
>>> And OI-daily-v2 -> oisst
>>>
>>>So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it
>>>works.
>>>Otherwise it does not work.
>>>
>>>Another example is : ftp://aftp.cmdl.noaa.gov
>>>
>>>/data/radiation/surfrad/dra/2013
>>>->/data/radiation/surfrad/Desert_Rock_NV/2013
>>>
>>>
>>>
>>>I don't know how to fix this. Well, not exactly sure where to look. It
>>>appears that the filename filters are being applied based on the path
>>>specified in DirStructParserFiles instead of just the filename. Because
>>>I
>>>can see the correct path when I print RemoteFile (path =
>>>'/san1/oisst/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz'
>>>)
>>>on STDOUT.
>>>
>>>
>>>Hopefully you guys can fix this fast or let me know where to look.
>>>
>>>Regards
>>>--
>>>Chintu Mistry
>>>NASA Goddard Space Flight Center
>>>Bldg L40B, Room S776
>>>Office: 240 684 0477
>>>Mobile: 770 310 1047
>>>
>>>
>>>
>>>
>>>
>>>
>>>On 3/28/13 10:15 AM, "Mattmann, Chris A" <ch...@jpl.nasa.gov>
>>>wrote:
>>>
>>>>Hey Chintu,
>>>>
>>>>This totally helped, thanks.
>>>>
>>>>Can you take a look at the plugins here?
>>>>
>>>>https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html
>>>>
>>>>
>>>>Try installing those, and let me know if either of those FTP
>>>>plugins works better and actually downloads the filesŠ
>>>>
>>>>In the meanwhile I'll look at the logs and see what I can come up
>>>>with.
>>>>
>>>>Cheers,
>>>>Chris
>>>>
>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>Chris Mattmann, Ph.D.
>>>>Senior Computer Scientist
>>>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>Office: 171-266B, Mailstop: 171-246
>>>>Email: chris.a.mattmann@nasa.gov
>>>>WWW: http://sunset.usc.edu/~mattmann/
>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>Adjunct Assistant Professor, Computer Science Department
>>>>University of Southern California, Los Angeles, CA 90089 USA
>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>>>
>>>>
>>>>
>>>>-----Original Message-----
>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>><ch...@nasa.gov>
>>>>Date: Thursday, March 28, 2013 7:12 AM
>>>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>>>><de...@oodt.apache.org>
>>>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>>>Subject: Re: pushPull component problem
>>>>
>>>>>I have attached verbose log. I have added extra print statements and
>>>>>compiled to see what is returned from remote site. Hope this helps.
>>>>>
>>>>>
>>>>>--
>>>>>Chintu Mistry
>>>>>NASA Goddard Space Flight Center
>>>>>Bldg L40B, Room S776
>>>>>Office: 240 684 0477
>>>>>Mobile: 770 310 1047
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>>>>>Date: Thursday, March 28, 2013 10:05 AM
>>>>>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
>>>>><de...@oodt.apache.org>
>>>>>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>>>>>Subject: Re: pushPull component problem
>>>>>
>>>>>
>>>>>
>>>>>Hey Chintu,
>>>>>
>>>>>
>>>>>Thanks for sending this ‹ really helped verify it's something else.
>>>>>
>>>>>
>>>>>Can you capture the log of push pull, with verbose logging turned on,
>>>>>and
>>>>>maybe do a pastebin?
>>>>>
>>>>>
>>>>>Cheers,
>>>>>Chris
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>>><ch...@nasa.gov>
>>>>>Date: Thursday, March 28, 2013 7:03 AM
>>>>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>>>>><de...@oodt.apache.org>
>>>>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>>>>Subject: Re: pushPull component problem
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>Something else is wrong. I know that the regex is fine. I don't even
>>>>>>have regex, it is straight up file name. It should match.
>>>>>>
>>>>>>
>>>>>>Thanks for looking into this.
>>>>>>--
>>>>>>Chintu Mistry
>>>>>>NASA Goddard Space Flight Center
>>>>>>Bldg L40B, Room S776
>>>>>>Office: 240 684 0477
>>>>>>Mobile: 770 310 1047
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>>>>>>Date: Thursday, March 28, 2013 9:01 AM
>>>>>>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
>>>>>><de...@oodt.apache.org>
>>>>>>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>>>>>>Subject: Re: pushPull component problem
>>>>>>
>>>>>>
>>>>>>
>>>>>>Hey Chintu,
>>>>>>
>>>>>>
>>>>>>Sorry that you have been having trouble and thanks for pasting the
>>>>>>relevant portions
>>>>>>of push pull config to diagnose this.
>>>>>>
>>>>>>
>>>>>>I think you have an issue in your RemoteFileSpecs file
>>>>>>(TEST_AVHRR_ONLY.xml) --
>>>>>>one quick way to debug this is to use the tool that Brian Foster
>>>>>>wrote:
>>>>>>
>>>>>>
>>>>>>(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
>>>>>>$ cd bin
>>>>>>$ java Djava.ext.dirs=../lib
>>>>>>org.apache.oodt.cas.pushpull.util.ExpressionValidator
>>>>>>
>>>>>>
>>>>>>It's a simple GUI that will allow you to put in regular expressions
>>>>>>from
>>>>>>your push pull
>>>>>>RmoteFileSpecs files, and then put in file names from the server to
>>>>>>see
>>>>>>if they match.
>>>>>>
>>>>>>
>>>>>>Can you run the above and let me know if it's matching? If not, we've
>>>>>>found your issue.
>>>>>>If it is matching, we'll move on to the next set of things to check.
>>>>>>
>>>>>>
>>>>>>Thanks!
>>>>>>
>>>>>>
>>>>>>Cheers,
>>>>>>Chris
>>>>>>
>>>>>>
>>>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>>>><ch...@nasa.gov>
>>>>>>Date: Thursday, March 28, 2013 5:47 AM
>>>>>>To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>>>>>Cc: jpluser <ch...@jpl.nasa.gov>, "Ramirez, Paul M (388J)"
>>>>>><pa...@jpl.nasa.gov>
>>>>>>Subject: pushPull component problem
>>>>>>
>>>>>>
>>>>>>
>>>>>>>Hi,
>>>>>>>
>>>>>>>
>>>>>>>I have been playing around with push pull for last two days and
>>>>>>>trying
>>>>>>>to download some data from remote location without success.
>>>>>>>I would appreciate if you guys can shed some light on what is going
>>>>>>>on
>>>>>>>and why isn't push pull find files to download even though its
>>>>>>>clearly
>>>>>>>there at remote location.
>>>>>>>
>>>>>>>
>>>>>>>Here is my configuration. I am trying to download
>>>>>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminar
>>>>>>>y
>>>>>>>.
>>>>>>>n
>>>>>>>c
>>>>>>>.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine
>>>>>>>and
>>>>>>>it lists all the files as well. However for some reason it
>>>>>>> does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I
>>>>>>>am
>>>>>>>doing wrong. I have tried almost all combinations and still could
>>>>>>>not
>>>>>>>get it to work.
>>>>>>>
>>>>>>>
>>>>>>>In my /etc/pushpull.properties file, I have types=false so that it
>>>>>>>downloads the file even if its not specified in mime-types.xml file.
>>>>>>>org.apache.oodt.cas.pushpull.allow.only.defined.types=false
>>>>>>>
>>>>>>>
>>>>>>>ExternalSources.xml
>>>>>>><sources>
>>>>>>> <source host="eclipse.ncdc.noaa.gov">
>>>>>>> <login type="ftp" alias="eclipseClass">
>>>>>>> <username>anonymous</username>
>>>>>>> <password>user@host.com</password>
>>>>>>> </login>
>>>>>>> </source>
>>>>>>></sources>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>RemoteSpecs.xml
>>>>>>> <daemon alias="eclipseClass" active="yes">
>>>>>>> <runInfo firstRunDateTime="2013-03-25T00:00:00Z"
>>>>>>>period="1m" runOnReboot="yes"/>
>>>>>>> <propInfo
>>>>>>>dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
>>>>>>> <propFiles regExp="TEST_AVHRR_ONLY\.xml"
>>>>>>>parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStr
>>>>>>>u
>>>>>>>c
>>>>>>>t
>>>>>>>X
>>>>>>>mlParser"/>
>>>>>>> </propInfo>
>>>>>>> <dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no"
>>>>>>>queryElement="Filename"/>
>>>>>>> </daemon>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>TEST_AVHRR_ONLY.xml
>>>>>>><root>
>>>>>>> <dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
>>>>>>> <nodirs/>
>>>>>>> <!-- <file
>>>>>>>name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
>>>>>>> <file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
>>>>>>> </dirstruct>
>>>>>>></root>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>Regards
>>>>>>>--
>>>>>>>Chintu Mistry
>>>>>>>NASA Goddard Space Flight Center
>>>>>>>Bldg L40B, Room S776
>>>>>>>Office: 240 684 0477
>>>>>>>Mobile: 770 310 1047
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: pushPull component problem
Posted by "Mistry, Chintu (GSFC-586.0)[SGT INC]" <ch...@nasa.gov>.
Since we are not using OODT 0.6, it did not compile straight away.
However, I did manage to compile it with "0.5" version of
cas-protocol-api. But when I was running it, It failed to run.
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
On 3/28/13 12:04 PM, "Mattmann, Chris A" <ch...@jpl.nasa.gov>
wrote:
>Hey Chintu,
>
>Thanks for identifying this!
>
>Have you tried the alternative plugins? They use a diff set of FTP
>protocol
>plugins (that unfortunately are not ALv2 licensed or Category A approved,
>so we can't ship with Apache OODT).
>
>They may do better at handling that symlink than the default version of
>commons-ftp
>we're using.
>
>Let me know. We'll investigate.
>
>Cheers,
>Chris
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>-----Original Message-----
>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
>Date: Thursday, March 28, 2013 9:02 AM
>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
><de...@oodt.apache.org>
>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>Subject: Re: pushPull component problem
>
>>Chris,
>>
>>I figured out the problem. Don't know the solution.
>>
>>So the problem is the symbolic links on FTP server.
>>
>>/pub/OI-daily-v2/NetCDF/2013/AVHRR is actually
>>/san1/oisst/NetCDF/2013/AVHRR
>>Where pub -> san1
>> And OI-daily-v2 -> oisst
>>
>>So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it
>>works.
>>Otherwise it does not work.
>>
>>Another example is : ftp://aftp.cmdl.noaa.gov
>>
>>/data/radiation/surfrad/dra/2013
>>->/data/radiation/surfrad/Desert_Rock_NV/2013
>>
>>
>>
>>I don't know how to fix this. Well, not exactly sure where to look. It
>>appears that the filename filters are being applied based on the path
>>specified in DirStructParserFiles instead of just the filename. Because I
>>can see the correct path when I print RemoteFile (path =
>>'/san1/oisst/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz'
>>)
>>on STDOUT.
>>
>>
>>Hopefully you guys can fix this fast or let me know where to look.
>>
>>Regards
>>--
>>Chintu Mistry
>>NASA Goddard Space Flight Center
>>Bldg L40B, Room S776
>>Office: 240 684 0477
>>Mobile: 770 310 1047
>>
>>
>>
>>
>>
>>
>>On 3/28/13 10:15 AM, "Mattmann, Chris A" <ch...@jpl.nasa.gov>
>>wrote:
>>
>>>Hey Chintu,
>>>
>>>This totally helped, thanks.
>>>
>>>Can you take a look at the plugins here?
>>>
>>>https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html
>>>
>>>
>>>Try installing those, and let me know if either of those FTP
>>>plugins works better and actually downloads the filesŠ
>>>
>>>In the meanwhile I'll look at the logs and see what I can come up
>>>with.
>>>
>>>Cheers,
>>>Chris
>>>
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>Chris Mattmann, Ph.D.
>>>Senior Computer Scientist
>>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>Office: 171-266B, Mailstop: 171-246
>>>Email: chris.a.mattmann@nasa.gov
>>>WWW: http://sunset.usc.edu/~mattmann/
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>Adjunct Assistant Professor, Computer Science Department
>>>University of Southern California, Los Angeles, CA 90089 USA
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>>
>>>
>>>
>>>-----Original Message-----
>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>><ch...@nasa.gov>
>>>Date: Thursday, March 28, 2013 7:12 AM
>>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>>><de...@oodt.apache.org>
>>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>>Subject: Re: pushPull component problem
>>>
>>>>I have attached verbose log. I have added extra print statements and
>>>>compiled to see what is returned from remote site. Hope this helps.
>>>>
>>>>
>>>>--
>>>>Chintu Mistry
>>>>NASA Goddard Space Flight Center
>>>>Bldg L40B, Room S776
>>>>Office: 240 684 0477
>>>>Mobile: 770 310 1047
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>>>>Date: Thursday, March 28, 2013 10:05 AM
>>>>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
>>>><de...@oodt.apache.org>
>>>>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>>>>Subject: Re: pushPull component problem
>>>>
>>>>
>>>>
>>>>Hey Chintu,
>>>>
>>>>
>>>>Thanks for sending this ‹ really helped verify it's something else.
>>>>
>>>>
>>>>Can you capture the log of push pull, with verbose logging turned on,
>>>>and
>>>>maybe do a pastebin?
>>>>
>>>>
>>>>Cheers,
>>>>Chris
>>>>
>>>>
>>>>
>>>>
>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>><ch...@nasa.gov>
>>>>Date: Thursday, March 28, 2013 7:03 AM
>>>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>>>><de...@oodt.apache.org>
>>>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>>>Subject: Re: pushPull component problem
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>Something else is wrong. I know that the regex is fine. I don't even
>>>>>have regex, it is straight up file name. It should match.
>>>>>
>>>>>
>>>>>Thanks for looking into this.
>>>>>--
>>>>>Chintu Mistry
>>>>>NASA Goddard Space Flight Center
>>>>>Bldg L40B, Room S776
>>>>>Office: 240 684 0477
>>>>>Mobile: 770 310 1047
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>>>>>Date: Thursday, March 28, 2013 9:01 AM
>>>>>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
>>>>><de...@oodt.apache.org>
>>>>>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>>>>>Subject: Re: pushPull component problem
>>>>>
>>>>>
>>>>>
>>>>>Hey Chintu,
>>>>>
>>>>>
>>>>>Sorry that you have been having trouble and thanks for pasting the
>>>>>relevant portions
>>>>>of push pull config to diagnose this.
>>>>>
>>>>>
>>>>>I think you have an issue in your RemoteFileSpecs file
>>>>>(TEST_AVHRR_ONLY.xml) --
>>>>>one quick way to debug this is to use the tool that Brian Foster
>>>>>wrote:
>>>>>
>>>>>
>>>>>(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
>>>>>$ cd bin
>>>>>$ java Djava.ext.dirs=../lib
>>>>>org.apache.oodt.cas.pushpull.util.ExpressionValidator
>>>>>
>>>>>
>>>>>It's a simple GUI that will allow you to put in regular expressions
>>>>>from
>>>>>your push pull
>>>>>RmoteFileSpecs files, and then put in file names from the server to
>>>>>see
>>>>>if they match.
>>>>>
>>>>>
>>>>>Can you run the above and let me know if it's matching? If not, we've
>>>>>found your issue.
>>>>>If it is matching, we'll move on to the next set of things to check.
>>>>>
>>>>>
>>>>>Thanks!
>>>>>
>>>>>
>>>>>Cheers,
>>>>>Chris
>>>>>
>>>>>
>>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>>><ch...@nasa.gov>
>>>>>Date: Thursday, March 28, 2013 5:47 AM
>>>>>To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>>>>Cc: jpluser <ch...@jpl.nasa.gov>, "Ramirez, Paul M (388J)"
>>>>><pa...@jpl.nasa.gov>
>>>>>Subject: pushPull component problem
>>>>>
>>>>>
>>>>>
>>>>>>Hi,
>>>>>>
>>>>>>
>>>>>>I have been playing around with push pull for last two days and
>>>>>>trying
>>>>>>to download some data from remote location without success.
>>>>>>I would appreciate if you guys can shed some light on what is going
>>>>>>on
>>>>>>and why isn't push pull find files to download even though its
>>>>>>clearly
>>>>>>there at remote location.
>>>>>>
>>>>>>
>>>>>>Here is my configuration. I am trying to download
>>>>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary
>>>>>>.
>>>>>>n
>>>>>>c
>>>>>>.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine
>>>>>>and
>>>>>>it lists all the files as well. However for some reason it
>>>>>> does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I
>>>>>>am
>>>>>>doing wrong. I have tried almost all combinations and still could not
>>>>>>get it to work.
>>>>>>
>>>>>>
>>>>>>In my /etc/pushpull.properties file, I have types=false so that it
>>>>>>downloads the file even if its not specified in mime-types.xml file.
>>>>>>org.apache.oodt.cas.pushpull.allow.only.defined.types=false
>>>>>>
>>>>>>
>>>>>>ExternalSources.xml
>>>>>><sources>
>>>>>> <source host="eclipse.ncdc.noaa.gov">
>>>>>> <login type="ftp" alias="eclipseClass">
>>>>>> <username>anonymous</username>
>>>>>> <password>user@host.com</password>
>>>>>> </login>
>>>>>> </source>
>>>>>></sources>
>>>>>>
>>>>>>
>>>>>>
>>>>>>RemoteSpecs.xml
>>>>>> <daemon alias="eclipseClass" active="yes">
>>>>>> <runInfo firstRunDateTime="2013-03-25T00:00:00Z"
>>>>>>period="1m" runOnReboot="yes"/>
>>>>>> <propInfo
>>>>>>dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
>>>>>> <propFiles regExp="TEST_AVHRR_ONLY\.xml"
>>>>>>parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStru
>>>>>>c
>>>>>>t
>>>>>>X
>>>>>>mlParser"/>
>>>>>> </propInfo>
>>>>>> <dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no"
>>>>>>queryElement="Filename"/>
>>>>>> </daemon>
>>>>>>
>>>>>>
>>>>>>
>>>>>>TEST_AVHRR_ONLY.xml
>>>>>><root>
>>>>>> <dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
>>>>>> <nodirs/>
>>>>>> <!-- <file
>>>>>>name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
>>>>>> <file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
>>>>>> </dirstruct>
>>>>>></root>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>Regards
>>>>>>--
>>>>>>Chintu Mistry
>>>>>>NASA Goddard Space Flight Center
>>>>>>Bldg L40B, Room S776
>>>>>>Office: 240 684 0477
>>>>>>Mobile: 770 310 1047
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: pushPull component problem
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Chintu,
Thanks for identifying this!
Have you tried the alternative plugins? They use a diff set of FTP protocol
plugins (that unfortunately are not ALv2 licensed or Category A approved,
so we can't ship with Apache OODT).
They may do better at handling that symlink than the default version of
commons-ftp
we're using.
Let me know. We'll investigate.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
Date: Thursday, March 28, 2013 9:02 AM
To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
<de...@oodt.apache.org>
Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
Subject: Re: pushPull component problem
>Chris,
>
>I figured out the problem. Don't know the solution.
>
>So the problem is the symbolic links on FTP server.
>
>/pub/OI-daily-v2/NetCDF/2013/AVHRR is actually
>/san1/oisst/NetCDF/2013/AVHRR
>Where pub -> san1
> And OI-daily-v2 -> oisst
>
>So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it works.
>Otherwise it does not work.
>
>Another example is : ftp://aftp.cmdl.noaa.gov
>
>/data/radiation/surfrad/dra/2013
>->/data/radiation/surfrad/Desert_Rock_NV/2013
>
>
>
>I don't know how to fix this. Well, not exactly sure where to look. It
>appears that the filename filters are being applied based on the path
>specified in DirStructParserFiles instead of just the filename. Because I
>can see the correct path when I print RemoteFile (path =
>'/san1/oisst/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz' )
>on STDOUT.
>
>
>Hopefully you guys can fix this fast or let me know where to look.
>
>Regards
>--
>Chintu Mistry
>NASA Goddard Space Flight Center
>Bldg L40B, Room S776
>Office: 240 684 0477
>Mobile: 770 310 1047
>
>
>
>
>
>
>On 3/28/13 10:15 AM, "Mattmann, Chris A" <ch...@jpl.nasa.gov>
>wrote:
>
>>Hey Chintu,
>>
>>This totally helped, thanks.
>>
>>Can you take a look at the plugins here?
>>
>>https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html
>>
>>
>>Try installing those, and let me know if either of those FTP
>>plugins works better and actually downloads the filesŠ
>>
>>In the meanwhile I'll look at the logs and see what I can come up
>>with.
>>
>>Cheers,
>>Chris
>>
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>Chris Mattmann, Ph.D.
>>Senior Computer Scientist
>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>Office: 171-266B, Mailstop: 171-246
>>Email: chris.a.mattmann@nasa.gov
>>WWW: http://sunset.usc.edu/~mattmann/
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>Adjunct Assistant Professor, Computer Science Department
>>University of Southern California, Los Angeles, CA 90089 USA
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>-----Original Message-----
>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
>>Date: Thursday, March 28, 2013 7:12 AM
>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>><de...@oodt.apache.org>
>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>Subject: Re: pushPull component problem
>>
>>>I have attached verbose log. I have added extra print statements and
>>>compiled to see what is returned from remote site. Hope this helps.
>>>
>>>
>>>--
>>>Chintu Mistry
>>>NASA Goddard Space Flight Center
>>>Bldg L40B, Room S776
>>>Office: 240 684 0477
>>>Mobile: 770 310 1047
>>>
>>>
>>>
>>>
>>>
>>>
>>>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>>>Date: Thursday, March 28, 2013 10:05 AM
>>>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
>>><de...@oodt.apache.org>
>>>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>>>Subject: Re: pushPull component problem
>>>
>>>
>>>
>>>Hey Chintu,
>>>
>>>
>>>Thanks for sending this ‹ really helped verify it's something else.
>>>
>>>
>>>Can you capture the log of push pull, with verbose logging turned on,
>>>and
>>>maybe do a pastebin?
>>>
>>>
>>>Cheers,
>>>Chris
>>>
>>>
>>>
>>>
>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
>>>Date: Thursday, March 28, 2013 7:03 AM
>>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>>><de...@oodt.apache.org>
>>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>>Subject: Re: pushPull component problem
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>Something else is wrong. I know that the regex is fine. I don't even
>>>>have regex, it is straight up file name. It should match.
>>>>
>>>>
>>>>Thanks for looking into this.
>>>>--
>>>>Chintu Mistry
>>>>NASA Goddard Space Flight Center
>>>>Bldg L40B, Room S776
>>>>Office: 240 684 0477
>>>>Mobile: 770 310 1047
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>>>>Date: Thursday, March 28, 2013 9:01 AM
>>>>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
>>>><de...@oodt.apache.org>
>>>>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>>>>Subject: Re: pushPull component problem
>>>>
>>>>
>>>>
>>>>Hey Chintu,
>>>>
>>>>
>>>>Sorry that you have been having trouble and thanks for pasting the
>>>>relevant portions
>>>>of push pull config to diagnose this.
>>>>
>>>>
>>>>I think you have an issue in your RemoteFileSpecs file
>>>>(TEST_AVHRR_ONLY.xml) --
>>>>one quick way to debug this is to use the tool that Brian Foster wrote:
>>>>
>>>>
>>>>(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
>>>>$ cd bin
>>>>$ java Djava.ext.dirs=../lib
>>>>org.apache.oodt.cas.pushpull.util.ExpressionValidator
>>>>
>>>>
>>>>It's a simple GUI that will allow you to put in regular expressions
>>>>from
>>>>your push pull
>>>>RmoteFileSpecs files, and then put in file names from the server to see
>>>>if they match.
>>>>
>>>>
>>>>Can you run the above and let me know if it's matching? If not, we've
>>>>found your issue.
>>>>If it is matching, we'll move on to the next set of things to check.
>>>>
>>>>
>>>>Thanks!
>>>>
>>>>
>>>>Cheers,
>>>>Chris
>>>>
>>>>
>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>><ch...@nasa.gov>
>>>>Date: Thursday, March 28, 2013 5:47 AM
>>>>To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>>>Cc: jpluser <ch...@jpl.nasa.gov>, "Ramirez, Paul M (388J)"
>>>><pa...@jpl.nasa.gov>
>>>>Subject: pushPull component problem
>>>>
>>>>
>>>>
>>>>>Hi,
>>>>>
>>>>>
>>>>>I have been playing around with push pull for last two days and trying
>>>>>to download some data from remote location without success.
>>>>>I would appreciate if you guys can shed some light on what is going on
>>>>>and why isn't push pull find files to download even though its clearly
>>>>>there at remote location.
>>>>>
>>>>>
>>>>>Here is my configuration. I am trying to download
>>>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.
>>>>>n
>>>>>c
>>>>>.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine and
>>>>>it lists all the files as well. However for some reason it
>>>>> does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I am
>>>>>doing wrong. I have tried almost all combinations and still could not
>>>>>get it to work.
>>>>>
>>>>>
>>>>>In my /etc/pushpull.properties file, I have types=false so that it
>>>>>downloads the file even if its not specified in mime-types.xml file.
>>>>>org.apache.oodt.cas.pushpull.allow.only.defined.types=false
>>>>>
>>>>>
>>>>>ExternalSources.xml
>>>>><sources>
>>>>> <source host="eclipse.ncdc.noaa.gov">
>>>>> <login type="ftp" alias="eclipseClass">
>>>>> <username>anonymous</username>
>>>>> <password>user@host.com</password>
>>>>> </login>
>>>>> </source>
>>>>></sources>
>>>>>
>>>>>
>>>>>
>>>>>RemoteSpecs.xml
>>>>> <daemon alias="eclipseClass" active="yes">
>>>>> <runInfo firstRunDateTime="2013-03-25T00:00:00Z"
>>>>>period="1m" runOnReboot="yes"/>
>>>>> <propInfo
>>>>>dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
>>>>> <propFiles regExp="TEST_AVHRR_ONLY\.xml"
>>>>>parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStruc
>>>>>t
>>>>>X
>>>>>mlParser"/>
>>>>> </propInfo>
>>>>> <dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no"
>>>>>queryElement="Filename"/>
>>>>> </daemon>
>>>>>
>>>>>
>>>>>
>>>>>TEST_AVHRR_ONLY.xml
>>>>><root>
>>>>> <dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
>>>>> <nodirs/>
>>>>> <!-- <file
>>>>>name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
>>>>> <file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
>>>>> </dirstruct>
>>>>></root>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>Regards
>>>>>--
>>>>>Chintu Mistry
>>>>>NASA Goddard Space Flight Center
>>>>>Bldg L40B, Room S776
>>>>>Office: 240 684 0477
>>>>>Mobile: 770 310 1047
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>
Re: pushPull component problem
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Chintu,
-----Original Message-----
From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
Date: Thursday, March 28, 2013 9:02 AM
To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
<de...@oodt.apache.org>
Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
Subject: Re: pushPull component problem
>Chris,
>
>I figured out the problem. Don't know the solution.
>
>So the problem is the symbolic links on FTP server.
>
>/pub/OI-daily-v2/NetCDF/2013/AVHRR is actually
>/san1/oisst/NetCDF/2013/AVHRR
>Where pub -> san1
> And OI-daily-v2 -> oisst
>
>So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it works.
>Otherwise it does not work.
>
>Another example is : ftp://aftp.cmdl.noaa.gov
>
>/data/radiation/surfrad/dra/2013
>->/data/radiation/surfrad/Desert_Rock_NV/2013
>
>
>
>I don't know how to fix this. Well, not exactly sure where to look. It
>appears that the filename filters are being applied based on the path
>specified in DirStructParserFiles instead of just the filename. Because I
>can see the correct path when I print RemoteFile (path =
>'/san1/oisst/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz' )
>on STDOUT.
Interesting. I know that different implementations of the Protocols handle
symlinks in various ways.
Brian Foster is usually excellent at tracking these types of issues down
(CC'ing him directly). Brian any thoughts?
Also see other replies from Rishi and I.
Cheers,
Chris
>
>
>Hopefully you guys can fix this fast or let me know where to look.
>
>Regards
>--
>Chintu Mistry
>NASA Goddard Space Flight Center
>Bldg L40B, Room S776
>Office: 240 684 0477
>Mobile: 770 310 1047
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: pushPull component problem
Posted by "Mistry, Chintu (GSFC-586.0)[SGT INC]" <ch...@nasa.gov>.
Chris,
I figured out the problem. Don't know the solution.
So the problem is the symbolic links on FTP server.
/pub/OI-daily-v2/NetCDF/2013/AVHRR is actually
/san1/oisst/NetCDF/2013/AVHRR
Where pub -> san1
And OI-daily-v2 -> oisst
So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it works.
Otherwise it does not work.
Another example is : ftp://aftp.cmdl.noaa.gov
/data/radiation/surfrad/dra/2013
->/data/radiation/surfrad/Desert_Rock_NV/2013
I don't know how to fix this. Well, not exactly sure where to look. It
appears that the filename filters are being applied based on the path
specified in DirStructParserFiles instead of just the filename. Because I
can see the correct path when I print RemoteFile (path =
'/san1/oisst/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz' )
on STDOUT.
Hopefully you guys can fix this fast or let me know where to look.
Regards
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
On 3/28/13 10:15 AM, "Mattmann, Chris A" <ch...@jpl.nasa.gov>
wrote:
>Hey Chintu,
>
>This totally helped, thanks.
>
>Can you take a look at the plugins here?
>
>https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html
>
>
>Try installing those, and let me know if either of those FTP
>plugins works better and actually downloads the filesŠ
>
>In the meanwhile I'll look at the logs and see what I can come up
>with.
>
>Cheers,
>Chris
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>-----Original Message-----
>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
>Date: Thursday, March 28, 2013 7:12 AM
>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
><de...@oodt.apache.org>
>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>Subject: Re: pushPull component problem
>
>>I have attached verbose log. I have added extra print statements and
>>compiled to see what is returned from remote site. Hope this helps.
>>
>>
>>--
>>Chintu Mistry
>>NASA Goddard Space Flight Center
>>Bldg L40B, Room S776
>>Office: 240 684 0477
>>Mobile: 770 310 1047
>>
>>
>>
>>
>>
>>
>>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>>Date: Thursday, March 28, 2013 10:05 AM
>>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
>><de...@oodt.apache.org>
>>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>>Subject: Re: pushPull component problem
>>
>>
>>
>>Hey Chintu,
>>
>>
>>Thanks for sending this ‹ really helped verify it's something else.
>>
>>
>>Can you capture the log of push pull, with verbose logging turned on, and
>>maybe do a pastebin?
>>
>>
>>Cheers,
>>Chris
>>
>>
>>
>>
>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
>>Date: Thursday, March 28, 2013 7:03 AM
>>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
>><de...@oodt.apache.org>
>>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>>Subject: Re: pushPull component problem
>>
>>
>>
>>>
>>>
>>>
>>>
>>>Something else is wrong. I know that the regex is fine. I don't even
>>>have regex, it is straight up file name. It should match.
>>>
>>>
>>>Thanks for looking into this.
>>>--
>>>Chintu Mistry
>>>NASA Goddard Space Flight Center
>>>Bldg L40B, Room S776
>>>Office: 240 684 0477
>>>Mobile: 770 310 1047
>>>
>>>
>>>
>>>
>>>
>>>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>>>Date: Thursday, March 28, 2013 9:01 AM
>>>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
>>><de...@oodt.apache.org>
>>>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>>>Subject: Re: pushPull component problem
>>>
>>>
>>>
>>>Hey Chintu,
>>>
>>>
>>>Sorry that you have been having trouble and thanks for pasting the
>>>relevant portions
>>>of push pull config to diagnose this.
>>>
>>>
>>>I think you have an issue in your RemoteFileSpecs file
>>>(TEST_AVHRR_ONLY.xml) --
>>>one quick way to debug this is to use the tool that Brian Foster wrote:
>>>
>>>
>>>(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
>>>$ cd bin
>>>$ java Djava.ext.dirs=../lib
>>>org.apache.oodt.cas.pushpull.util.ExpressionValidator
>>>
>>>
>>>It's a simple GUI that will allow you to put in regular expressions from
>>>your push pull
>>>RmoteFileSpecs files, and then put in file names from the server to see
>>>if they match.
>>>
>>>
>>>Can you run the above and let me know if it's matching? If not, we've
>>>found your issue.
>>>If it is matching, we'll move on to the next set of things to check.
>>>
>>>
>>>Thanks!
>>>
>>>
>>>Cheers,
>>>Chris
>>>
>>>
>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
>>>Date: Thursday, March 28, 2013 5:47 AM
>>>To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>>Cc: jpluser <ch...@jpl.nasa.gov>, "Ramirez, Paul M (388J)"
>>><pa...@jpl.nasa.gov>
>>>Subject: pushPull component problem
>>>
>>>
>>>
>>>>Hi,
>>>>
>>>>
>>>>I have been playing around with push pull for last two days and trying
>>>>to download some data from remote location without success.
>>>>I would appreciate if you guys can shed some light on what is going on
>>>>and why isn't push pull find files to download even though its clearly
>>>>there at remote location.
>>>>
>>>>
>>>>Here is my configuration. I am trying to download
>>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.n
>>>>c
>>>>.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine and
>>>>it lists all the files as well. However for some reason it
>>>> does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I am
>>>>doing wrong. I have tried almost all combinations and still could not
>>>>get it to work.
>>>>
>>>>
>>>>In my /etc/pushpull.properties file, I have types=false so that it
>>>>downloads the file even if its not specified in mime-types.xml file.
>>>>org.apache.oodt.cas.pushpull.allow.only.defined.types=false
>>>>
>>>>
>>>>ExternalSources.xml
>>>><sources>
>>>> <source host="eclipse.ncdc.noaa.gov">
>>>> <login type="ftp" alias="eclipseClass">
>>>> <username>anonymous</username>
>>>> <password>user@host.com</password>
>>>> </login>
>>>> </source>
>>>></sources>
>>>>
>>>>
>>>>
>>>>RemoteSpecs.xml
>>>> <daemon alias="eclipseClass" active="yes">
>>>> <runInfo firstRunDateTime="2013-03-25T00:00:00Z"
>>>>period="1m" runOnReboot="yes"/>
>>>> <propInfo
>>>>dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
>>>> <propFiles regExp="TEST_AVHRR_ONLY\.xml"
>>>>parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStruct
>>>>X
>>>>mlParser"/>
>>>> </propInfo>
>>>> <dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no"
>>>>queryElement="Filename"/>
>>>> </daemon>
>>>>
>>>>
>>>>
>>>>TEST_AVHRR_ONLY.xml
>>>><root>
>>>> <dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
>>>> <nodirs/>
>>>> <!-- <file
>>>>name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
>>>> <file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
>>>> </dirstruct>
>>>></root>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>Regards
>>>>--
>>>>Chintu Mistry
>>>>NASA Goddard Space Flight Center
>>>>Bldg L40B, Room S776
>>>>Office: 240 684 0477
>>>>Mobile: 770 310 1047
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>
Re: pushPull component problem
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Chintu,
This totally helped, thanks.
Can you take a look at the plugins here?
https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html
Try installing those, and let me know if either of those FTP
plugins works better and actually downloads the filesŠ
In the meanwhile I'll look at the logs and see what I can come up
with.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
Date: Thursday, March 28, 2013 7:12 AM
To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
<de...@oodt.apache.org>
Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
Subject: Re: pushPull component problem
>I have attached verbose log. I have added extra print statements and
>compiled to see what is returned from remote site. Hope this helps.
>
>
>--
>Chintu Mistry
>NASA Goddard Space Flight Center
>Bldg L40B, Room S776
>Office: 240 684 0477
>Mobile: 770 310 1047
>
>
>
>
>
>
>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>Date: Thursday, March 28, 2013 10:05 AM
>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
><de...@oodt.apache.org>
>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>Subject: Re: pushPull component problem
>
>
>
>Hey Chintu,
>
>
>Thanks for sending this ‹ really helped verify it's something else.
>
>
>Can you capture the log of push pull, with verbose logging turned on, and
>maybe do a pastebin?
>
>
>Cheers,
>Chris
>
>
>
>
>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
>Date: Thursday, March 28, 2013 7:03 AM
>To: jpluser <ch...@jpl.nasa.gov>, "dev@oodt.apache.org"
><de...@oodt.apache.org>
>Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
>Subject: Re: pushPull component problem
>
>
>
>>
>>
>>
>>
>>Something else is wrong. I know that the regex is fine. I don't even
>>have regex, it is straight up file name. It should match.
>>
>>
>>Thanks for looking into this.
>>--
>>Chintu Mistry
>>NASA Goddard Space Flight Center
>>Bldg L40B, Room S776
>>Office: 240 684 0477
>>Mobile: 770 310 1047
>>
>>
>>
>>
>>
>>From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>
>>Date: Thursday, March 28, 2013 9:01 AM
>>To: Chintu Mistry <ch...@nasa.gov>, "dev@oodt.apache.org"
>><de...@oodt.apache.org>
>>Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>
>>Subject: Re: pushPull component problem
>>
>>
>>
>>Hey Chintu,
>>
>>
>>Sorry that you have been having trouble and thanks for pasting the
>>relevant portions
>>of push pull config to diagnose this.
>>
>>
>>I think you have an issue in your RemoteFileSpecs file
>>(TEST_AVHRR_ONLY.xml) --
>>one quick way to debug this is to use the tool that Brian Foster wrote:
>>
>>
>>(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
>>$ cd bin
>>$ java Djava.ext.dirs=../lib
>>org.apache.oodt.cas.pushpull.util.ExpressionValidator
>>
>>
>>It's a simple GUI that will allow you to put in regular expressions from
>>your push pull
>>RmoteFileSpecs files, and then put in file names from the server to see
>>if they match.
>>
>>
>>Can you run the above and let me know if it's matching? If not, we've
>>found your issue.
>>If it is matching, we'll move on to the next set of things to check.
>>
>>
>>Thanks!
>>
>>
>>Cheers,
>>Chris
>>
>>
>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>
>>Date: Thursday, March 28, 2013 5:47 AM
>>To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>Cc: jpluser <ch...@jpl.nasa.gov>, "Ramirez, Paul M (388J)"
>><pa...@jpl.nasa.gov>
>>Subject: pushPull component problem
>>
>>
>>
>>>Hi,
>>>
>>>
>>>I have been playing around with push pull for last two days and trying
>>>to download some data from remote location without success.
>>>I would appreciate if you guys can shed some light on what is going on
>>>and why isn't push pull find files to download even though its clearly
>>>there at remote location.
>>>
>>>
>>>Here is my configuration. I am trying to download
>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc
>>>.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine and
>>>it lists all the files as well. However for some reason it
>>> does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I am
>>>doing wrong. I have tried almost all combinations and still could not
>>>get it to work.
>>>
>>>
>>>In my /etc/pushpull.properties file, I have types=false so that it
>>>downloads the file even if its not specified in mime-types.xml file.
>>>org.apache.oodt.cas.pushpull.allow.only.defined.types=false
>>>
>>>
>>>ExternalSources.xml
>>><sources>
>>> <source host="eclipse.ncdc.noaa.gov">
>>> <login type="ftp" alias="eclipseClass">
>>> <username>anonymous</username>
>>> <password>user@host.com</password>
>>> </login>
>>> </source>
>>></sources>
>>>
>>>
>>>
>>>RemoteSpecs.xml
>>> <daemon alias="eclipseClass" active="yes">
>>> <runInfo firstRunDateTime="2013-03-25T00:00:00Z"
>>>period="1m" runOnReboot="yes"/>
>>> <propInfo
>>>dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
>>> <propFiles regExp="TEST_AVHRR_ONLY\.xml"
>>>parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructX
>>>mlParser"/>
>>> </propInfo>
>>> <dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no"
>>>queryElement="Filename"/>
>>> </daemon>
>>>
>>>
>>>
>>>TEST_AVHRR_ONLY.xml
>>><root>
>>> <dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
>>> <nodirs/>
>>> <!-- <file
>>>name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
>>> <file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
>>> </dirstruct>
>>></root>
>>>
>>>
>>>
>>>
>>>
>>>Regards
>>>--
>>>Chintu Mistry
>>>NASA Goddard Space Flight Center
>>>Bldg L40B, Room S776
>>>Office: 240 684 0477
>>>Mobile: 770 310 1047
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
Re: pushPull component problem
Posted by "Mistry, Chintu (GSFC-586.0)[SGT INC]" <ch...@nasa.gov>.
I have attached verbose log. I have added extra print statements and compiled to see what is returned from remote site. Hope this helps.
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>>
Date: Thursday, March 28, 2013 10:05 AM
To: Chintu Mistry <ch...@nasa.gov>>, "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>>
Subject: Re: pushPull component problem
Hey Chintu,
Thanks for sending this — really helped verify it's something else.
Can you capture the log of push pull, with verbose logging turned on, and maybe do a pastebin?
Cheers,
Chris
From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>>
Date: Thursday, March 28, 2013 7:03 AM
To: jpluser <ch...@jpl.nasa.gov>>, "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>>
Subject: Re: pushPull component problem
[cid:63493DAB-916B-45FE-B016-391572B374E2]
[cid:0CE0F486-1CE2-490B-B015-2666D41AF8EC]
Something else is wrong. I know that the regex is fine. I don't even have regex, it is straight up file name. It should match.
Thanks for looking into this.
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>>
Date: Thursday, March 28, 2013 9:01 AM
To: Chintu Mistry <ch...@nasa.gov>>, "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>>
Subject: Re: pushPull component problem
Hey Chintu,
Sorry that you have been having trouble and thanks for pasting the relevant portions
of push pull config to diagnose this.
I think you have an issue in your RemoteFileSpecs file (TEST_AVHRR_ONLY.xml) --
one quick way to debug this is to use the tool that Brian Foster wrote:
(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
$ cd bin
$ java –Djava.ext.dirs=../lib org.apache.oodt.cas.pushpull.util.ExpressionValidator
It's a simple GUI that will allow you to put in regular expressions from your push pull
RmoteFileSpecs files, and then put in file names from the server to see if they match.
Can you run the above and let me know if it's matching? If not, we've found your issue.
If it is matching, we'll move on to the next set of things to check.
Thanks!
Cheers,
Chris
From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>>
Date: Thursday, March 28, 2013 5:47 AM
To: "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: jpluser <ch...@jpl.nasa.gov>>, "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>>
Subject: pushPull component problem
Hi,
I have been playing around with push pull for last two days and trying to download some data from remote location without success.
I would appreciate if you guys can shed some light on what is going on and why isn't push pull find files to download even though its clearly there at remote location.
Here is my configuration. I am trying to download /pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine and it lists all the files as well. However for some reason it does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I am doing wrong. I have tried almost all combinations and still could not get it to work.
In my /etc/pushpull.properties file, I have types=false so that it downloads the file even if its not specified in mime-types.xml file.
org.apache.oodt.cas.pushpull.allow.only.defined.types=false
ExternalSources.xml
<sources>
<source host="eclipse.ncdc.noaa.gov">
<login type="ftp" alias="eclipseClass">
<username>anonymous</username>
<pa...@host.com></password>
</login>
</source>
</sources>
RemoteSpecs.xml
<daemon alias="eclipseClass" active="yes">
<runInfo firstRunDateTime="2013-03-25T00:00:00Z" period="1m" runOnReboot="yes"/>
<propInfo dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
<propFiles regExp="TEST_AVHRR_ONLY\.xml" parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser"/>
</propInfo>
<dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no" queryElement="Filename"/>
</daemon>
TEST_AVHRR_ONLY.xml
<root>
<dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
<nodirs/>
<!-- <file name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
<file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
</dirstruct>
</root>
Regards
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
Re: pushPull component problem
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Chintu,
Thanks for sending this — really helped verify it's something else.
Can you capture the log of push pull, with verbose logging turned on, and maybe do a pastebin?
Cheers,
Chris
From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>>
Date: Thursday, March 28, 2013 7:03 AM
To: jpluser <ch...@jpl.nasa.gov>>, "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>>
Subject: Re: pushPull component problem
[cid:63493DAB-916B-45FE-B016-391572B374E2]
[cid:0CE0F486-1CE2-490B-B015-2666D41AF8EC]
Something else is wrong. I know that the regex is fine. I don't even have regex, it is straight up file name. It should match.
Thanks for looking into this.
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>>
Date: Thursday, March 28, 2013 9:01 AM
To: Chintu Mistry <ch...@nasa.gov>>, "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>>
Subject: Re: pushPull component problem
Hey Chintu,
Sorry that you have been having trouble and thanks for pasting the relevant portions
of push pull config to diagnose this.
I think you have an issue in your RemoteFileSpecs file (TEST_AVHRR_ONLY.xml) --
one quick way to debug this is to use the tool that Brian Foster wrote:
(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
$ cd bin
$ java –Djava.ext.dirs=../lib org.apache.oodt.cas.pushpull.util.ExpressionValidator
It's a simple GUI that will allow you to put in regular expressions from your push pull
RmoteFileSpecs files, and then put in file names from the server to see if they match.
Can you run the above and let me know if it's matching? If not, we've found your issue.
If it is matching, we'll move on to the next set of things to check.
Thanks!
Cheers,
Chris
From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>>
Date: Thursday, March 28, 2013 5:47 AM
To: "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: jpluser <ch...@jpl.nasa.gov>>, "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>>
Subject: pushPull component problem
Hi,
I have been playing around with push pull for last two days and trying to download some data from remote location without success.
I would appreciate if you guys can shed some light on what is going on and why isn't push pull find files to download even though its clearly there at remote location.
Here is my configuration. I am trying to download /pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine and it lists all the files as well. However for some reason it does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I am doing wrong. I have tried almost all combinations and still could not get it to work.
In my /etc/pushpull.properties file, I have types=false so that it downloads the file even if its not specified in mime-types.xml file.
org.apache.oodt.cas.pushpull.allow.only.defined.types=false
ExternalSources.xml
<sources>
<source host="eclipse.ncdc.noaa.gov">
<login type="ftp" alias="eclipseClass">
<username>anonymous</username>
<pa...@host.com></password>
</login>
</source>
</sources>
RemoteSpecs.xml
<daemon alias="eclipseClass" active="yes">
<runInfo firstRunDateTime="2013-03-25T00:00:00Z" period="1m" runOnReboot="yes"/>
<propInfo dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
<propFiles regExp="TEST_AVHRR_ONLY\.xml" parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser"/>
</propInfo>
<dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no" queryElement="Filename"/>
</daemon>
TEST_AVHRR_ONLY.xml
<root>
<dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
<nodirs/>
<!-- <file name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
<file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
</dirstruct>
</root>
Regards
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
Re: pushPull component problem
Posted by "Mistry, Chintu (GSFC-586.0)[SGT INC]" <ch...@nasa.gov>.
[cid:63493DAB-916B-45FE-B016-391572B374E2]
[cid:0CE0F486-1CE2-490B-B015-2666D41AF8EC]
Something else is wrong. I know that the regex is fine. I don't even have regex, it is straight up file name. It should match.
Thanks for looking into this.
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
From: <Mattmann>, Chris A <ch...@jpl.nasa.gov>>
Date: Thursday, March 28, 2013 9:01 AM
To: Chintu Mistry <ch...@nasa.gov>>, "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: "Ramirez, Paul M" <pa...@jpl.nasa.gov>>
Subject: Re: pushPull component problem
Hey Chintu,
Sorry that you have been having trouble and thanks for pasting the relevant portions
of push pull config to diagnose this.
I think you have an issue in your RemoteFileSpecs file (TEST_AVHRR_ONLY.xml) --
one quick way to debug this is to use the tool that Brian Foster wrote:
(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
$ cd bin
$ java –Djava.ext.dirs=../lib org.apache.oodt.cas.pushpull.util.ExpressionValidator
It's a simple GUI that will allow you to put in regular expressions from your push pull
RmoteFileSpecs files, and then put in file names from the server to see if they match.
Can you run the above and let me know if it's matching? If not, we've found your issue.
If it is matching, we'll move on to the next set of things to check.
Thanks!
Cheers,
Chris
From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>>
Date: Thursday, March 28, 2013 5:47 AM
To: "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: jpluser <ch...@jpl.nasa.gov>>, "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>>
Subject: pushPull component problem
Hi,
I have been playing around with push pull for last two days and trying to download some data from remote location without success.
I would appreciate if you guys can shed some light on what is going on and why isn't push pull find files to download even though its clearly there at remote location.
Here is my configuration. I am trying to download /pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine and it lists all the files as well. However for some reason it does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I am doing wrong. I have tried almost all combinations and still could not get it to work.
In my /etc/pushpull.properties file, I have types=false so that it downloads the file even if its not specified in mime-types.xml file.
org.apache.oodt.cas.pushpull.allow.only.defined.types=false
ExternalSources.xml
<sources>
<source host="eclipse.ncdc.noaa.gov">
<login type="ftp" alias="eclipseClass">
<username>anonymous</username>
<pa...@host.com></password>
</login>
</source>
</sources>
RemoteSpecs.xml
<daemon alias="eclipseClass" active="yes">
<runInfo firstRunDateTime="2013-03-25T00:00:00Z" period="1m" runOnReboot="yes"/>
<propInfo dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
<propFiles regExp="TEST_AVHRR_ONLY\.xml" parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser"/>
</propInfo>
<dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no" queryElement="Filename"/>
</daemon>
TEST_AVHRR_ONLY.xml
<root>
<dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
<nodirs/>
<!-- <file name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
<file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
</dirstruct>
</root>
Regards
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
Re: pushPull component problem
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Chintu,
Sorry that you have been having trouble and thanks for pasting the relevant portions
of push pull config to diagnose this.
I think you have an issue in your RemoteFileSpecs file (TEST_AVHRR_ONLY.xml) --
one quick way to debug this is to use the tool that Brian Foster wrote:
(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
$ cd bin
$ java –Djava.ext.dirs=../lib org.apache.oodt.cas.pushpull.util.ExpressionValidator
It's a simple GUI that will allow you to put in regular expressions from your push pull
RmoteFileSpecs files, and then put in file names from the server to see if they match.
Can you run the above and let me know if it's matching? If not, we've found your issue.
If it is matching, we'll move on to the next set of things to check.
Thanks!
Cheers,
Chris
From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <ch...@nasa.gov>>
Date: Thursday, March 28, 2013 5:47 AM
To: "dev@oodt.apache.org<ma...@oodt.apache.org>" <de...@oodt.apache.org>>
Cc: jpluser <ch...@jpl.nasa.gov>>, "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>>
Subject: pushPull component problem
Hi,
I have been playing around with push pull for last two days and trying to download some data from remote location without success.
I would appreciate if you guys can shed some light on what is going on and why isn't push pull find files to download even though its clearly there at remote location.
Here is my configuration. I am trying to download /pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine and it lists all the files as well. However for some reason it does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I am doing wrong. I have tried almost all combinations and still could not get it to work.
In my /etc/pushpull.properties file, I have types=false so that it downloads the file even if its not specified in mime-types.xml file.
org.apache.oodt.cas.pushpull.allow.only.defined.types=false
ExternalSources.xml
<sources>
<source host="eclipse.ncdc.noaa.gov">
<login type="ftp" alias="eclipseClass">
<username>anonymous</username>
<pa...@host.com></password>
</login>
</source>
</sources>
RemoteSpecs.xml
<daemon alias="eclipseClass" active="yes">
<runInfo firstRunDateTime="2013-03-25T00:00:00Z" period="1m" runOnReboot="yes"/>
<propInfo dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
<propFiles regExp="TEST_AVHRR_ONLY\.xml" parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser"/>
</propInfo>
<dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no" queryElement="Filename"/>
</daemon>
TEST_AVHRR_ONLY.xml
<root>
<dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
<nodirs/>
<!-- <file name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
<file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
</dirstruct>
</root>
Regards
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047