You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Lewis John Mcgibbney <le...@gmail.com> on 2014/11/01 08:56:56 UTC
Why is pushpull not always working?
Hi Folks,
I really couldn't think of a better title for this thread so I apologize in
advance.
I recently set up pushpull with FTP to crawl remote ftp site [0] which is
publicly available.
I am getting the following
INFO: Launching Daemons . . .
Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon
registerRMIServer
INFO: Created Daemon ID = 90121 on RMI registry port 9012
Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.DaemonLauncher
launchDaemons
INFO: Creating Daemon with ID = 1
Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
setAsRunning
INFO: Daemon with ID = 90121 was given permission to run
Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
until Sat Nov 01 00:48:00 PDT 2014
Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
setAsRunning
INFO: Daemon with ID = 90121 was given permission to run
Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
until Sat Nov 01 00:51:00 PDT 2014
Nov 01, 2014 12:50:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
setAsRunning
INFO: Daemon with ID = 90121 was given permission to run
Nov 01, 2014 12:51:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
until Sat Nov 01 00:54:00 PDT 2014
Nov 01, 2014 12:53:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
setAsRunning
INFO: Daemon with ID = 90121 was given permission to run
Nov 01, 2014 12:54:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
until Sat Nov 01 00:57:00 PDT 2014
No errors... no pulling... no nothing!
I also came across this FAQ (thanks Cam/Chris) [1] leading to Chris'
plugins page [2].
My question is simple... why is pushpull known NOT to work on occasion? I
am going to debug this tomorrow but I thought I would ask this
none-the-less.
Thanks
Lewis
[0] ftp://n5eil01u.ecs.nsidc.org/SAN/AQUARIUS/AQ3_SSS.004/
[1]
https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide#OODTPush-PullUserGuide-Nodatafileisdownloadedtomystagingdirectoryafterrunningthe./pushpullscript.WhatshouldIdo
?
[2] https://cwiki.apache.org/confluence/display/OODT/OODT+Push+Pull+Plugins
--
*Lewis*
Re: Why is pushpull not always working?
Posted by Cameron Goodale <si...@gmail.com>.
Hey Lewis,
Last time I dug into this I was running toward the "Active vs. Passive" FTP
settings, but I am not sure that is the root cause. One option or fix
would be to add in a timeout parameter, where if the pushpull daemon
doesn't get a reply in a set amount of time, it should stop and restart.
Every time I have seen pushpull stall or hang, a simple stop and restart
gets it working again.
I know Ross Laidlaw is using pushpull on a his Near Real Time snow
processing pipeline, so I am sure he will be very interested in your work
Lewis.
Thanks,
Cameron
On Sat, Nov 1, 2014 at 10:58 AM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Thanks Chris. OK doke. i am going to debug this today and find out what is
> happening.
> Thanks for heads up.
> Lewis
>
> On Sat, Nov 1, 2014 at 10:31 AM, Chris Mattmann <ch...@gmail.com>
> wrote:
>
> > Thanks Lewis - in my mind it boils down somewhat to
> > the protocols that are implemented and the way they
> > behave with FTP servers and so forth. Ultimately, we
> > should be using the ones that are in PushPull plugins,
> > but they are virally licensed and I haven¹t found a
> > better alternative.
> >
> > Another @bfoster update here would be great, Brian?
> >
> > Cheers,
> > Chris
> >
> >
> > ------------------------
> > Chris Mattmann
> > chris.mattmann@gmail.com
> >
> >
> >
> >
> > -----Original Message-----
> > From: Lewis John Mcgibbney <le...@gmail.com>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Saturday, November 1, 2014 at 12:56 AM
> > To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> > Subject: Why is pushpull not always working?
> >
> > >Hi Folks,
> > >I really couldn't think of a better title for this thread so I apologize
> > >in
> > >advance.
> > >I recently set up pushpull with FTP to crawl remote ftp site [0] which
> is
> > >publicly available.
> > >I am getting the following
> > >
> > >INFO: Launching Daemons . . .
> > >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> > >registerRMIServer
> > >INFO: Created Daemon ID = 90121 on RMI registry port 9012
> > >Nov 01, 2014 12:46:01 AM
> > >org.apache.oodt.cas.pushpull.daemon.DaemonLauncher
> > >launchDaemons
> > >INFO: Creating Daemon with ID = 1
> > >Nov 01, 2014 12:46:01 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonManager
> > >setAsRunning
> > >INFO: Daemon with ID = 90121 was given permission to run
> > >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> sleep
> > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> > >until Sat Nov 01 00:48:00 PDT 2014
> > >Nov 01, 2014 12:48:00 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonManager
> > >setAsRunning
> > >INFO: Daemon with ID = 90121 was given permission to run
> > >Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> sleep
> > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> > >until Sat Nov 01 00:51:00 PDT 2014
> > >Nov 01, 2014 12:50:59 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonManager
> > >setAsRunning
> > >INFO: Daemon with ID = 90121 was given permission to run
> > >Nov 01, 2014 12:51:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> sleep
> > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> > >until Sat Nov 01 00:54:00 PDT 2014
> > >Nov 01, 2014 12:53:59 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonManager
> > >setAsRunning
> > >INFO: Daemon with ID = 90121 was given permission to run
> > >Nov 01, 2014 12:54:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> sleep
> > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> > >until Sat Nov 01 00:57:00 PDT 2014
> > >
> > >No errors... no pulling... no nothing!
> > >I also came across this FAQ (thanks Cam/Chris) [1] leading to Chris'
> > >plugins page [2].
> > >My question is simple... why is pushpull known NOT to work on occasion?
> I
> > >am going to debug this tomorrow but I thought I would ask this
> > >none-the-less.
> > >Thanks
> > >Lewis
> > >
> > >[0] ftp://n5eil01u.ecs.nsidc.org/SAN/AQUARIUS/AQ3_SSS.004/
> > >[1]
> > >
> >
> https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide
> >
> >#OODTPush-PullUserGuide-Nodatafileisdownloadedtomystagingdirectoryafterrun
> > >ningthe./pushpullscript.WhatshouldIdo
> > >?
> > >[2]
> > >https://cwiki.apache.org/confluence/display/OODT/OODT+Push+Pull+Plugins
> > >
> > >
> > >--
> > >*Lewis*
> >
> >
> >
>
>
> --
> *Lewis*
>
--
Sent from a Tin Can attached to a String
Re: Why is pushpull not always working?
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Thanks Chris. OK doke. i am going to debug this today and find out what is
happening.
Thanks for heads up.
Lewis
On Sat, Nov 1, 2014 at 10:31 AM, Chris Mattmann <ch...@gmail.com>
wrote:
> Thanks Lewis - in my mind it boils down somewhat to
> the protocols that are implemented and the way they
> behave with FTP servers and so forth. Ultimately, we
> should be using the ones that are in PushPull plugins,
> but they are virally licensed and I haven¹t found a
> better alternative.
>
> Another @bfoster update here would be great, Brian?
>
> Cheers,
> Chris
>
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Lewis John Mcgibbney <le...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Saturday, November 1, 2014 at 12:56 AM
> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: Why is pushpull not always working?
>
> >Hi Folks,
> >I really couldn't think of a better title for this thread so I apologize
> >in
> >advance.
> >I recently set up pushpull with FTP to crawl remote ftp site [0] which is
> >publicly available.
> >I am getting the following
> >
> >INFO: Launching Daemons . . .
> >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> >registerRMIServer
> >INFO: Created Daemon ID = 90121 on RMI registry port 9012
> >Nov 01, 2014 12:46:01 AM
> >org.apache.oodt.cas.pushpull.daemon.DaemonLauncher
> >launchDaemons
> >INFO: Creating Daemon with ID = 1
> >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
> >setAsRunning
> >INFO: Daemon with ID = 90121 was given permission to run
> >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
> >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> >until Sat Nov 01 00:48:00 PDT 2014
> >Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
> >setAsRunning
> >INFO: Daemon with ID = 90121 was given permission to run
> >Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
> >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> >until Sat Nov 01 00:51:00 PDT 2014
> >Nov 01, 2014 12:50:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
> >setAsRunning
> >INFO: Daemon with ID = 90121 was given permission to run
> >Nov 01, 2014 12:51:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
> >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> >until Sat Nov 01 00:54:00 PDT 2014
> >Nov 01, 2014 12:53:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
> >setAsRunning
> >INFO: Daemon with ID = 90121 was given permission to run
> >Nov 01, 2014 12:54:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
> >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> >until Sat Nov 01 00:57:00 PDT 2014
> >
> >No errors... no pulling... no nothing!
> >I also came across this FAQ (thanks Cam/Chris) [1] leading to Chris'
> >plugins page [2].
> >My question is simple... why is pushpull known NOT to work on occasion? I
> >am going to debug this tomorrow but I thought I would ask this
> >none-the-less.
> >Thanks
> >Lewis
> >
> >[0] ftp://n5eil01u.ecs.nsidc.org/SAN/AQUARIUS/AQ3_SSS.004/
> >[1]
> >
> https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide
> >#OODTPush-PullUserGuide-Nodatafileisdownloadedtomystagingdirectoryafterrun
> >ningthe./pushpullscript.WhatshouldIdo
> >?
> >[2]
> >https://cwiki.apache.org/confluence/display/OODT/OODT+Push+Pull+Plugins
> >
> >
> >--
> >*Lewis*
>
>
>
--
*Lewis*
Re: Why is pushpull not always working?
Posted by Chris Mattmann <ch...@gmail.com>.
Thanks Lewis - in my mind it boils down somewhat to
the protocols that are implemented and the way they
behave with FTP servers and so forth. Ultimately, we
should be using the ones that are in PushPull plugins,
but they are virally licensed and I haven¹t found a
better alternative.
Another @bfoster update here would be great, Brian?
Cheers,
Chris
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Lewis John Mcgibbney <le...@gmail.com>
Reply-To: <de...@oodt.apache.org>
Date: Saturday, November 1, 2014 at 12:56 AM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: Why is pushpull not always working?
>Hi Folks,
>I really couldn't think of a better title for this thread so I apologize
>in
>advance.
>I recently set up pushpull with FTP to crawl remote ftp site [0] which is
>publicly available.
>I am getting the following
>
>INFO: Launching Daemons . . .
>Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon
>registerRMIServer
>INFO: Created Daemon ID = 90121 on RMI registry port 9012
>Nov 01, 2014 12:46:01 AM
>org.apache.oodt.cas.pushpull.daemon.DaemonLauncher
>launchDaemons
>INFO: Creating Daemon with ID = 1
>Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
>setAsRunning
>INFO: Daemon with ID = 90121 was given permission to run
>Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
>until Sat Nov 01 00:48:00 PDT 2014
>Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
>setAsRunning
>INFO: Daemon with ID = 90121 was given permission to run
>Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
>until Sat Nov 01 00:51:00 PDT 2014
>Nov 01, 2014 12:50:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
>setAsRunning
>INFO: Daemon with ID = 90121 was given permission to run
>Nov 01, 2014 12:51:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
>until Sat Nov 01 00:54:00 PDT 2014
>Nov 01, 2014 12:53:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
>setAsRunning
>INFO: Daemon with ID = 90121 was given permission to run
>Nov 01, 2014 12:54:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
>until Sat Nov 01 00:57:00 PDT 2014
>
>No errors... no pulling... no nothing!
>I also came across this FAQ (thanks Cam/Chris) [1] leading to Chris'
>plugins page [2].
>My question is simple... why is pushpull known NOT to work on occasion? I
>am going to debug this tomorrow but I thought I would ask this
>none-the-less.
>Thanks
>Lewis
>
>[0] ftp://n5eil01u.ecs.nsidc.org/SAN/AQUARIUS/AQ3_SSS.004/
>[1]
>https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide
>#OODTPush-PullUserGuide-Nodatafileisdownloadedtomystagingdirectoryafterrun
>ningthe./pushpullscript.WhatshouldIdo
>?
>[2]
>https://cwiki.apache.org/confluence/display/OODT/OODT+Push+Pull+Plugins
>
>
>--
>*Lewis*