You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Kenneth Tyler <ke...@8thfold.com> on 2010/07/26 18:49:13 UTC
scheduled tasks
is there any way to run a scheduled task in couchdb?
i'd like to create a bunch of documents that reresent data in records
that need to be updated... without having to do the update immediately
thanks
ken tyler
Re: scheduled tasks
Posted by Sivan Greenberg <si...@omniqueue.com>.
I shall also be waiting for this, as I want to be able to run a
garbage collection process for session documents from within CouchDB.
Staying tuned....
-Sivan
On Mon, Jul 26, 2010 at 8:18 PM, J Chris Anderson <jc...@apache.org> wrote:
>
> On Jul 26, 2010, at 9:49 AM, Kenneth Tyler wrote:
>
>> is there any way to run a scheduled task in couchdb?
>>
>
> There is a technique which uses the changes feed. I believe Jason from Couchio is working on an in-depth blog post about it, which this channel there should be some info later this week:
>
> http://blog.couch.io/
>
>> i'd like to create a bunch of documents that reresent data in records
>> that need to be updated... without having to do the update immediately
>>
>> thanks
>>
>> ken tyler
>
>
Re: scheduled tasks
Posted by Jason Smith <jh...@couch.io>.
Haha, speak of the Devil.
You were right. There is pent-up desire to do stuff like this.
On Tue, Jul 27, 2010 at 01:18, J Chris Anderson <jc...@apache.org> wrote:
>
> On Jul 26, 2010, at 9:49 AM, Kenneth Tyler wrote:
>
> > is there any way to run a scheduled task in couchdb?
> >
>
> There is a technique which uses the changes feed. I believe Jason from
> Couchio is working on an in-depth blog post about it, which this channel
> there should be some info later this week:
>
> http://blog.couch.io/
>
> > i'd like to create a bunch of documents that reresent data in records
> > that need to be updated... without having to do the update immediately
> >
> > thanks
> >
> > ken tyler
>
>
--
Jason Smith
Couchio Hosting
Re: scheduled tasks
Posted by J Chris Anderson <jc...@apache.org>.
On Jul 26, 2010, at 9:49 AM, Kenneth Tyler wrote:
> is there any way to run a scheduled task in couchdb?
>
There is a technique which uses the changes feed. I believe Jason from Couchio is working on an in-depth blog post about it, which this channel there should be some info later this week:
http://blog.couch.io/
> i'd like to create a bunch of documents that reresent data in records
> that need to be updated... without having to do the update immediately
>
> thanks
>
> ken tyler
Re: scheduled tasks
Posted by J Chris Anderson <jc...@apache.org>.
On Jul 26, 2010, at 2:28 PM, Mark J. Reed wrote:
> The "fun and ease of Python" is not at issue here. Glad you're having
> fun programming in your favorite language, but it's not as if you can
> just do 'from scheduler import magic' and be done.
>
> The point is, cron didn't work for you because you wanted sub-minute
> precision and automatic overlap prevention.
> For folks who don't need those things, cron is a viable solution.
> Either way, whether you're writing your own scheduler or writing the
> scripts to be run out of one, you can use whatever language you like;
> one of the goals of Couch is to be language-agnostic. As long as you
> have JSON and HTTP libraries for it, you can even code in Fortran...
>
This is a good way to keep a python program running and reacting to couchdb updates:
http://blog.fupps.com/2010/05/10/update-notifications-in-couchdb-tweeting-urgent-documents/
>
> On Mon, Jul 26, 2010 at 5:10 PM, Sivan Greenberg <si...@omniqueue.com> wrote:
>> Hi Nils,
>>
>> I am using Python to have 2 separate background process that:
>> 1) Replication connections are working/ restart/ fix if needed
>> (including wrapper SSH tunnels).
>> 2) Conflict resolution that is triggered by the _changes feed,
>> asynchronous and parallelized.
>>
>> I've also used CouchDBKit to manage CouchDB from Python.
>>
>> Doing the response/fix logic in a cron triggered script proved
>> problematic for me, so I went and added my own scheduling bits, on the
>> way enabling sub-minute precision quite easily, attributing to the fun
>> and ease of Python. (Scheduling being mostly while loops with
>> configurable sleep intervals, at its simplest form, and time/date
>> based sleeps)
>>
>> Being able to suspend interval triggered execution until a previous
>> run came back, without having to use locking or flag files and such as
>> when using cron, was a breeze.
>>
>> I guess that is "different" not better perhaps.
>>
>> -Sivan
>>
>> On Mon, Jul 26, 2010 at 11:57 PM, Nils Breunese <N....@vpro.nl> wrote:
>>> Python is a programming language, not a scheduler. (Although I'm sure there are scheduler libraries and things like that for Python.) What do you mean by 'works better than cron'? Yes, if you need sub-minute precision you'll need to look elsewhere since cron doesn't support that, but otherwise I wouldn't go and write code to do what a scheduler like cron already provides (and it's usually already running anyway).
>>>
>>> We're actually using a Python script (using couchdbkit) run by cron for our CouchDB maintenance needs.
>>>
>>> Nils.
>>> _nn_______________________________________
>>> Van: sivang@gmail.com [sivang@gmail.com] namens Sivan Greenberg [sivan@omniqueue.com]
>>> Verzonden: maandag 26 juli 2010 22:26
>>> Aan: user@couchdb.apache.org
>>> Onderwerp: Re: scheduled tasks
>>>
>>> I am actually using python to do that in my specific project. Works
>>> better than cron and allows finer granularity of intervals.
>>>
>>> Sivan
>>>
>>> 2010/7/26 Nils Breunese <N....@vpro.nl>:
>>>> I don't think so. Until there is I guess you could use an external scheduler like cron?
>>>>
>>>> Nils.
>>>> ________________________________________
>>>> Van: Kenneth Tyler [ken@8thfold.com]
>>>> Verzonden: maandag 26 juli 2010 18:49
>>>> Aan: user@couchdb.apache.org
>>>> Onderwerp: scheduled tasks
>>>>
>>>> is there any way to run a scheduled task in couchdb?
>>>>
>>>> i'd like to create a bunch of documents that reresent data in records
>>>> that need to be updated... without having to do the update immediately
>>>>
>>>> thanks
>>>>
>>>> ken tyler
>>>>
>>>> De informatie vervat in deze e-mail en meegezonden bijlagen is uitsluitend bedoeld voor gebruik door de geadresseerde en kan vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van deze informatie aan derden is voorbehouden aan geadresseerde. De VPRO staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mail, noch voor tijdige ontvangst daarvan.
>>>>
>>>
>>> De informatie vervat in deze e-mail en meegezonden bijlagen is uitsluitend bedoeld voor gebruik door de geadresseerde en kan vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van deze informatie aan derden is voorbehouden aan geadresseerde. De VPRO staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mail, noch voor tijdige ontvangst daarvan.
>>>
>>
>
>
>
> --
> Mark J. Reed <ma...@gmail.com>
Re: scheduled tasks
Posted by "Mark J. Reed" <ma...@gmail.com>.
The "fun and ease of Python" is not at issue here. Glad you're having
fun programming in your favorite language, but it's not as if you can
just do 'from scheduler import magic' and be done.
The point is, cron didn't work for you because you wanted sub-minute
precision and automatic overlap prevention.
For folks who don't need those things, cron is a viable solution.
Either way, whether you're writing your own scheduler or writing the
scripts to be run out of one, you can use whatever language you like;
one of the goals of Couch is to be language-agnostic. As long as you
have JSON and HTTP libraries for it, you can even code in Fortran...
On Mon, Jul 26, 2010 at 5:10 PM, Sivan Greenberg <si...@omniqueue.com> wrote:
> Hi Nils,
>
> I am using Python to have 2 separate background process that:
> 1) Replication connections are working/ restart/ fix if needed
> (including wrapper SSH tunnels).
> 2) Conflict resolution that is triggered by the _changes feed,
> asynchronous and parallelized.
>
> I've also used CouchDBKit to manage CouchDB from Python.
>
> Doing the response/fix logic in a cron triggered script proved
> problematic for me, so I went and added my own scheduling bits, on the
> way enabling sub-minute precision quite easily, attributing to the fun
> and ease of Python. (Scheduling being mostly while loops with
> configurable sleep intervals, at its simplest form, and time/date
> based sleeps)
>
> Being able to suspend interval triggered execution until a previous
> run came back, without having to use locking or flag files and such as
> when using cron, was a breeze.
>
> I guess that is "different" not better perhaps.
>
> -Sivan
>
> On Mon, Jul 26, 2010 at 11:57 PM, Nils Breunese <N....@vpro.nl> wrote:
>> Python is a programming language, not a scheduler. (Although I'm sure there are scheduler libraries and things like that for Python.) What do you mean by 'works better than cron'? Yes, if you need sub-minute precision you'll need to look elsewhere since cron doesn't support that, but otherwise I wouldn't go and write code to do what a scheduler like cron already provides (and it's usually already running anyway).
>>
>> We're actually using a Python script (using couchdbkit) run by cron for our CouchDB maintenance needs.
>>
>> Nils.
>> _nn_______________________________________
>> Van: sivang@gmail.com [sivang@gmail.com] namens Sivan Greenberg [sivan@omniqueue.com]
>> Verzonden: maandag 26 juli 2010 22:26
>> Aan: user@couchdb.apache.org
>> Onderwerp: Re: scheduled tasks
>>
>> I am actually using python to do that in my specific project. Works
>> better than cron and allows finer granularity of intervals.
>>
>> Sivan
>>
>> 2010/7/26 Nils Breunese <N....@vpro.nl>:
>>> I don't think so. Until there is I guess you could use an external scheduler like cron?
>>>
>>> Nils.
>>> ________________________________________
>>> Van: Kenneth Tyler [ken@8thfold.com]
>>> Verzonden: maandag 26 juli 2010 18:49
>>> Aan: user@couchdb.apache.org
>>> Onderwerp: scheduled tasks
>>>
>>> is there any way to run a scheduled task in couchdb?
>>>
>>> i'd like to create a bunch of documents that reresent data in records
>>> that need to be updated... without having to do the update immediately
>>>
>>> thanks
>>>
>>> ken tyler
>>>
>>> De informatie vervat in deze e-mail en meegezonden bijlagen is uitsluitend bedoeld voor gebruik door de geadresseerde en kan vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van deze informatie aan derden is voorbehouden aan geadresseerde. De VPRO staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mail, noch voor tijdige ontvangst daarvan.
>>>
>>
>> De informatie vervat in deze e-mail en meegezonden bijlagen is uitsluitend bedoeld voor gebruik door de geadresseerde en kan vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van deze informatie aan derden is voorbehouden aan geadresseerde. De VPRO staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mail, noch voor tijdige ontvangst daarvan.
>>
>
--
Mark J. Reed <ma...@gmail.com>
Re: scheduled tasks
Posted by Sivan Greenberg <si...@omniqueue.com>.
Hi Nils,
I am using Python to have 2 separate background process that:
1) Replication connections are working/ restart/ fix if needed
(including wrapper SSH tunnels).
2) Conflict resolution that is triggered by the _changes feed,
asynchronous and parallelized.
I've also used CouchDBKit to manage CouchDB from Python.
Doing the response/fix logic in a cron triggered script proved
problematic for me, so I went and added my own scheduling bits, on the
way enabling sub-minute precision quite easily, attributing to the fun
and ease of Python. (Scheduling being mostly while loops with
configurable sleep intervals, at its simplest form, and time/date
based sleeps)
Being able to suspend interval triggered execution until a previous
run came back, without having to use locking or flag files and such as
when using cron, was a breeze.
I guess that is "different" not better perhaps.
-Sivan
On Mon, Jul 26, 2010 at 11:57 PM, Nils Breunese <N....@vpro.nl> wrote:
> Python is a programming language, not a scheduler. (Although I'm sure there are scheduler libraries and things like that for Python.) What do you mean by 'works better than cron'? Yes, if you need sub-minute precision you'll need to look elsewhere since cron doesn't support that, but otherwise I wouldn't go and write code to do what a scheduler like cron already provides (and it's usually already running anyway).
>
> We're actually using a Python script (using couchdbkit) run by cron for our CouchDB maintenance needs.
>
> Nils.
> _nn_______________________________________
> Van: sivang@gmail.com [sivang@gmail.com] namens Sivan Greenberg [sivan@omniqueue.com]
> Verzonden: maandag 26 juli 2010 22:26
> Aan: user@couchdb.apache.org
> Onderwerp: Re: scheduled tasks
>
> I am actually using python to do that in my specific project. Works
> better than cron and allows finer granularity of intervals.
>
> Sivan
>
> 2010/7/26 Nils Breunese <N....@vpro.nl>:
>> I don't think so. Until there is I guess you could use an external scheduler like cron?
>>
>> Nils.
>> ________________________________________
>> Van: Kenneth Tyler [ken@8thfold.com]
>> Verzonden: maandag 26 juli 2010 18:49
>> Aan: user@couchdb.apache.org
>> Onderwerp: scheduled tasks
>>
>> is there any way to run a scheduled task in couchdb?
>>
>> i'd like to create a bunch of documents that reresent data in records
>> that need to be updated... without having to do the update immediately
>>
>> thanks
>>
>> ken tyler
>>
>> De informatie vervat in deze e-mail en meegezonden bijlagen is uitsluitend bedoeld voor gebruik door de geadresseerde en kan vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van deze informatie aan derden is voorbehouden aan geadresseerde. De VPRO staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mail, noch voor tijdige ontvangst daarvan.
>>
>
> De informatie vervat in deze e-mail en meegezonden bijlagen is uitsluitend bedoeld voor gebruik door de geadresseerde en kan vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van deze informatie aan derden is voorbehouden aan geadresseerde. De VPRO staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mail, noch voor tijdige ontvangst daarvan.
>
Re: scheduled tasks
Posted by Jason Smith <jh...@couch.io>.
On Tue, Jul 27, 2010 at 03:57, Nils Breunese <N....@vpro.nl> wrote:
> Python is a programming language, not a scheduler. (Although I'm sure there
> are scheduler libraries and things like that for Python.) What do you mean
> by 'works better than cron'? Yes, if you need sub-minute precision you'll
> need to look elsewhere since cron doesn't support that, but otherwise I
> wouldn't go and write code to do what a scheduler like cron already provides
> (and it's usually already running anyway).
>
I have worked a ton with cron as a sysadmin. I have also worked with
BackgrounDRb which FWIW I like.
People take for granted how hard it is to write a stable job runner. Cron is
ubiquitous and mature and well-understood by programmers and sysadmins.
The one thing against cron is that it is a little out-of-band from your
software's normal code. When installing, you have to remember to hook into
cron. Eventually all of my projects have dedicated code just to *install*
other code into cron.
Finally, the CouchDB _changes feed is not directly related to job scheduling
and periodic tasks. However it can be used to implement them. The
implementation (which, yes, I am writing up in the Couchio blog) is robust
for the same reason Erlang is robust: loosely-coupled components
independently send messages to each other.
--
Jason Smith
Couchio Hosting
RE: scheduled tasks
Posted by Nils Breunese <N....@vpro.nl>.
Python is a programming language, not a scheduler. (Although I'm sure there are scheduler libraries and things like that for Python.) What do you mean by 'works better than cron'? Yes, if you need sub-minute precision you'll need to look elsewhere since cron doesn't support that, but otherwise I wouldn't go and write code to do what a scheduler like cron already provides (and it's usually already running anyway).
We're actually using a Python script (using couchdbkit) run by cron for our CouchDB maintenance needs.
Nils.
________________________________________
Van: sivang@gmail.com [sivang@gmail.com] namens Sivan Greenberg [sivan@omniqueue.com]
Verzonden: maandag 26 juli 2010 22:26
Aan: user@couchdb.apache.org
Onderwerp: Re: scheduled tasks
I am actually using python to do that in my specific project. Works
better than cron and allows finer granularity of intervals.
Sivan
2010/7/26 Nils Breunese <N....@vpro.nl>:
> I don't think so. Until there is I guess you could use an external scheduler like cron?
>
> Nils.
> ________________________________________
> Van: Kenneth Tyler [ken@8thfold.com]
> Verzonden: maandag 26 juli 2010 18:49
> Aan: user@couchdb.apache.org
> Onderwerp: scheduled tasks
>
> is there any way to run a scheduled task in couchdb?
>
> i'd like to create a bunch of documents that reresent data in records
> that need to be updated... without having to do the update immediately
>
> thanks
>
> ken tyler
>
> De informatie vervat in deze e-mail en meegezonden bijlagen is uitsluitend bedoeld voor gebruik door de geadresseerde en kan vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van deze informatie aan derden is voorbehouden aan geadresseerde. De VPRO staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mail, noch voor tijdige ontvangst daarvan.
>
De informatie vervat in deze e-mail en meegezonden bijlagen is uitsluitend bedoeld voor gebruik door de geadresseerde en kan vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van deze informatie aan derden is voorbehouden aan geadresseerde. De VPRO staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mail, noch voor tijdige ontvangst daarvan.
Re: scheduled tasks
Posted by Sivan Greenberg <si...@omniqueue.com>.
I am actually using python to do that in my specific project. Works
better than cron and allows finer granularity of intervals.
Sivan
2010/7/26 Nils Breunese <N....@vpro.nl>:
> I don't think so. Until there is I guess you could use an external scheduler like cron?
>
> Nils.
> ________________________________________
> Van: Kenneth Tyler [ken@8thfold.com]
> Verzonden: maandag 26 juli 2010 18:49
> Aan: user@couchdb.apache.org
> Onderwerp: scheduled tasks
>
> is there any way to run a scheduled task in couchdb?
>
> i'd like to create a bunch of documents that reresent data in records
> that need to be updated... without having to do the update immediately
>
> thanks
>
> ken tyler
>
> De informatie vervat in deze e-mail en meegezonden bijlagen is uitsluitend bedoeld voor gebruik door de geadresseerde en kan vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van deze informatie aan derden is voorbehouden aan geadresseerde. De VPRO staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mail, noch voor tijdige ontvangst daarvan.
>
RE: scheduled tasks
Posted by Nils Breunese <N....@vpro.nl>.
I don't think so. Until there is I guess you could use an external scheduler like cron?
Nils.
________________________________________
Van: Kenneth Tyler [ken@8thfold.com]
Verzonden: maandag 26 juli 2010 18:49
Aan: user@couchdb.apache.org
Onderwerp: scheduled tasks
is there any way to run a scheduled task in couchdb?
i'd like to create a bunch of documents that reresent data in records
that need to be updated... without having to do the update immediately
thanks
ken tyler
De informatie vervat in deze e-mail en meegezonden bijlagen is uitsluitend bedoeld voor gebruik door de geadresseerde en kan vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van deze informatie aan derden is voorbehouden aan geadresseerde. De VPRO staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mail, noch voor tijdige ontvangst daarvan.