how to run jobs every 30 minutes?

[edward choi <mp...@gmail.com>]

Hi,

I'm planning to crawl a certain web site every 30 minutes.
How would I get it done in Hadoop?

In pure Java, I used Thread.sleep() method, but I guess this won't work in
Hadoop.
Or if it could work, could anyone show me an example?

Ed.

Re: how to run jobs every 30 minutes?

[edward choi <mp...@gmail.com>]

Thanks for the reply, but I was hoping to load the program on the memory all
the time:-)

2010/12/7 li ping <li...@gmail.com>

> submit the job every 30 mins.
>
> On Tue, Dec 7, 2010 at 4:55 PM, edward choi <mp...@gmail.com> wrote:
>
> > Hi,
> >
> > I'm planning to crawl a certain web site every 30 minutes.
> > How would I get it done in Hadoop?
> >
> > In pure Java, I used Thread.sleep() method, but I guess this won't work
> in
> > Hadoop.
> > Or if it could work, could anyone show me an example?
> >
> > Ed.
> >
>
>
>
> --
> -----李平
>

Re: how to run jobs every 30 minutes?

[li ping <li...@gmail.com>]

submit the job every 30 mins.

On Tue, Dec 7, 2010 at 4:55 PM, edward choi <mp...@gmail.com> wrote:

> Hi,
>
> I'm planning to crawl a certain web site every 30 minutes.
> How would I get it done in Hadoop?
>
> In pure Java, I used Thread.sleep() method, but I guess this won't work in
> Hadoop.
> Or if it could work, could anyone show me an example?
>
> Ed.
>



-- 
-----李平

Re: how to run jobs every 30 minutes?

[edward choi <mp...@gmail.com>]

This one doesn't seem so complex for even a newbie like myself. Thanks!!!

2010/12/14 Ted Dunning <td...@maprtech.com>

> Or even simpler, try Azkaban: http://sna-projects.com/azkaban/
>
> On Mon, Dec 13, 2010 at 9:26 PM, edward choi <mp...@gmail.com> wrote:
>
> > Thanks for the tip. I took a look at it.
> > Looks similar to Cascading I guess...?
> > Anyway thanks for the info!!
> >
> > Ed
> >
> > 2010/12/8 Alejandro Abdelnur <tu...@cloudera.com>
> >
> > > Or, if you want to do it in a reliable way you could use an Oozie
> > > coordinator job.
> > >
> > > On Wed, Dec 8, 2010 at 1:53 PM, edward choi <mp...@gmail.com> wrote:
> > > > My mistake. Come to think about it, you are right, I can just make an
> > > > infinite loop inside the Hadoop application.
> > > > Thanks for the reply.
> > > >
> > > > 2010/12/7 Harsh J <qw...@gmail.com>
> > > >
> > > >> Hi,
> > > >>
> > > >> On Tue, Dec 7, 2010 at 2:25 PM, edward choi <mp...@gmail.com>
> wrote:
> > > >> > Hi,
> > > >> >
> > > >> > I'm planning to crawl a certain web site every 30 minutes.
> > > >> > How would I get it done in Hadoop?
> > > >> >
> > > >> > In pure Java, I used Thread.sleep() method, but I guess this won't
> > > work
> > > >> in
> > > >> > Hadoop.
> > > >>
> > > >> Why wouldn't it? You need to manage your post-job logic mostly, but
> > > >> sleep and resubmission should work just fine.
> > > >>
> > > >> > Or if it could work, could anyone show me an example?
> > > >> >
> > > >> > Ed.
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Harsh J
> > > >> www.harshj.com
> > > >>
> > > >
> > >
> >
>

Re: how to run jobs every 30 minutes?

[Ted Dunning <td...@maprtech.com>]

Or even simpler, try Azkaban: http://sna-projects.com/azkaban/

On Mon, Dec 13, 2010 at 9:26 PM, edward choi <mp...@gmail.com> wrote:

> Thanks for the tip. I took a look at it.
> Looks similar to Cascading I guess...?
> Anyway thanks for the info!!
>
> Ed
>
> 2010/12/8 Alejandro Abdelnur <tu...@cloudera.com>
>
> > Or, if you want to do it in a reliable way you could use an Oozie
> > coordinator job.
> >
> > On Wed, Dec 8, 2010 at 1:53 PM, edward choi <mp...@gmail.com> wrote:
> > > My mistake. Come to think about it, you are right, I can just make an
> > > infinite loop inside the Hadoop application.
> > > Thanks for the reply.
> > >
> > > 2010/12/7 Harsh J <qw...@gmail.com>
> > >
> > >> Hi,
> > >>
> > >> On Tue, Dec 7, 2010 at 2:25 PM, edward choi <mp...@gmail.com> wrote:
> > >> > Hi,
> > >> >
> > >> > I'm planning to crawl a certain web site every 30 minutes.
> > >> > How would I get it done in Hadoop?
> > >> >
> > >> > In pure Java, I used Thread.sleep() method, but I guess this won't
> > work
> > >> in
> > >> > Hadoop.
> > >>
> > >> Why wouldn't it? You need to manage your post-job logic mostly, but
> > >> sleep and resubmission should work just fine.
> > >>
> > >> > Or if it could work, could anyone show me an example?
> > >> >
> > >> > Ed.
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Harsh J
> > >> www.harshj.com
> > >>
> > >
> >
>

Re: how to run jobs every 30 minutes?

[edward choi <mp...@gmail.com>]

The first recommendation (gluing all my command line apps) is what I am
currently using.
The other ones you mentioned are just out of my league right now, since I am
quite new to Java world, not to mention JRuby, Groovy, Jython, etc.
But when I get comfortable with the environment and start to look for more
options I'll refer to your message. Thanks for the advanced info :-)

2010/12/15 Chris K Wensel <ch...@wensel.net>

>
> I see it this way.
>
> You can glue a bunch of discrete command line apps together that may or may
> not have dependencies between one another in a new syntax. which is darn
> nice if you already have a bunch of discrete ready to run command line apps
> sitting around that need to be strung together, that can't be used as
> libraries and instantiated through their APIs.
>
> Or, you can string all your work together through the APIs with a turing
> complete language and run them all from a single command line interface (and
> hand that to cron, or some other tool).
>
> In this case you can use Java, or easier languages like JRuby, Groovy,
> Jython, Clojure, etc which were designed for this purpose. (They don't run
> on the cluster, they only run Hadoop client side).
>
> Think ant vs graddle (or any other build tool that uses a scripting
> language and not a configuration file) if you want a concrete example.
>
> Cascading itself is a query API (and query planner). But it also exposes to
> the user the ability to run discrete 'processes' in dependency order for
> you. Either Cascading (Hadoop) Flows or Riffle annotated process objects.
> They all can be intermingled and managed from the same dependency scheduler.
> Cascading has one, and Riffle has one.
>
> So you can run> Flow -> Mahout -> Pig -> Mahout -> Flow -> shell ->
> whattheheckever from the same application.
>
> Cascading also has the ability to only run 'stale' processes. Think 'make'
> file. When re-running a job where only one file of many has changed, this is
> a big win.
>
> I personally like parameterizing my applications via the command line and
> letting my cli options drive the workflows. for example, my testing,
> integration, production environments are much different, so its very easy to
> drive specific runs of the jobs by changing a cli arg. (args4j makes this
> darn simple)
>
> if I am chaining multiple CLI apps into a bigger production app,
> parameterizing that I suspect will be error prone, esp if the input/output
> data points (jdbc vs file) are different in different contexts.
>
> you can find Riffle here, https://github.com/cwensel/riffle  (its Apache
> Licensed, contributions welcomed)
>
> ckw
>
> On Dec 14, 2010, at 1:30 AM, Alejandro Abdelnur wrote:
>
> > Ed,
> >
> > Actually Oozie is quite different from Cascading.
> >
> > * Cascading allows you to write 'queries' using a Java API and they get
> > translated into MR jobs.
> > * Oozie allows you compose sequences of MR/Pig/Hive/Java/SSH jobs in a
> DAG
> > (workflow jobs) and has timer+data dependency triggers (coordinator
> jobs).
> >
> > Regards.
> >
> > Alejandro
> >
> > On Tue, Dec 14, 2010 at 1:26 PM, edward choi <mp...@gmail.com> wrote:
> >
> >> Thanks for the tip. I took a look at it.
> >> Looks similar to Cascading I guess...?
> >> Anyway thanks for the info!!
> >>
> >> Ed
> >>
> >> 2010/12/8 Alejandro Abdelnur <tu...@cloudera.com>
> >>
> >>> Or, if you want to do it in a reliable way you could use an Oozie
> >>> coordinator job.
> >>>
> >>> On Wed, Dec 8, 2010 at 1:53 PM, edward choi <mp...@gmail.com> wrote:
> >>>> My mistake. Come to think about it, you are right, I can just make an
> >>>> infinite loop inside the Hadoop application.
> >>>> Thanks for the reply.
> >>>>
> >>>> 2010/12/7 Harsh J <qw...@gmail.com>
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> On Tue, Dec 7, 2010 at 2:25 PM, edward choi <mp...@gmail.com>
> wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> I'm planning to crawl a certain web site every 30 minutes.
> >>>>>> How would I get it done in Hadoop?
> >>>>>>
> >>>>>> In pure Java, I used Thread.sleep() method, but I guess this won't
> >>> work
> >>>>> in
> >>>>>> Hadoop.
> >>>>>
> >>>>> Why wouldn't it? You need to manage your post-job logic mostly, but
> >>>>> sleep and resubmission should work just fine.
> >>>>>
> >>>>>> Or if it could work, could anyone show me an example?
> >>>>>>
> >>>>>> Ed.
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Harsh J
> >>>>> www.harshj.com
> >>>>>
> >>>>
> >>>
> >>
>
> --
> Chris K Wensel
> chris@concurrentinc.com
> http://www.concurrentinc.com
>
> -- Concurrent, Inc. offers mentoring, support, and licensing for Cascading
>
>

Re: how to run jobs every 30 minutes?

[Ted Dunning <td...@maprtech.com>]

Good quote.

On Tue, Dec 28, 2010 at 3:46 PM, Chris K Wensel <ch...@wensel.net> wrote:

>
> deprecated is the new stable.
>
> https://issues.apache.org/jira/browse/MAPREDUCE-1734
>
> ckw
>
> On Dec 28, 2010, at 2:56 PM, Jimmy Wan wrote:
>
> > I've been using Cascading to act as "make" for my Hadoop processes for
> quite
> > some time. Unfortunately, even the most recent distribution of Cascading
> was
> > written against the deprecated Hadoop APIs (JobConf) that I'm looking to
> > replace. Does anyone have an alternative?
> >
> > On Tue, Dec 14, 2010 at 18:02, Chris K Wensel <ch...@wensel.net> wrote:
> >
> >>
> >> Cascading also has the ability to only run 'stale' processes. Think
> 'make'
> >> file. When re-running a job where only one file of many has changed,
> this is
> >> a big win.
> >>
>
> --
> Chris K Wensel
> chris@concurrentinc.com
> http://www.concurrentinc.com
>
> -- Concurrent, Inc. offers mentoring, support, and licensing for Cascading
>
>

Re: how to run jobs every 30 minutes?

[Chris K Wensel <ch...@wensel.net>]

deprecated is the new stable.

https://issues.apache.org/jira/browse/MAPREDUCE-1734

ckw

On Dec 28, 2010, at 2:56 PM, Jimmy Wan wrote:

> I've been using Cascading to act as "make" for my Hadoop processes for quite
> some time. Unfortunately, even the most recent distribution of Cascading was
> written against the deprecated Hadoop APIs (JobConf) that I'm looking to
> replace. Does anyone have an alternative?
> 
> On Tue, Dec 14, 2010 at 18:02, Chris K Wensel <ch...@wensel.net> wrote:
> 
>> 
>> Cascading also has the ability to only run 'stale' processes. Think 'make'
>> file. When re-running a job where only one file of many has changed, this is
>> a big win.
>> 

--
Chris K Wensel
chris@concurrentinc.com
http://www.concurrentinc.com

-- Concurrent, Inc. offers mentoring, support, and licensing for Cascading

Re: how to run jobs every 30 minutes?

[Jimmy Wan <ji...@indeed.com>]

I've been using Cascading to act as "make" for my Hadoop processes for quite
some time. Unfortunately, even the most recent distribution of Cascading was
written against the deprecated Hadoop APIs (JobConf) that I'm looking to
replace. Does anyone have an alternative?

On Tue, Dec 14, 2010 at 18:02, Chris K Wensel <ch...@wensel.net> wrote:

>
> Cascading also has the ability to only run 'stale' processes. Think 'make'
> file. When re-running a job where only one file of many has changed, this is
> a big win.
>

Re: how to run jobs every 30 minutes?

[Chris K Wensel <ch...@wensel.net>]

I see it this way.

You can glue a bunch of discrete command line apps together that may or may not have dependencies between one another in a new syntax. which is darn nice if you already have a bunch of discrete ready to run command line apps sitting around that need to be strung together, that can't be used as libraries and instantiated through their APIs.

Or, you can string all your work together through the APIs with a turing complete language and run them all from a single command line interface (and hand that to cron, or some other tool).

In this case you can use Java, or easier languages like JRuby, Groovy, Jython, Clojure, etc which were designed for this purpose. (They don't run on the cluster, they only run Hadoop client side).

Think ant vs graddle (or any other build tool that uses a scripting language and not a configuration file) if you want a concrete example.

Cascading itself is a query API (and query planner). But it also exposes to the user the ability to run discrete 'processes' in dependency order for you. Either Cascading (Hadoop) Flows or Riffle annotated process objects. They all can be intermingled and managed from the same dependency scheduler. Cascading has one, and Riffle has one.

So you can run> Flow -> Mahout -> Pig -> Mahout -> Flow -> shell -> whattheheckever from the same application.

Cascading also has the ability to only run 'stale' processes. Think 'make' file. When re-running a job where only one file of many has changed, this is a big win.

I personally like parameterizing my applications via the command line and letting my cli options drive the workflows. for example, my testing, integration, production environments are much different, so its very easy to drive specific runs of the jobs by changing a cli arg. (args4j makes this darn simple)

if I am chaining multiple CLI apps into a bigger production app, parameterizing that I suspect will be error prone, esp if the input/output data points (jdbc vs file) are different in different contexts.

you can find Riffle here, https://github.com/cwensel/riffle  (its Apache Licensed, contributions welcomed)

ckw

On Dec 14, 2010, at 1:30 AM, Alejandro Abdelnur wrote:

> Ed,
> 
> Actually Oozie is quite different from Cascading.
> 
> * Cascading allows you to write 'queries' using a Java API and they get
> translated into MR jobs.
> * Oozie allows you compose sequences of MR/Pig/Hive/Java/SSH jobs in a DAG
> (workflow jobs) and has timer+data dependency triggers (coordinator jobs).
> 
> Regards.
> 
> Alejandro
> 
> On Tue, Dec 14, 2010 at 1:26 PM, edward choi <mp...@gmail.com> wrote:
> 
>> Thanks for the tip. I took a look at it.
>> Looks similar to Cascading I guess...?
>> Anyway thanks for the info!!
>> 
>> Ed
>> 
>> 2010/12/8 Alejandro Abdelnur <tu...@cloudera.com>
>> 
>>> Or, if you want to do it in a reliable way you could use an Oozie
>>> coordinator job.
>>> 
>>> On Wed, Dec 8, 2010 at 1:53 PM, edward choi <mp...@gmail.com> wrote:
>>>> My mistake. Come to think about it, you are right, I can just make an
>>>> infinite loop inside the Hadoop application.
>>>> Thanks for the reply.
>>>> 
>>>> 2010/12/7 Harsh J <qw...@gmail.com>
>>>> 
>>>>> Hi,
>>>>> 
>>>>> On Tue, Dec 7, 2010 at 2:25 PM, edward choi <mp...@gmail.com> wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I'm planning to crawl a certain web site every 30 minutes.
>>>>>> How would I get it done in Hadoop?
>>>>>> 
>>>>>> In pure Java, I used Thread.sleep() method, but I guess this won't
>>> work
>>>>> in
>>>>>> Hadoop.
>>>>> 
>>>>> Why wouldn't it? You need to manage your post-job logic mostly, but
>>>>> sleep and resubmission should work just fine.
>>>>> 
>>>>>> Or if it could work, could anyone show me an example?
>>>>>> 
>>>>>> Ed.
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Harsh J
>>>>> www.harshj.com
>>>>> 
>>>> 
>>> 
>> 

--
Chris K Wensel
chris@concurrentinc.com
http://www.concurrentinc.com

-- Concurrent, Inc. offers mentoring, support, and licensing for Cascading

Re: how to run jobs every 30 minutes?

[edward choi <mp...@gmail.com>]

That clears the confusion. Thanks.
There are just too many tools for Hadoop :-)

2010/12/14 Alejandro Abdelnur <tu...@cloudera.com>

> Ed,
>
> Actually Oozie is quite different from Cascading.
>
> * Cascading allows you to write 'queries' using a Java API and they get
> translated into MR jobs.
> * Oozie allows you compose sequences of MR/Pig/Hive/Java/SSH jobs in a DAG
> (workflow jobs) and has timer+data dependency triggers (coordinator jobs).
>
> Regards.
>
> Alejandro
>
> On Tue, Dec 14, 2010 at 1:26 PM, edward choi <mp...@gmail.com> wrote:
>
> > Thanks for the tip. I took a look at it.
> > Looks similar to Cascading I guess...?
> > Anyway thanks for the info!!
> >
> > Ed
> >
> > 2010/12/8 Alejandro Abdelnur <tu...@cloudera.com>
> >
> > > Or, if you want to do it in a reliable way you could use an Oozie
> > > coordinator job.
> > >
> > > On Wed, Dec 8, 2010 at 1:53 PM, edward choi <mp...@gmail.com> wrote:
> > > > My mistake. Come to think about it, you are right, I can just make an
> > > > infinite loop inside the Hadoop application.
> > > > Thanks for the reply.
> > > >
> > > > 2010/12/7 Harsh J <qw...@gmail.com>
> > > >
> > > >> Hi,
> > > >>
> > > >> On Tue, Dec 7, 2010 at 2:25 PM, edward choi <mp...@gmail.com>
> wrote:
> > > >> > Hi,
> > > >> >
> > > >> > I'm planning to crawl a certain web site every 30 minutes.
> > > >> > How would I get it done in Hadoop?
> > > >> >
> > > >> > In pure Java, I used Thread.sleep() method, but I guess this won't
> > > work
> > > >> in
> > > >> > Hadoop.
> > > >>
> > > >> Why wouldn't it? You need to manage your post-job logic mostly, but
> > > >> sleep and resubmission should work just fine.
> > > >>
> > > >> > Or if it could work, could anyone show me an example?
> > > >> >
> > > >> > Ed.
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Harsh J
> > > >> www.harshj.com
> > > >>
> > > >
> > >
> >
>

Re: how to run jobs every 30 minutes?

[Alejandro Abdelnur <tu...@cloudera.com>]

Ed,

Actually Oozie is quite different from Cascading.

* Cascading allows you to write 'queries' using a Java API and they get
translated into MR jobs.
* Oozie allows you compose sequences of MR/Pig/Hive/Java/SSH jobs in a DAG
(workflow jobs) and has timer+data dependency triggers (coordinator jobs).

Regards.

Alejandro

On Tue, Dec 14, 2010 at 1:26 PM, edward choi <mp...@gmail.com> wrote:

> Thanks for the tip. I took a look at it.
> Looks similar to Cascading I guess...?
> Anyway thanks for the info!!
>
> Ed
>
> 2010/12/8 Alejandro Abdelnur <tu...@cloudera.com>
>
> > Or, if you want to do it in a reliable way you could use an Oozie
> > coordinator job.
> >
> > On Wed, Dec 8, 2010 at 1:53 PM, edward choi <mp...@gmail.com> wrote:
> > > My mistake. Come to think about it, you are right, I can just make an
> > > infinite loop inside the Hadoop application.
> > > Thanks for the reply.
> > >
> > > 2010/12/7 Harsh J <qw...@gmail.com>
> > >
> > >> Hi,
> > >>
> > >> On Tue, Dec 7, 2010 at 2:25 PM, edward choi <mp...@gmail.com> wrote:
> > >> > Hi,
> > >> >
> > >> > I'm planning to crawl a certain web site every 30 minutes.
> > >> > How would I get it done in Hadoop?
> > >> >
> > >> > In pure Java, I used Thread.sleep() method, but I guess this won't
> > work
> > >> in
> > >> > Hadoop.
> > >>
> > >> Why wouldn't it? You need to manage your post-job logic mostly, but
> > >> sleep and resubmission should work just fine.
> > >>
> > >> > Or if it could work, could anyone show me an example?
> > >> >
> > >> > Ed.
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Harsh J
> > >> www.harshj.com
> > >>
> > >
> >
>

Re: how to run jobs every 30 minutes?

[edward choi <mp...@gmail.com>]

Thanks for the tip. I took a look at it.
Looks similar to Cascading I guess...?
Anyway thanks for the info!!

Ed

2010/12/8 Alejandro Abdelnur <tu...@cloudera.com>

> Or, if you want to do it in a reliable way you could use an Oozie
> coordinator job.
>
> On Wed, Dec 8, 2010 at 1:53 PM, edward choi <mp...@gmail.com> wrote:
> > My mistake. Come to think about it, you are right, I can just make an
> > infinite loop inside the Hadoop application.
> > Thanks for the reply.
> >
> > 2010/12/7 Harsh J <qw...@gmail.com>
> >
> >> Hi,
> >>
> >> On Tue, Dec 7, 2010 at 2:25 PM, edward choi <mp...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I'm planning to crawl a certain web site every 30 minutes.
> >> > How would I get it done in Hadoop?
> >> >
> >> > In pure Java, I used Thread.sleep() method, but I guess this won't
> work
> >> in
> >> > Hadoop.
> >>
> >> Why wouldn't it? You need to manage your post-job logic mostly, but
> >> sleep and resubmission should work just fine.
> >>
> >> > Or if it could work, could anyone show me an example?
> >> >
> >> > Ed.
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >> www.harshj.com
> >>
> >
>

Re: how to run jobs every 30 minutes?

[Alejandro Abdelnur <tu...@cloudera.com>]

Or, if you want to do it in a reliable way you could use an Oozie
coordinator job.

On Wed, Dec 8, 2010 at 1:53 PM, edward choi <mp...@gmail.com> wrote:
> My mistake. Come to think about it, you are right, I can just make an
> infinite loop inside the Hadoop application.
> Thanks for the reply.
>
> 2010/12/7 Harsh J <qw...@gmail.com>
>
>> Hi,
>>
>> On Tue, Dec 7, 2010 at 2:25 PM, edward choi <mp...@gmail.com> wrote:
>> > Hi,
>> >
>> > I'm planning to crawl a certain web site every 30 minutes.
>> > How would I get it done in Hadoop?
>> >
>> > In pure Java, I used Thread.sleep() method, but I guess this won't work
>> in
>> > Hadoop.
>>
>> Why wouldn't it? You need to manage your post-job logic mostly, but
>> sleep and resubmission should work just fine.
>>
>> > Or if it could work, could anyone show me an example?
>> >
>> > Ed.
>> >
>>
>>
>>
>> --
>> Harsh J
>> www.harshj.com
>>
>

Re: how to run jobs every 30 minutes?

[edward choi <mp...@gmail.com>]

My mistake. Come to think about it, you are right, I can just make an
infinite loop inside the Hadoop application.
Thanks for the reply.

2010/12/7 Harsh J <qw...@gmail.com>

> Hi,
>
> On Tue, Dec 7, 2010 at 2:25 PM, edward choi <mp...@gmail.com> wrote:
> > Hi,
> >
> > I'm planning to crawl a certain web site every 30 minutes.
> > How would I get it done in Hadoop?
> >
> > In pure Java, I used Thread.sleep() method, but I guess this won't work
> in
> > Hadoop.
>
> Why wouldn't it? You need to manage your post-job logic mostly, but
> sleep and resubmission should work just fine.
>
> > Or if it could work, could anyone show me an example?
> >
> > Ed.
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>

Re: how to run jobs every 30 minutes?

[Harsh J <qw...@gmail.com>]

Hi,

On Tue, Dec 7, 2010 at 2:25 PM, edward choi <mp...@gmail.com> wrote:
> Hi,
>
> I'm planning to crawl a certain web site every 30 minutes.
> How would I get it done in Hadoop?
>
> In pure Java, I used Thread.sleep() method, but I guess this won't work in
> Hadoop.

Why wouldn't it? You need to manage your post-job logic mostly, but
sleep and resubmission should work just fine.

> Or if it could work, could anyone show me an example?
>
> Ed.
>



-- 
Harsh J
www.harshj.com