You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Kandoi, Nikhil" <Ni...@emc.com> on 2014/08/27 13:38:57 UTC
What happens when .....?
Hi All,
I have a system where files are coming in hdfs at regular intervals and I perform an operation everytime the directory size goes above a particular point.
My Question is that when I submit a map reduce job, would it only work on the files present at that point ??
Regards,
Nikhil Kandoi
Re: What happens when .....?
Posted by Mahesh Khandewal <ma...@gmail.com>.
unsubscribe
On Thu, Aug 28, 2014 at 6:42 PM, Eric Payne <er...@yahoo.com>
wrote:
> Or, maybe have a look at Apache Falcon:
> Falcon - Apache Falcon - Data management and processing platform
> <http://falcon.incubator.apache.org/>
>
>
>
>
>
>
> Falcon - Apache Falcon - Data management and processing platform
> <http://falcon.incubator.apache.org/>
> Apache Falcon - Data management and processing platform
> View on falcon.incubator.apache.org <http://falcon.incubator.apache.org/>
> Preview by Yahoo
>
>
> *From:* Stanley Shi <ss...@pivotal.io>
> *To:* "user@hadoop.apache.org" <us...@hadoop.apache.org>
> *Sent:* Thursday, August 28, 2014 1:15 AM
> *Subject:* Re: What happens when .....?
>
> Normally MR job is used for batch processing. So I don't think this is a
> good use case here for MR.
> Since you need to run the program periodically, you cannot submit a single
> mapreduce job for this.
> An possible way is to create a cron job to scan the folder size and submit
> a MR job if necessary;
>
>
>
> On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Ni...@emc.com>
> wrote:
>
> Hi All,
>
> I have a system where files are coming in hdfs at regular intervals and I
> perform an operation everytime the directory size goes above a particular
> point.
> My Question is that when I submit a map reduce job, would it only work on
> the files present at that point ??
>
> Regards,
> Nikhil Kandoi
>
>
>
>
>
>
> --
> Regards,
> *Stanley Shi,*
>
>
>
>
Re: What happens when .....?
Posted by Mahesh Khandewal <ma...@gmail.com>.
unsubscribe
On Thu, Aug 28, 2014 at 6:42 PM, Eric Payne <er...@yahoo.com>
wrote:
> Or, maybe have a look at Apache Falcon:
> Falcon - Apache Falcon - Data management and processing platform
> <http://falcon.incubator.apache.org/>
>
>
>
>
>
>
> Falcon - Apache Falcon - Data management and processing platform
> <http://falcon.incubator.apache.org/>
> Apache Falcon - Data management and processing platform
> View on falcon.incubator.apache.org <http://falcon.incubator.apache.org/>
> Preview by Yahoo
>
>
> *From:* Stanley Shi <ss...@pivotal.io>
> *To:* "user@hadoop.apache.org" <us...@hadoop.apache.org>
> *Sent:* Thursday, August 28, 2014 1:15 AM
> *Subject:* Re: What happens when .....?
>
> Normally MR job is used for batch processing. So I don't think this is a
> good use case here for MR.
> Since you need to run the program periodically, you cannot submit a single
> mapreduce job for this.
> An possible way is to create a cron job to scan the folder size and submit
> a MR job if necessary;
>
>
>
> On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Ni...@emc.com>
> wrote:
>
> Hi All,
>
> I have a system where files are coming in hdfs at regular intervals and I
> perform an operation everytime the directory size goes above a particular
> point.
> My Question is that when I submit a map reduce job, would it only work on
> the files present at that point ??
>
> Regards,
> Nikhil Kandoi
>
>
>
>
>
>
> --
> Regards,
> *Stanley Shi,*
>
>
>
>
Re: What happens when .....?
Posted by Mahesh Khandewal <ma...@gmail.com>.
unsubscribe
On Thu, Aug 28, 2014 at 6:42 PM, Eric Payne <er...@yahoo.com>
wrote:
> Or, maybe have a look at Apache Falcon:
> Falcon - Apache Falcon - Data management and processing platform
> <http://falcon.incubator.apache.org/>
>
>
>
>
>
>
> Falcon - Apache Falcon - Data management and processing platform
> <http://falcon.incubator.apache.org/>
> Apache Falcon - Data management and processing platform
> View on falcon.incubator.apache.org <http://falcon.incubator.apache.org/>
> Preview by Yahoo
>
>
> *From:* Stanley Shi <ss...@pivotal.io>
> *To:* "user@hadoop.apache.org" <us...@hadoop.apache.org>
> *Sent:* Thursday, August 28, 2014 1:15 AM
> *Subject:* Re: What happens when .....?
>
> Normally MR job is used for batch processing. So I don't think this is a
> good use case here for MR.
> Since you need to run the program periodically, you cannot submit a single
> mapreduce job for this.
> An possible way is to create a cron job to scan the folder size and submit
> a MR job if necessary;
>
>
>
> On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Ni...@emc.com>
> wrote:
>
> Hi All,
>
> I have a system where files are coming in hdfs at regular intervals and I
> perform an operation everytime the directory size goes above a particular
> point.
> My Question is that when I submit a map reduce job, would it only work on
> the files present at that point ??
>
> Regards,
> Nikhil Kandoi
>
>
>
>
>
>
> --
> Regards,
> *Stanley Shi,*
>
>
>
>
Re: What happens when .....?
Posted by Mahesh Khandewal <ma...@gmail.com>.
unsubscribe
On Thu, Aug 28, 2014 at 6:42 PM, Eric Payne <er...@yahoo.com>
wrote:
> Or, maybe have a look at Apache Falcon:
> Falcon - Apache Falcon - Data management and processing platform
> <http://falcon.incubator.apache.org/>
>
>
>
>
>
>
> Falcon - Apache Falcon - Data management and processing platform
> <http://falcon.incubator.apache.org/>
> Apache Falcon - Data management and processing platform
> View on falcon.incubator.apache.org <http://falcon.incubator.apache.org/>
> Preview by Yahoo
>
>
> *From:* Stanley Shi <ss...@pivotal.io>
> *To:* "user@hadoop.apache.org" <us...@hadoop.apache.org>
> *Sent:* Thursday, August 28, 2014 1:15 AM
> *Subject:* Re: What happens when .....?
>
> Normally MR job is used for batch processing. So I don't think this is a
> good use case here for MR.
> Since you need to run the program periodically, you cannot submit a single
> mapreduce job for this.
> An possible way is to create a cron job to scan the folder size and submit
> a MR job if necessary;
>
>
>
> On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Ni...@emc.com>
> wrote:
>
> Hi All,
>
> I have a system where files are coming in hdfs at regular intervals and I
> perform an operation everytime the directory size goes above a particular
> point.
> My Question is that when I submit a map reduce job, would it only work on
> the files present at that point ??
>
> Regards,
> Nikhil Kandoi
>
>
>
>
>
>
> --
> Regards,
> *Stanley Shi,*
>
>
>
>
Re: What happens when .....?
Posted by Eric Payne <er...@yahoo.com>.
Or, maybe have a look at Apache Falcon:
Falcon - Apache Falcon - Data management and processing platform
Falcon - Apache Falcon - Data management and processing platform
Apache Falcon - Data management and processing platform
View on falcon.incubator.apache.org Preview by Yahoo
From: Stanley Shi <ss...@pivotal.io>
>To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>Sent: Thursday, August 28, 2014 1:15 AM
>Subject: Re: What happens when .....?
>
>
>
>Normally MR job is used for batch processing. So I don't think this is a good use case here for MR.
>Since you need to run the program periodically, you cannot submit a single mapreduce job for this.
>An possible way is to create a cron job to scan the folder size and submit a MR job if necessary;
>
>
>
>
>
>On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Ni...@emc.com> wrote:
>
>Hi All,
>>
>>I have a system where files are coming in hdfs at regular intervals and I perform an operation everytime the directory size goes above a particular point.
>>My Question is that when I submit a map reduce job, would it only work on the files present at that point ??
>>
>>Regards,
>>Nikhil Kandoi
>>
>>
>>
>
>
>
>--
>
>Regards,
>Stanley Shi,
>
>
>
Re: What happens when .....?
Posted by Eric Payne <er...@yahoo.com>.
Or, maybe have a look at Apache Falcon:
Falcon - Apache Falcon - Data management and processing platform
Falcon - Apache Falcon - Data management and processing platform
Apache Falcon - Data management and processing platform
View on falcon.incubator.apache.org Preview by Yahoo
From: Stanley Shi <ss...@pivotal.io>
>To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>Sent: Thursday, August 28, 2014 1:15 AM
>Subject: Re: What happens when .....?
>
>
>
>Normally MR job is used for batch processing. So I don't think this is a good use case here for MR.
>Since you need to run the program periodically, you cannot submit a single mapreduce job for this.
>An possible way is to create a cron job to scan the folder size and submit a MR job if necessary;
>
>
>
>
>
>On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Ni...@emc.com> wrote:
>
>Hi All,
>>
>>I have a system where files are coming in hdfs at regular intervals and I perform an operation everytime the directory size goes above a particular point.
>>My Question is that when I submit a map reduce job, would it only work on the files present at that point ??
>>
>>Regards,
>>Nikhil Kandoi
>>
>>
>>
>
>
>
>--
>
>Regards,
>Stanley Shi,
>
>
>
Re: What happens when .....?
Posted by Eric Payne <er...@yahoo.com>.
Or, maybe have a look at Apache Falcon:
Falcon - Apache Falcon - Data management and processing platform
Falcon - Apache Falcon - Data management and processing platform
Apache Falcon - Data management and processing platform
View on falcon.incubator.apache.org Preview by Yahoo
From: Stanley Shi <ss...@pivotal.io>
>To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>Sent: Thursday, August 28, 2014 1:15 AM
>Subject: Re: What happens when .....?
>
>
>
>Normally MR job is used for batch processing. So I don't think this is a good use case here for MR.
>Since you need to run the program periodically, you cannot submit a single mapreduce job for this.
>An possible way is to create a cron job to scan the folder size and submit a MR job if necessary;
>
>
>
>
>
>On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Ni...@emc.com> wrote:
>
>Hi All,
>>
>>I have a system where files are coming in hdfs at regular intervals and I perform an operation everytime the directory size goes above a particular point.
>>My Question is that when I submit a map reduce job, would it only work on the files present at that point ??
>>
>>Regards,
>>Nikhil Kandoi
>>
>>
>>
>
>
>
>--
>
>Regards,
>Stanley Shi,
>
>
>
Re: What happens when .....?
Posted by Eric Payne <er...@yahoo.com>.
Or, maybe have a look at Apache Falcon:
Falcon - Apache Falcon - Data management and processing platform
Falcon - Apache Falcon - Data management and processing platform
Apache Falcon - Data management and processing platform
View on falcon.incubator.apache.org Preview by Yahoo
From: Stanley Shi <ss...@pivotal.io>
>To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>Sent: Thursday, August 28, 2014 1:15 AM
>Subject: Re: What happens when .....?
>
>
>
>Normally MR job is used for batch processing. So I don't think this is a good use case here for MR.
>Since you need to run the program periodically, you cannot submit a single mapreduce job for this.
>An possible way is to create a cron job to scan the folder size and submit a MR job if necessary;
>
>
>
>
>
>On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Ni...@emc.com> wrote:
>
>Hi All,
>>
>>I have a system where files are coming in hdfs at regular intervals and I perform an operation everytime the directory size goes above a particular point.
>>My Question is that when I submit a map reduce job, would it only work on the files present at that point ??
>>
>>Regards,
>>Nikhil Kandoi
>>
>>
>>
>
>
>
>--
>
>Regards,
>Stanley Shi,
>
>
>
Re: What happens when .....?
Posted by Stanley Shi <ss...@pivotal.io>.
Normally MR job is used for batch processing. So I don't think this is a
good use case here for MR.
Since you need to run the program periodically, you cannot submit a single
mapreduce job for this.
An possible way is to create a cron job to scan the folder size and submit
a MR job if necessary;
On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Ni...@emc.com>
wrote:
> Hi All,
>
>
>
> I have a system where files are coming in hdfs at regular intervals and I
> perform an operation everytime the directory size goes above a particular
> point.
>
> My Question is that when I submit a map reduce job, would it only work on
> the files present at that point ??
>
>
>
> Regards,
> Nikhil Kandoi
>
>
>
--
Regards,
*Stanley Shi,*
Re: What happens when .....?
Posted by Stanley Shi <ss...@pivotal.io>.
Normally MR job is used for batch processing. So I don't think this is a
good use case here for MR.
Since you need to run the program periodically, you cannot submit a single
mapreduce job for this.
An possible way is to create a cron job to scan the folder size and submit
a MR job if necessary;
On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Ni...@emc.com>
wrote:
> Hi All,
>
>
>
> I have a system where files are coming in hdfs at regular intervals and I
> perform an operation everytime the directory size goes above a particular
> point.
>
> My Question is that when I submit a map reduce job, would it only work on
> the files present at that point ??
>
>
>
> Regards,
> Nikhil Kandoi
>
>
>
--
Regards,
*Stanley Shi,*
Re: What happens when .....?
Posted by Stanley Shi <ss...@pivotal.io>.
Normally MR job is used for batch processing. So I don't think this is a
good use case here for MR.
Since you need to run the program periodically, you cannot submit a single
mapreduce job for this.
An possible way is to create a cron job to scan the folder size and submit
a MR job if necessary;
On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Ni...@emc.com>
wrote:
> Hi All,
>
>
>
> I have a system where files are coming in hdfs at regular intervals and I
> perform an operation everytime the directory size goes above a particular
> point.
>
> My Question is that when I submit a map reduce job, would it only work on
> the files present at that point ??
>
>
>
> Regards,
> Nikhil Kandoi
>
>
>
--
Regards,
*Stanley Shi,*
Re: What happens when .....?
Posted by Stanley Shi <ss...@pivotal.io>.
Normally MR job is used for batch processing. So I don't think this is a
good use case here for MR.
Since you need to run the program periodically, you cannot submit a single
mapreduce job for this.
An possible way is to create a cron job to scan the folder size and submit
a MR job if necessary;
On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Ni...@emc.com>
wrote:
> Hi All,
>
>
>
> I have a system where files are coming in hdfs at regular intervals and I
> perform an operation everytime the directory size goes above a particular
> point.
>
> My Question is that when I submit a map reduce job, would it only work on
> the files present at that point ??
>
>
>
> Regards,
> Nikhil Kandoi
>
>
>
--
Regards,
*Stanley Shi,*