You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Manoj Babu <ma...@gmail.com> on 2012/09/09 17:58:54 UTC
Reg: parsing all files & file append
Hi All,
I have two questions, providing info on it will be helpful.
1, I am using hadoop to analyze and to find top n search term metric's from
logs.
If any new log file is added to HDFS then again we are running the job to
find the metrics.
Daily we will be getting log files and we are parsing the whole file and
getting the metric's.
All the log file's are parsed daily to get the latest metric's is there any
way is there any way to avoid this?
2, Does file append is production stable?
Cheers!
Manoj.
Re: Reg: parsing all files & file append
Posted by Manoj Babu <ma...@gmail.com>.
Thank you Bejoy.
Cheers!
Manoj.
On Mon, Sep 10, 2012 at 1:36 PM, Bejoy Ks <be...@gmail.com> wrote:
> Hi Manoj
>
> From my limited knowledge on file appends in hdfs , i have seen more
> recommendations to use sync() in the latest releases than using append().
> Let us wait for some commiter to authoritatively comment on 'the production
> readiness of append()' . :)
>
> Regards
> Bejoy KS
>
>
> On Mon, Sep 10, 2012 at 11:03 AM, Manoj Babu <ma...@gmail.com> wrote:
>
>> Thank you Bejoy.
>>
>> Does file append is production stable?
>>
>>
>> Cheers!
>> Manoj.
>>
>>
>>
>> On Sun, Sep 9, 2012 at 10:19 PM, Bejoy KS <be...@gmail.com> wrote:
>>
>>> **
>>> Hi Manoj
>>>
>>> You can load daily logs into a individual directories in hdfs and
>>> process them daily. Keep those results in hdfs or hbase or dbs etc. Every
>>> day do the processing, get the results and aggregate the same with the
>>> previously aggregated results till date.
>>>
>>> Regards
>>> Bejoy KS
>>>
>>> Sent from handheld, please excuse typos.
>>> ------------------------------
>>> *From: * Manoj Babu <ma...@gmail.com>
>>> *Date: *Sun, 9 Sep 2012 21:28:54 +0530
>>> *To: *<ma...@hadoop.apache.org>
>>> *ReplyTo: * mapreduce-user@hadoop.apache.org
>>> *Subject: *Reg: parsing all files & file append
>>>
>>> Hi All,
>>>
>>> I have two questions, providing info on it will be helpful.
>>>
>>> 1, I am using hadoop to analyze and to find top n search term metric's
>>> from logs.
>>> If any new log file is added to HDFS then again we are running the job
>>> to find the metrics.
>>> Daily we will be getting log files and we are parsing the whole file and
>>> getting the metric's.
>>> All the log file's are parsed daily to get the latest metric's is there
>>> any way is there any way to avoid this?
>>>
>>> 2, Does file append is production stable?
>>>
>>> Cheers!
>>> Manoj.
>>>
>>>
>>
>
Re: Reg: parsing all files & file append
Posted by Bejoy Ks <be...@gmail.com>.
Hi Manoj
>From my limited knowledge on file appends in hdfs , i have seen more
recommendations to use sync() in the latest releases than using append().
Let us wait for some commiter to authoritatively comment on 'the production
readiness of append()' . :)
Regards
Bejoy KS
On Mon, Sep 10, 2012 at 11:03 AM, Manoj Babu <ma...@gmail.com> wrote:
> Thank you Bejoy.
>
> Does file append is production stable?
>
>
> Cheers!
> Manoj.
>
>
>
> On Sun, Sep 9, 2012 at 10:19 PM, Bejoy KS <be...@gmail.com> wrote:
>
>> **
>> Hi Manoj
>>
>> You can load daily logs into a individual directories in hdfs and process
>> them daily. Keep those results in hdfs or hbase or dbs etc. Every day do
>> the processing, get the results and aggregate the same with the previously
>> aggregated results till date.
>>
>> Regards
>> Bejoy KS
>>
>> Sent from handheld, please excuse typos.
>> ------------------------------
>> *From: * Manoj Babu <ma...@gmail.com>
>> *Date: *Sun, 9 Sep 2012 21:28:54 +0530
>> *To: *<ma...@hadoop.apache.org>
>> *ReplyTo: * mapreduce-user@hadoop.apache.org
>> *Subject: *Reg: parsing all files & file append
>>
>> Hi All,
>>
>> I have two questions, providing info on it will be helpful.
>>
>> 1, I am using hadoop to analyze and to find top n search term metric's
>> from logs.
>> If any new log file is added to HDFS then again we are running the job to
>> find the metrics.
>> Daily we will be getting log files and we are parsing the whole file and
>> getting the metric's.
>> All the log file's are parsed daily to get the latest metric's is there
>> any way is there any way to avoid this?
>>
>> 2, Does file append is production stable?
>>
>> Cheers!
>> Manoj.
>>
>>
>
Re: Reg: parsing all files & file append
Posted by Manoj Babu <ma...@gmail.com>.
Thank you Bejoy.
Does file append is production stable?
Cheers!
Manoj.
On Sun, Sep 9, 2012 at 10:19 PM, Bejoy KS <be...@gmail.com> wrote:
> **
> Hi Manoj
>
> You can load daily logs into a individual directories in hdfs and process
> them daily. Keep those results in hdfs or hbase or dbs etc. Every day do
> the processing, get the results and aggregate the same with the previously
> aggregated results till date.
>
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
> ------------------------------
> *From: * Manoj Babu <ma...@gmail.com>
> *Date: *Sun, 9 Sep 2012 21:28:54 +0530
> *To: *<ma...@hadoop.apache.org>
> *ReplyTo: * mapreduce-user@hadoop.apache.org
> *Subject: *Reg: parsing all files & file append
>
> Hi All,
>
> I have two questions, providing info on it will be helpful.
>
> 1, I am using hadoop to analyze and to find top n search term metric's
> from logs.
> If any new log file is added to HDFS then again we are running the job to
> find the metrics.
> Daily we will be getting log files and we are parsing the whole file and
> getting the metric's.
> All the log file's are parsed daily to get the latest metric's is there
> any way is there any way to avoid this?
>
> 2, Does file append is production stable?
>
> Cheers!
> Manoj.
>
>
Re: Reg: parsing all files & file append
Posted by Bejoy KS <be...@gmail.com>.
Hi Manoj
You can load daily logs into a individual directories in hdfs and process them daily. Keep those results in hdfs or hbase or dbs etc. Every day do the processing, get the results and aggregate the same with the previously aggregated results till date.
Regards
Bejoy KS
Sent from handheld, please excuse typos.
-----Original Message-----
From: Manoj Babu <ma...@gmail.com>
Date: Sun, 9 Sep 2012 21:28:54
To: <ma...@hadoop.apache.org>
Reply-To: mapreduce-user@hadoop.apache.org
Subject: Reg: parsing all files & file append
Hi All,
I have two questions, providing info on it will be helpful.
1, I am using hadoop to analyze and to find top n search term metric's from
logs.
If any new log file is added to HDFS then again we are running the job to
find the metrics.
Daily we will be getting log files and we are parsing the whole file and
getting the metric's.
All the log file's are parsed daily to get the latest metric's is there any
way is there any way to avoid this?
2, Does file append is production stable?
Cheers!
Manoj.