You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-dev@logging.apache.org by Pranav Bhole <pr...@gmail.com> on 2013/03/22 10:04:12 UTC

Idea of Cloud as target for the bulk of log file

Hello to all,
        This is Pranav Bhole, I am Master student at The University of
Texas at Dallas. My research interest is Big Data. I haven been using Log4j
extensively as core since 5-6 years in my academic and professional work.
Recently an idea came up in my mind by facing some of the difficulties in
managing the TeraBytes of Log files. I would like to implement this idea as
plug in or functionality in the existing log4j appender module as student
of Google Summer of Code 2013.

Short description of the idea:
Server appends the bulk of log files and in the most cases server lacks
with the storage space for these logs files and also computing on such bulk
of file is costly for the server. With the consideration of this problem,
idea proposes to write a module which could be able to move these files
into Public (S3 of AWS, Azure) or private cloud (Hadoop) on the rolling
basis based on the configuration file. To resolve the computing layer
objective, the idea proposes the Big Data Query generator based on the
logging format used. Such Big Data Queries will include MapReduce, PIG etc.
Administrator would be able to run these BigData queries generated by Log4j
to track the keywords in the logs like error number, TimeStamp or any other
arbitrary string.

I would like to appreciate to all of you for reading this idea. I would
really love to get involved in Log4j development team with your support and
suggestion on this idea.

Thank you very much.

-- 
Pranav Bhole
Student of MS in Computer Science for Fall 2012,
University of Texas at Dallas
http://www.linkedin.com/in/pranavbhole
Cell Phone No: 972-978-6108.

Re: Idea of Cloud as target for the bulk of log file

Posted by Tushar Kapila <tg...@gmail.com>.
 From what I could see you plan to edit the rolling file appender so it 
can take a custom action on some events (roll over when already rolled 
over files is > X) at which point files are pushed to another system. 
Rolling file appender would continue to do what it does natively. I 
think a better design would be to enhance the appender in a generic way 
so it can have a RollingFileListeners and tell these objects about 
important events and pass on config and current state to them. The 
listeners can then do further processing, there might be a need to tell 
the rolling appender of action taken like files were moved or deleted. 
The appender does not need to know where the files are gone (to hadoop 
or just zipped for long term storage) but it might help with its 
internal state being more consistent with changes to the folder.


On 3/29/2013 9:48 AM, Pranav Bhole wrote:
> Hello to All,
> As per our discussion I am attaching the "Blueprint of Simple, 
> Efficient and Reliable HadoopAppender in Log4j". I would love see the 
> suggestions.
>
> Thank you
>
>
> On Fri, Mar 22, 2013 at 8:42 PM, Gary Gregory <garydgregory@gmail.com 
> <ma...@gmail.com>> wrote:
>
>     An AppScale appender might be interesting:
>     http://appscale.cs.ucsb.edu/datastores.html
>
>     Gary
>
>
>     On Fri, Mar 22, 2013 at 4:34 PM, Christian Grobmeier
>     <grobmeier@gmail.com <ma...@gmail.com>> wrote:
>
>         +1
>
>         I would love to support a GSOC student, and if your more concrete
>         proposal meets some interest here I am willing to actually help.
>         That said, while Apache Flume is great, its maybe a bit "too
>         much". I
>         already have had some thoughts on some kind of a server which
>         utilizes
>         receivers to send data to $x. Less features than Flume, but
>         easy to
>         setup. Not sure if that has some value.
>
>         Pranav, please let us hear more of your ideas.
>
>
>
>         On Fri, Mar 22, 2013 at 4:40 PM, Ralph Goers
>         <ralph.goers@dslextreme.com
>         <ma...@dslextreme.com>> wrote:
>         > The Flume Appender leverages Apache Flume to route data into
>         various places.
>         > The primary sponsor of Flume is Cloudera, so naturally Flume
>         supports
>         > writing data into Hadoop. In addition, my employer is using
>         Flume to write
>         > data into Cassandra.  That said, we would welcome
>         contributions and if you
>         > can provide more details on how you would implement your
>         idea I'd love to
>         > see them.  Perhaps you can create a page on the logging wiki
>         with your
>         > proposal.
>         >
>         > Ralph
>         >
>         > On Mar 22, 2013, at 2:04 AM, Pranav Bhole wrote:
>         >
>         > Hello to all,
>         >         This is Pranav Bhole, I am Master student at The
>         University of Texas
>         > at Dallas. My research interest is Big Data. I haven been
>         using Log4j
>         > extensively as core since 5-6 years in my academic and
>         professional work.
>         > Recently an idea came up in my mind by facing some of the
>         difficulties in
>         > managing the TeraBytes of Log files. I would like to
>         implement this idea as
>         > plug in or functionality in the existing log4j appender
>         module as student of
>         > Google Summer of Code 2013.
>         >
>         > Short description of the idea:
>         > Server appends the bulk of log files and in the most cases
>         server lacks with
>         > the storage space for these logs files and also computing on
>         such bulk of
>         > file is costly for the server. With the consideration of
>         this problem, idea
>         > proposes to write a module which could be able to move these
>         files into
>         > Public (S3 of AWS, Azure) or private cloud (Hadoop) on the
>         rolling basis
>         > based on the configuration file. To resolve the computing
>         layer objective,
>         > the idea proposes the Big Data Query generator based on the
>         logging format
>         > used. Such Big Data Queries will include MapReduce, PIG etc.
>         Administrator
>         > would be able to run these BigData queries generated by
>         Log4j to track the
>         > keywords in the logs like error number, TimeStamp or any
>         other arbitrary
>         > string.
>         >
>         > I would like to appreciate to all of you for reading this
>         idea. I would
>         > really love to get involved in Log4j development team with
>         your support and
>         > suggestion on this idea.
>         >
>         > Thank you very much.
>         >
>         > --
>         > Pranav Bhole
>         > Student of MS in Computer Science for Fall 2012,
>         > University of Texas at Dallas
>         > http://www.linkedin.com/in/pranavbhole
>         > Cell Phone No: 972-978-6108 <tel:972-978-6108>.
>         >
>         >
>
>
>
>         --
>         http://www.grobmeier.de
>         https://www.timeandbill.de
>
>         ---------------------------------------------------------------------
>         To unsubscribe, e-mail:
>         log4j-dev-unsubscribe@logging.apache.org
>         <ma...@logging.apache.org>
>         For additional commands, e-mail:
>         log4j-dev-help@logging.apache.org
>         <ma...@logging.apache.org>
>
>
>
>
>     -- 
>     E-Mail: garydgregory@gmail.com <ma...@gmail.com> |
>     ggregory@apache.org <ma...@apache.org>
>     JUnit in Action, 2nd Ed: http://bit.ly/ECvg0
>     Spring Batch in Action: http://bit.ly/bqpbCK
>     Blog: http://garygregory.wordpress.com
>     <http://garygregory.wordpress.com/>
>     Home: http://garygregory.com/
>     Tweet! http://twitter.com/GaryGregory
>
>
>
>
> -- 
> Pranav Bhole
> Student of MS in Computer Science for Fall 2012,
> University of Texas at Dallas
> http://www.linkedin.com/in/pranavbhole
> Cell Phone No: 972-978-6108.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-dev-help@logging.apache.org


Re: Idea of Cloud as target for the bulk of log file

Posted by Christian Grobmeier <gr...@gmail.com>.
Hi,

sorry for my delay.

It looks pretty interesting, and your proposal seems to be well thought.

Ralph, Gary esp, do you have any more comments?

Pranav, did you speak to the Hadoop community about this proposal?
While I think it would fit very good into Logging, I personally have
no clue on Hadoop. Maybe it would be good to have a "mixed mentor"
team from Hadoop and Logging which can help you on technical things.

Cheers
Christian

On Fri, Mar 29, 2013 at 5:18 AM, Pranav Bhole <pr...@gmail.com> wrote:
> Hello to All,
> As per our discussion I am attaching the "Blueprint of Simple, Efficient and
> Reliable HadoopAppender in Log4j". I would love see the suggestions.
>
> Thank you
>
>
> On Fri, Mar 22, 2013 at 8:42 PM, Gary Gregory <ga...@gmail.com>
> wrote:
>>
>> An AppScale appender might be interesting:
>> http://appscale.cs.ucsb.edu/datastores.html
>>
>> Gary
>>
>>
>> On Fri, Mar 22, 2013 at 4:34 PM, Christian Grobmeier <gr...@gmail.com>
>> wrote:
>>>
>>> +1
>>>
>>> I would love to support a GSOC student, and if your more concrete
>>> proposal meets some interest here I am willing to actually help.
>>> That said, while Apache Flume is great, its maybe a bit "too much". I
>>> already have had some thoughts on some kind of a server which utilizes
>>> receivers to send data to $x. Less features than Flume, but easy to
>>> setup. Not sure if that has some value.
>>>
>>> Pranav, please let us hear more of your ideas.
>>>
>>>
>>>
>>> On Fri, Mar 22, 2013 at 4:40 PM, Ralph Goers <ra...@dslextreme.com>
>>> wrote:
>>> > The Flume Appender leverages Apache Flume to route data into various
>>> > places.
>>> > The primary sponsor of Flume is Cloudera, so naturally Flume supports
>>> > writing data into Hadoop. In addition, my employer is using Flume to
>>> > write
>>> > data into Cassandra.  That said, we would welcome contributions and if
>>> > you
>>> > can provide more details on how you would implement your idea I'd love
>>> > to
>>> > see them.  Perhaps you can create a page on the logging wiki with your
>>> > proposal.
>>> >
>>> > Ralph
>>> >
>>> > On Mar 22, 2013, at 2:04 AM, Pranav Bhole wrote:
>>> >
>>> > Hello to all,
>>> >         This is Pranav Bhole, I am Master student at The University of
>>> > Texas
>>> > at Dallas. My research interest is Big Data. I haven been using Log4j
>>> > extensively as core since 5-6 years in my academic and professional
>>> > work.
>>> > Recently an idea came up in my mind by facing some of the difficulties
>>> > in
>>> > managing the TeraBytes of Log files. I would like to implement this
>>> > idea as
>>> > plug in or functionality in the existing log4j appender module as
>>> > student of
>>> > Google Summer of Code 2013.
>>> >
>>> > Short description of the idea:
>>> > Server appends the bulk of log files and in the most cases server lacks
>>> > with
>>> > the storage space for these logs files and also computing on such bulk
>>> > of
>>> > file is costly for the server. With the consideration of this problem,
>>> > idea
>>> > proposes to write a module which could be able to move these files into
>>> > Public (S3 of AWS, Azure) or private cloud (Hadoop) on the rolling
>>> > basis
>>> > based on the configuration file. To resolve the computing layer
>>> > objective,
>>> > the idea proposes the Big Data Query generator based on the logging
>>> > format
>>> > used. Such Big Data Queries will include MapReduce, PIG etc.
>>> > Administrator
>>> > would be able to run these BigData queries generated by Log4j to track
>>> > the
>>> > keywords in the logs like error number, TimeStamp or any other
>>> > arbitrary
>>> > string.
>>> >
>>> > I would like to appreciate to all of you for reading this idea. I would
>>> > really love to get involved in Log4j development team with your support
>>> > and
>>> > suggestion on this idea.
>>> >
>>> > Thank you very much.
>>> >
>>> > --
>>> > Pranav Bhole
>>> > Student of MS in Computer Science for Fall 2012,
>>> > University of Texas at Dallas
>>> > http://www.linkedin.com/in/pranavbhole
>>> > Cell Phone No: 972-978-6108.
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> http://www.grobmeier.de
>>> https://www.timeandbill.de
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
>>> For additional commands, e-mail: log4j-dev-help@logging.apache.org
>>>
>>
>>
>>
>> --
>> E-Mail: garydgregory@gmail.com | ggregory@apache.org
>> JUnit in Action, 2nd Ed: http://bit.ly/ECvg0
>> Spring Batch in Action: http://bit.ly/bqpbCK
>> Blog: http://garygregory.wordpress.com
>> Home: http://garygregory.com/
>> Tweet! http://twitter.com/GaryGregory
>
>
>
>
> --
> Pranav Bhole
> Student of MS in Computer Science for Fall 2012,
> University of Texas at Dallas
> http://www.linkedin.com/in/pranavbhole
> Cell Phone No: 972-978-6108.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-dev-help@logging.apache.org



-- 
http://www.grobmeier.de
https://www.timeandbill.de

---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org


Re: Idea of Cloud as target for the bulk of log file

Posted by Pranav Bhole <pr...@gmail.com>.
Hello to All,
As per our discussion I am attaching the "Blueprint of Simple, Efficient
and Reliable HadoopAppender in Log4j". I would love see the suggestions.

Thank you


On Fri, Mar 22, 2013 at 8:42 PM, Gary Gregory <ga...@gmail.com>wrote:

> An AppScale appender might be interesting:
> http://appscale.cs.ucsb.edu/datastores.html
>
> Gary
>
>
> On Fri, Mar 22, 2013 at 4:34 PM, Christian Grobmeier <gr...@gmail.com>wrote:
>
>> +1
>>
>> I would love to support a GSOC student, and if your more concrete
>> proposal meets some interest here I am willing to actually help.
>> That said, while Apache Flume is great, its maybe a bit "too much". I
>> already have had some thoughts on some kind of a server which utilizes
>> receivers to send data to $x. Less features than Flume, but easy to
>> setup. Not sure if that has some value.
>>
>> Pranav, please let us hear more of your ideas.
>>
>>
>>
>> On Fri, Mar 22, 2013 at 4:40 PM, Ralph Goers <ra...@dslextreme.com>
>> wrote:
>> > The Flume Appender leverages Apache Flume to route data into various
>> places.
>> > The primary sponsor of Flume is Cloudera, so naturally Flume supports
>> > writing data into Hadoop. In addition, my employer is using Flume to
>> write
>> > data into Cassandra.  That said, we would welcome contributions and if
>> you
>> > can provide more details on how you would implement your idea I'd love
>> to
>> > see them.  Perhaps you can create a page on the logging wiki with your
>> > proposal.
>> >
>> > Ralph
>> >
>> > On Mar 22, 2013, at 2:04 AM, Pranav Bhole wrote:
>> >
>> > Hello to all,
>> >         This is Pranav Bhole, I am Master student at The University of
>> Texas
>> > at Dallas. My research interest is Big Data. I haven been using Log4j
>> > extensively as core since 5-6 years in my academic and professional
>> work.
>> > Recently an idea came up in my mind by facing some of the difficulties
>> in
>> > managing the TeraBytes of Log files. I would like to implement this
>> idea as
>> > plug in or functionality in the existing log4j appender module as
>> student of
>> > Google Summer of Code 2013.
>> >
>> > Short description of the idea:
>> > Server appends the bulk of log files and in the most cases server lacks
>> with
>> > the storage space for these logs files and also computing on such bulk
>> of
>> > file is costly for the server. With the consideration of this problem,
>> idea
>> > proposes to write a module which could be able to move these files into
>> > Public (S3 of AWS, Azure) or private cloud (Hadoop) on the rolling basis
>> > based on the configuration file. To resolve the computing layer
>> objective,
>> > the idea proposes the Big Data Query generator based on the logging
>> format
>> > used. Such Big Data Queries will include MapReduce, PIG etc.
>> Administrator
>> > would be able to run these BigData queries generated by Log4j to track
>> the
>> > keywords in the logs like error number, TimeStamp or any other arbitrary
>> > string.
>> >
>> > I would like to appreciate to all of you for reading this idea. I would
>> > really love to get involved in Log4j development team with your support
>> and
>> > suggestion on this idea.
>> >
>> > Thank you very much.
>> >
>> > --
>> > Pranav Bhole
>> > Student of MS in Computer Science for Fall 2012,
>> > University of Texas at Dallas
>> > http://www.linkedin.com/in/pranavbhole
>> > Cell Phone No: 972-978-6108.
>> >
>> >
>>
>>
>>
>> --
>> http://www.grobmeier.de
>> https://www.timeandbill.de
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
>> For additional commands, e-mail: log4j-dev-help@logging.apache.org
>>
>>
>
>
> --
> E-Mail: garydgregory@gmail.com | ggregory@apache.org
> JUnit in Action, 2nd Ed: <http://goog_1249600977>http://bit.ly/ECvg0
> Spring Batch in Action: <http://s.apache.org/HOq>http://bit.ly/bqpbCK
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
>



-- 
Pranav Bhole
Student of MS in Computer Science for Fall 2012,
University of Texas at Dallas
http://www.linkedin.com/in/pranavbhole
Cell Phone No: 972-978-6108.

Re: Idea of Cloud as target for the bulk of log file

Posted by Gary Gregory <ga...@gmail.com>.
An AppScale appender might be interesting:
http://appscale.cs.ucsb.edu/datastores.html

Gary


On Fri, Mar 22, 2013 at 4:34 PM, Christian Grobmeier <gr...@gmail.com>wrote:

> +1
>
> I would love to support a GSOC student, and if your more concrete
> proposal meets some interest here I am willing to actually help.
> That said, while Apache Flume is great, its maybe a bit "too much". I
> already have had some thoughts on some kind of a server which utilizes
> receivers to send data to $x. Less features than Flume, but easy to
> setup. Not sure if that has some value.
>
> Pranav, please let us hear more of your ideas.
>
>
>
> On Fri, Mar 22, 2013 at 4:40 PM, Ralph Goers <ra...@dslextreme.com>
> wrote:
> > The Flume Appender leverages Apache Flume to route data into various
> places.
> > The primary sponsor of Flume is Cloudera, so naturally Flume supports
> > writing data into Hadoop. In addition, my employer is using Flume to
> write
> > data into Cassandra.  That said, we would welcome contributions and if
> you
> > can provide more details on how you would implement your idea I'd love to
> > see them.  Perhaps you can create a page on the logging wiki with your
> > proposal.
> >
> > Ralph
> >
> > On Mar 22, 2013, at 2:04 AM, Pranav Bhole wrote:
> >
> > Hello to all,
> >         This is Pranav Bhole, I am Master student at The University of
> Texas
> > at Dallas. My research interest is Big Data. I haven been using Log4j
> > extensively as core since 5-6 years in my academic and professional work.
> > Recently an idea came up in my mind by facing some of the difficulties in
> > managing the TeraBytes of Log files. I would like to implement this idea
> as
> > plug in or functionality in the existing log4j appender module as
> student of
> > Google Summer of Code 2013.
> >
> > Short description of the idea:
> > Server appends the bulk of log files and in the most cases server lacks
> with
> > the storage space for these logs files and also computing on such bulk of
> > file is costly for the server. With the consideration of this problem,
> idea
> > proposes to write a module which could be able to move these files into
> > Public (S3 of AWS, Azure) or private cloud (Hadoop) on the rolling basis
> > based on the configuration file. To resolve the computing layer
> objective,
> > the idea proposes the Big Data Query generator based on the logging
> format
> > used. Such Big Data Queries will include MapReduce, PIG etc.
> Administrator
> > would be able to run these BigData queries generated by Log4j to track
> the
> > keywords in the logs like error number, TimeStamp or any other arbitrary
> > string.
> >
> > I would like to appreciate to all of you for reading this idea. I would
> > really love to get involved in Log4j development team with your support
> and
> > suggestion on this idea.
> >
> > Thank you very much.
> >
> > --
> > Pranav Bhole
> > Student of MS in Computer Science for Fall 2012,
> > University of Texas at Dallas
> > http://www.linkedin.com/in/pranavbhole
> > Cell Phone No: 972-978-6108.
> >
> >
>
>
>
> --
> http://www.grobmeier.de
> https://www.timeandbill.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-dev-help@logging.apache.org
>
>


-- 
E-Mail: garydgregory@gmail.com | ggregory@apache.org
JUnit in Action, 2nd Ed: <http://goog_1249600977>http://bit.ly/ECvg0
Spring Batch in Action: <http://s.apache.org/HOq>http://bit.ly/bqpbCK
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

Re: Idea of Cloud as target for the bulk of log file

Posted by Christian Grobmeier <gr...@gmail.com>.
+1

I would love to support a GSOC student, and if your more concrete
proposal meets some interest here I am willing to actually help.
That said, while Apache Flume is great, its maybe a bit "too much". I
already have had some thoughts on some kind of a server which utilizes
receivers to send data to $x. Less features than Flume, but easy to
setup. Not sure if that has some value.

Pranav, please let us hear more of your ideas.



On Fri, Mar 22, 2013 at 4:40 PM, Ralph Goers <ra...@dslextreme.com> wrote:
> The Flume Appender leverages Apache Flume to route data into various places.
> The primary sponsor of Flume is Cloudera, so naturally Flume supports
> writing data into Hadoop. In addition, my employer is using Flume to write
> data into Cassandra.  That said, we would welcome contributions and if you
> can provide more details on how you would implement your idea I'd love to
> see them.  Perhaps you can create a page on the logging wiki with your
> proposal.
>
> Ralph
>
> On Mar 22, 2013, at 2:04 AM, Pranav Bhole wrote:
>
> Hello to all,
>         This is Pranav Bhole, I am Master student at The University of Texas
> at Dallas. My research interest is Big Data. I haven been using Log4j
> extensively as core since 5-6 years in my academic and professional work.
> Recently an idea came up in my mind by facing some of the difficulties in
> managing the TeraBytes of Log files. I would like to implement this idea as
> plug in or functionality in the existing log4j appender module as student of
> Google Summer of Code 2013.
>
> Short description of the idea:
> Server appends the bulk of log files and in the most cases server lacks with
> the storage space for these logs files and also computing on such bulk of
> file is costly for the server. With the consideration of this problem, idea
> proposes to write a module which could be able to move these files into
> Public (S3 of AWS, Azure) or private cloud (Hadoop) on the rolling basis
> based on the configuration file. To resolve the computing layer objective,
> the idea proposes the Big Data Query generator based on the logging format
> used. Such Big Data Queries will include MapReduce, PIG etc. Administrator
> would be able to run these BigData queries generated by Log4j to track the
> keywords in the logs like error number, TimeStamp or any other arbitrary
> string.
>
> I would like to appreciate to all of you for reading this idea. I would
> really love to get involved in Log4j development team with your support and
> suggestion on this idea.
>
> Thank you very much.
>
> --
> Pranav Bhole
> Student of MS in Computer Science for Fall 2012,
> University of Texas at Dallas
> http://www.linkedin.com/in/pranavbhole
> Cell Phone No: 972-978-6108.
>
>



-- 
http://www.grobmeier.de
https://www.timeandbill.de

---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org


Re: Idea of Cloud as target for the bulk of log file

Posted by Ralph Goers <ra...@dslextreme.com>.
The Flume Appender leverages Apache Flume to route data into various places. The primary sponsor of Flume is Cloudera, so naturally Flume supports writing data into Hadoop. In addition, my employer is using Flume to write data into Cassandra.  That said, we would welcome contributions and if you can provide more details on how you would implement your idea I'd love to see them.  Perhaps you can create a page on the logging wiki with your proposal.

Ralph

On Mar 22, 2013, at 2:04 AM, Pranav Bhole wrote:

> Hello to all, 
>         This is Pranav Bhole, I am Master student at The University of Texas at Dallas. My research interest is Big Data. I haven been using Log4j extensively as core since 5-6 years in my academic and professional work. Recently an idea came up in my mind by facing some of the difficulties in managing the TeraBytes of Log files. I would like to implement this idea as plug in or functionality in the existing log4j appender module as student of Google Summer of Code 2013.
> 
> Short description of the idea: 
> Server appends the bulk of log files and in the most cases server lacks with the storage space for these logs files and also computing on such bulk of file is costly for the server. With the consideration of this problem, idea proposes to write a module which could be able to move these files into Public (S3 of AWS, Azure) or private cloud (Hadoop) on the rolling basis based on the configuration file. To resolve the computing layer objective, the idea proposes the Big Data Query generator based on the logging format used. Such Big Data Queries will include MapReduce, PIG etc. Administrator would be able to run these BigData queries generated by Log4j to track the keywords in the logs like error number, TimeStamp or any other arbitrary string.
> 
> I would like to appreciate to all of you for reading this idea. I would really love to get involved in Log4j development team with your support and suggestion on this idea. 
> 
> Thank you very much.
> 
> -- 
> Pranav Bhole
> Student of MS in Computer Science for Fall 2012,
> University of Texas at Dallas
> http://www.linkedin.com/in/pranavbhole
> Cell Phone No: 972-978-6108.