You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-dev@logging.apache.org by Ralph Goers <ra...@dslextreme.com> on 2013/03/02 18:15:55 UTC
Berkeley DB for Java
Does anyone have experience using Berkeley DB for Java? I am looking at creating something similar to the embedded Flume appender using Berkeley DB as the "file channel" so that I don't have to bring in all the Flume dependencies and the socket connection to Flume can be done asynchronously. In effect, I would be using Berkeley DB as the queue between the client and the asynchronous portion of the Appender.
I'd like to know if I am opening myself up for problems in doing this so if anyone has experience with using it I'd like to know about them.
Ralph
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org
Re: Berkeley DB for Java
Posted by Tushar Kapila <tg...@gmail.com>.
would a memory mapped file as the back end for the db help?
If you can use j7 why not the in built one - apache.
We use a db mapped to ram (linux) server restart would be catatrophe but
works great for debug logs.
On Mar 5, 2013 12:02 AM, "Ralph Goers" <ra...@dslextreme.com> wrote:
> Do you have personal experience with any of these?
>
> Ralph
>
>
> On Mar 4, 2013, at 1:19 AM, Noel Grandin wrote:
>
> > On 2013-03-02 19:15, Ralph Goers wrote:
> >> Does anyone have experience using Berkeley DB for Java? I am looking
> at creating something similar to the embedded Flume appender using Berkeley
> DB as the "file channel" so that I don't have to bring in all the Flume
> dependencies and the socket connection to Flume can be done asynchronously.
> In effect, I would be using Berkeley DB as the queue between the client
> and the asynchronous portion of the Appender.
> >>
> >> I'd like to know if I am opening myself up for problems in doing this
> so if anyone has experience with using it I'd like to know about them.
> >>
> >>
> >
> >
> > If you want something really simply, then something like one of these
> should fit the bill: ( in order of increasing complexity)
> >
> > http://www.gaborcselle.com/open_source/java/persistent_queue.html
> > http://www.michaelgoldschmidt.eu/memoryfilebackedqueue/index.html
> > http://www.metanotion.net/software/sandbox/block.html
> > http://jdbm.sourceforge.net/
> >
> >
> >
> > Disclaimer: http://www.peralex.com/disclaimer.html
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-dev-help@logging.apache.org
>
>
Re: Berkeley DB for Java
Posted by Ralph Goers <ra...@dslextreme.com>.
Do you have personal experience with any of these?
Ralph
On Mar 4, 2013, at 1:19 AM, Noel Grandin wrote:
> On 2013-03-02 19:15, Ralph Goers wrote:
>> Does anyone have experience using Berkeley DB for Java? I am looking at creating something similar to the embedded Flume appender using Berkeley DB as the "file channel" so that I don't have to bring in all the Flume dependencies and the socket connection to Flume can be done asynchronously. In effect, I would be using Berkeley DB as the queue between the client and the asynchronous portion of the Appender.
>>
>> I'd like to know if I am opening myself up for problems in doing this so if anyone has experience with using it I'd like to know about them.
>>
>>
>
>
> If you want something really simply, then something like one of these should fit the bill: ( in order of increasing complexity)
>
> http://www.gaborcselle.com/open_source/java/persistent_queue.html
> http://www.michaelgoldschmidt.eu/memoryfilebackedqueue/index.html
> http://www.metanotion.net/software/sandbox/block.html
> http://jdbm.sourceforge.net/
>
>
>
> Disclaimer: http://www.peralex.com/disclaimer.html
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org
Re: Berkeley DB for Java
Posted by Noel Grandin <no...@peralex.com>.
On 2013-03-02 19:15, Ralph Goers wrote:
> Does anyone have experience using Berkeley DB for Java? I am looking at creating something similar to the embedded Flume appender using Berkeley DB as the "file channel" so that I don't have to bring in all the Flume dependencies and the socket connection to Flume can be done asynchronously. In effect, I would be using Berkeley DB as the queue between the client and the asynchronous portion of the Appender.
>
> I'd like to know if I am opening myself up for problems in doing this so if anyone has experience with using it I'd like to know about them.
>
>
If you want something really simply, then something like one of these
should fit the bill: ( in order of increasing complexity)
http://www.gaborcselle.com/open_source/java/persistent_queue.html
http://www.michaelgoldschmidt.eu/memoryfilebackedqueue/index.html
http://www.metanotion.net/software/sandbox/block.html
http://jdbm.sourceforge.net/
Disclaimer: http://www.peralex.com/disclaimer.html
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org
Re: Berkeley DB for Java
Posted by Ralph Goers <rg...@apache.org>.
JMS guaranteed delivery is achieved by sending the message to the JMS server and then writing it to disk similar to what Flume does. Doing it locally before sending is an order of magnitude faster. At least that is the difference between the remote and embedded Flume appenders.
Ralph
On Mar 2, 2013, at 4:00 PM, Gary Gregory <ga...@gmail.com> wrote:
> Sounds like a job for a JMS provider like ActiveMQ.
>
> Gary
>
> On Mar 2, 2013, at 18:50, Ralph Goers <ra...@dslextreme.com> wrote:
>
>> What I am looking for is simply a Queue interface that persists log events to disk such that control can be returned to the caller immediately after writing to the queue and even if the JVM crashes it is guaranteed that the log event will be delivered even after a restart. I've found some code that lets me do that with Berkeley DB for Java. I know that that code base is quite a bit newer than Berkeley DB itself and I haven't ever heard discussions from anyone using it to know what kinds of problems there may be. All the uses for HSQL I've ever seen basically say not to use it for production stuff, although I don't know why. Plus, I really don't need a full RDBMS to implement a queue.
>>
>> Ralph
>>
>> On Mar 2, 2013, at 12:00 PM, Gary Gregory wrote:
>>
>>> If you are looking for a Java embeddable database, I've used H2 for unit testing a JDBC piece instead of using heavier databases like MySQL, Oracle, MSSQL and so on.
>>>
>>> G
>>>
>>>
>>> On Sat, Mar 2, 2013 at 12:15 PM, Ralph Goers <ra...@dslextreme.com> wrote:
>>>> Does anyone have experience using Berkeley DB for Java? I am looking at creating something similar to the embedded Flume appender using Berkeley DB as the "file channel" so that I don't have to bring in all the Flume dependencies and the socket connection to Flume can be done asynchronously. In effect, I would be using Berkeley DB as the queue between the client and the asynchronous portion of the Appender.
>>>>
>>>> I'd like to know if I am opening myself up for problems in doing this so if anyone has experience with using it I'd like to know about them.
>>>>
>>>> Ralph
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
>>>> For additional commands, e-mail: log4j-dev-help@logging.apache.org
>>>
>>>
>>>
>>> --
>>> E-Mail: garydgregory@gmail.com | ggregory@apache.org
>>> JUnit in Action, 2nd Ed: http://bit.ly/ECvg0
>>> Spring Batch in Action: http://bit.ly/bqpbCK
>>> Blog: http://garygregory.wordpress.com
>>> Home: http://garygregory.com/
>>> Tweet! http://twitter.com/GaryGregory
Re: Berkeley DB for Java
Posted by Gary Gregory <ga...@gmail.com>.
Sounds like a job for a JMS provider like ActiveMQ.
Gary
On Mar 2, 2013, at 18:50, Ralph Goers <ra...@dslextreme.com> wrote:
What I am looking for is simply a Queue interface that persists log events
to disk such that control can be returned to the caller immediately after
writing to the queue and even if the JVM crashes it is guaranteed that the
log event will be delivered even after a restart. I've found some code
that lets me do that with Berkeley DB for Java. I know that that code base
is quite a bit newer than Berkeley DB itself and I haven't ever heard
discussions from anyone using it to know what kinds of problems there may
be. All the uses for HSQL I've ever seen basically say not to use it for
production stuff, although I don't know why. Plus, I really don't need a
full RDBMS to implement a queue.
Ralph
On Mar 2, 2013, at 12:00 PM, Gary Gregory wrote:
If you are looking for a Java embeddable database, I've used H2 for unit
testing a JDBC piece instead of using heavier databases like MySQL, Oracle,
MSSQL and so on.
G
On Sat, Mar 2, 2013 at 12:15 PM, Ralph Goers <ra...@dslextreme.com>wrote:
> Does anyone have experience using Berkeley DB for Java? I am looking at
> creating something similar to the embedded Flume appender using Berkeley DB
> as the "file channel" so that I don't have to bring in all the Flume
> dependencies and the socket connection to Flume can be done asynchronously.
> In effect, I would be using Berkeley DB as the queue between the client
> and the asynchronous portion of the Appender.
>
> I'd like to know if I am opening myself up for problems in doing this so
> if anyone has experience with using it I'd like to know about them.
>
> Ralph
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-dev-help@logging.apache.org
>
>
--
E-Mail: garydgregory@gmail.com | ggregory@apache.org
JUnit in Action, 2nd Ed: <http://goog_1249600977/>http://bit.ly/ECvg0
Spring Batch in Action: <http://s.apache.org/HOq>http://bit.ly/bqpbCK
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory
Re: Berkeley DB for Java
Posted by Ralph Goers <ra...@dslextreme.com>.
What I am looking for is simply a Queue interface that persists log events to disk such that control can be returned to the caller immediately after writing to the queue and even if the JVM crashes it is guaranteed that the log event will be delivered even after a restart. I've found some code that lets me do that with Berkeley DB for Java. I know that that code base is quite a bit newer than Berkeley DB itself and I haven't ever heard discussions from anyone using it to know what kinds of problems there may be. All the uses for HSQL I've ever seen basically say not to use it for production stuff, although I don't know why. Plus, I really don't need a full RDBMS to implement a queue.
Ralph
On Mar 2, 2013, at 12:00 PM, Gary Gregory wrote:
> If you are looking for a Java embeddable database, I've used H2 for unit testing a JDBC piece instead of using heavier databases like MySQL, Oracle, MSSQL and so on.
>
> G
>
>
> On Sat, Mar 2, 2013 at 12:15 PM, Ralph Goers <ra...@dslextreme.com> wrote:
> Does anyone have experience using Berkeley DB for Java? I am looking at creating something similar to the embedded Flume appender using Berkeley DB as the "file channel" so that I don't have to bring in all the Flume dependencies and the socket connection to Flume can be done asynchronously. In effect, I would be using Berkeley DB as the queue between the client and the asynchronous portion of the Appender.
>
> I'd like to know if I am opening myself up for problems in doing this so if anyone has experience with using it I'd like to know about them.
>
> Ralph
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-dev-help@logging.apache.org
>
>
>
>
> --
> E-Mail: garydgregory@gmail.com | ggregory@apache.org
> JUnit in Action, 2nd Ed: http://bit.ly/ECvg0
> Spring Batch in Action: http://bit.ly/bqpbCK
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
Re: Berkeley DB for Java
Posted by Gary Gregory <ga...@gmail.com>.
If you are looking for a Java embeddable database, I've used H2 for unit
testing a JDBC piece instead of using heavier databases like MySQL, Oracle,
MSSQL and so on.
G
On Sat, Mar 2, 2013 at 12:15 PM, Ralph Goers <ra...@dslextreme.com>wrote:
> Does anyone have experience using Berkeley DB for Java? I am looking at
> creating something similar to the embedded Flume appender using Berkeley DB
> as the "file channel" so that I don't have to bring in all the Flume
> dependencies and the socket connection to Flume can be done asynchronously.
> In effect, I would be using Berkeley DB as the queue between the client
> and the asynchronous portion of the Appender.
>
> I'd like to know if I am opening myself up for problems in doing this so
> if anyone has experience with using it I'd like to know about them.
>
> Ralph
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-dev-help@logging.apache.org
>
>
--
E-Mail: garydgregory@gmail.com | ggregory@apache.org
JUnit in Action, 2nd Ed: <http://goog_1249600977>http://bit.ly/ECvg0
Spring Batch in Action: <http://s.apache.org/HOq>http://bit.ly/bqpbCK
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory
Re: Berkeley DB for Java
Posted by Ralph Goers <ra...@dslextreme.com>.
Christian - I looked at the web site but the documentation is so sparse that I can't figure out how directmemory could be used to solve the problem.
Ralph
On Mar 4, 2013, at 9:44 PM, Christian Grobmeier wrote:
> Not used Berkely DB for anything. But as other throw in options, I
> would like to add another one: http://directmemory.apache.org/
> DirectMemory is a caching system, but they seem to support
> serializers. Not sure how DM works at all, I never dived so deeply
> into it. But I can imagine its worth a look.
>
> On Sat, Mar 2, 2013 at 6:15 PM, Ralph Goers <ra...@dslextreme.com> wrote:
>> Does anyone have experience using Berkeley DB for Java? I am looking at creating something similar to the embedded Flume appender using Berkeley DB as the "file channel" so that I don't have to bring in all the Flume dependencies and the socket connection to Flume can be done asynchronously. In effect, I would be using Berkeley DB as the queue between the client and the asynchronous portion of the Appender.
>>
>> I'd like to know if I am opening myself up for problems in doing this so if anyone has experience with using it I'd like to know about them.
>>
>> Ralph
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
>> For additional commands, e-mail: log4j-dev-help@logging.apache.org
>>
>
>
>
> --
> http://www.grobmeier.de
> https://www.timeandbill.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-dev-help@logging.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org
Re: Berkeley DB for Java
Posted by Christian Grobmeier <gr...@gmail.com>.
Not used Berkely DB for anything. But as other throw in options, I
would like to add another one: http://directmemory.apache.org/
DirectMemory is a caching system, but they seem to support
serializers. Not sure how DM works at all, I never dived so deeply
into it. But I can imagine its worth a look.
On Sat, Mar 2, 2013 at 6:15 PM, Ralph Goers <ra...@dslextreme.com> wrote:
> Does anyone have experience using Berkeley DB for Java? I am looking at creating something similar to the embedded Flume appender using Berkeley DB as the "file channel" so that I don't have to bring in all the Flume dependencies and the socket connection to Flume can be done asynchronously. In effect, I would be using Berkeley DB as the queue between the client and the asynchronous portion of the Appender.
>
> I'd like to know if I am opening myself up for problems in doing this so if anyone has experience with using it I'd like to know about them.
>
> Ralph
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-dev-help@logging.apache.org
>
--
http://www.grobmeier.de
https://www.timeandbill.de
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org