You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@esme.apache.org by Ethan Jewett <es...@gmail.com> on 2011/08/03 13:56:32 UTC
Re: Scalability was Re: What are pools? was Re: [VOTE] Approve the
release of apache-esme-1.3
Hi Jack,
It should be able to run on any Java application server with any
database via JNDI or with any JDBC database by providing JDBC
connection information in a .props file. I believe that on Stax (now
CloudBees) we are using a MySQL database.
Regarding scalability, the design of ESME is such that it doesn't rely
much on the database for scalability. Operations should generally
happen on data in memory, and there shouldn't be access of the
database on most operations. We can do this because the data model has
messages as immutable objects (you can't change or delete them), so
they can be safely cached and we can rest assured that they will never
be out of sync with the database.
I'm fairly sure we did some performance tests a while ago that showed
ESME was able to support thousands of concurrent users on a fairly
small server without trouble, but I can't currently find that
documentation. Maybe someone else knows where it ran off to? Seems
like something we should have on a our website (and a test that we
should redo at some point).
Cheers,
Ethan
On Tue, Aug 2, 2011 at 7:01 PM, Jack Park <ja...@topicquests.org> wrote:
> Thanks!
> Changing subjects again:
>
> Running on Derby,it seems to me, is not a path to a scalable platform. What
> is known or thought possible in terms of scalability?
>
> Thanks
> Jack
>
> On 8/2/2011 5:51 AM, Ethan Jewett wrote:
>>
>> Hi Jack,
>>
>> On the esmecloudserverapache.dickhirsch.staxapps.net server I am
>> seeing that users in pools with "Write" privilege can read and write.
>>
>> *However* there might be a bug where a user that hasn't logged in
>> recently doesn't receive messages from pools. It is possible the
>> user's mailbox is not being set up properly. We'll need to look into
>> it.
>>
>> Thanks,
>> Ethan
>>
>> On Fri, Jul 29, 2011 at 5:24 PM, Jack Park<ja...@topicquests.org>
>> wrote:
>>>
>>> Ethan,
>>>
>>> That makes perfect sense.
>>> In 1.2, I am noticing that if I assign "write" to "joe" in a pool, "joe"
>>> can, indeed, write, but not read. Hope that's fixed in the newer
>>> versions.
>>>
>>> Thanks
>>> Jack
>>>
>>> On 7/28/2011 11:47 PM, Ethan Jewett wrote:
>>>>
>>>> Hi Jack,
>>>>
>>>> Just by way of explanation, I think that there have been some
>>>> significant changes to the timeline Javascript files between 1.2, 1.3,
>>>> and the upcoming 1.4. Most likely your browser was trying to use a
>>>> cached 1.3 .js file with the 1.2 backend code-base. That would
>>>> definitely not work as we changed some function and variable names, so
>>>> the backend was spitting out JS variables and functions to the browser
>>>> but the javascript already running in the browser had no idea what to
>>>> do with it.
>>>>
>>>> Hopefully that explains the problem. Sorry for the apparent flakiness.
>>>>
>>>> Cheers,
>>>> Ethan
>>>>
>>>> On Fri, Jul 29, 2011 at 5:42 AM, Jack Park<ja...@topicquests.org>
>>>> wrote:
>>>>>
>>>>> My "virgin" browser was IE whatever comes with Win7. The others are
>>>>> Chrome
>>>>> and FF. Clearing their cache fixed it. Everything is running well now,
>>>>> with
>>>>> the 1.2 distro. I have not migrated back to trunk or 1.3 yet. Overall,
>>>>> it
>>>>> was the cache issue, but I did drop the database and start over. I now
>>>>> have
>>>>> three users running in it.
>>>>> Created quite a few messages, some into a shared pool for two of the
>>>>> users.
>>>>> I gave a member Write roll. He can write and the admin sees, but he
>>>>> cannot
>>>>> read. Not sure if that's a bug?
>>>>>
>>>>> Looking forward to the latest build running well.
>>>>> Thanks
>>>>> Jack
>>>>>
>>>>> On 7/28/2011 8:18 PM, Richard Hirsch wrote:
>>>>>>
>>>>>> Did you delete the old db? If you deleted the old db, then the old
>>>>>> users are gone and you have to sign up again. If it was the first
>>>>>> message after starting then it might take a while for the message to
>>>>>> appear. Create 5 messages over a few minutes time span and see if the
>>>>>> problem still exists.
>>>>>>
>>>>>> What browser are you using? There was an older bug with
>>>>>> InternetExplorer that some messages weren't being displayed.
>>>>>>
>>>>>> On Fri, Jul 29, 2011 at 5:09 AM, Jack Park<ja...@topicquests.org>
>>>>>> wrote:
>>>>>>>
>>>>>>> I did a clean rebuild of the 1.2 distribution and ran mvn jetty:run
>>>>>>> Logging in does not work at all. If I Sign Up and create a user, then
>>>>>>> I'm
>>>>>>> in
>>>>>>> and I can send a message with a tag. The tag shows, the timeline does
>>>>>>> not.
>>>>>>> But, when clicking users, that user's last message is visible.
>>>>>>> Something
>>>>>>> fails in painting timelines.
>>>>>>> Next trial will be to attempt to replicate this on a completely
>>>>>>> different
>>>>>>> computer.
>>>>>>>
>>>>>>> On 7/28/2011 7:55 PM, Jack Park wrote:
>>>>>>>>
>>>>>>>> All trials were with mvn jetty:run.
>>>>>>>> Thanks for the pointer: every database is separate.
>>>>>>>> I just noticed that each, whether I'm using it or not, has a db.lck
>>>>>>>>
>>>>>>>> I just ran an experiment: removed the db.lck file in the 1.2
>>>>>>>> distribution (I don't know that much about Derby) and now it refuses
>>>>>>>> to
>>>>>>>> respond to the Sign On button.
>>>>>>>>
>>>>>>>> Still fiddling.
>>>>>>>>
>>>>>>>> Jack
>>>>>>>>
>>>>>>>> On 7/28/2011 7:46 PM, Richard Hirsch wrote:
>>>>>>>>>
>>>>>>>>> If you are using the integrated derby database, then you are
>>>>>>>>> probably
>>>>>>>>> using it for multiple versions - which might not be a good thing. I
>>>>>>>>> haven't tried it yet. Look for a folder called esme_db.
>>>>>>>>>
>>>>>>>>> If you are using tomcat, try a clean build and then run esme with
>>>>>>>>> jetty:run to see if the problem still exists. You may also want to
>>>>>>>>> delete your browser cookies. That might be a problem as well.
>>>>>>>>>
>>>>>>>>> D.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jul 29, 2011 at 4:39 AM, Jack
>>>>>>>>> Park<ja...@topicquests.org>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> What I mean is that the timelines stopped working entirely, but
>>>>>>>>>> the
>>>>>>>>>> tags
>>>>>>>>>> that were visible (this is the 1.2 distribution tarball).
>>>>>>>>>>
>>>>>>>>>> I shut the system down and built the 1.3 rc-1 distribution tarball
>>>>>>>>>> and ran
>>>>>>>>>> that only to discover that tags were not working there. I then
>>>>>>>>>> went
>>>>>>>>>> back to
>>>>>>>>>> the original 1.2 distro, booted it, noted my two users could still
>>>>>>>>>> log in,
>>>>>>>>>> but there are no timelines visible. Sending messages do not appear
>>>>>>>>>> anywhere.
>>>>>>>>>>
>>>>>>>>>> I then went and booted the subversion 1.2 rc-1 tag; no tags, and
>>>>>>>>>> adding a
>>>>>>>>>> user to a pool fails.
>>>>>>>>>>
>>>>>>>>>> I then went and booted the subversion trunk. Same issues.
>>>>>>>>>>
>>>>>>>>>> All that to say that I had active timelines in the 1.2
>>>>>>>>>> distribution
>>>>>>>>>> until I
>>>>>>>>>> started running other distributions. By way of weak inference, I
>>>>>>>>>> have
>>>>>>>>>> to
>>>>>>>>>> wonder if there is anything like a shared database among those
>>>>>>>>>> builds.
>>>>>>>>>>
>>>>>>>>>> Jack
>>>>>>>>>>
>>>>>>>>>> On 7/28/2011 7:33 PM, Richard Hirsch wrote:
>>>>>>>>>>>
>>>>>>>>>>> What do you mean by " no messages sent from it show up anywhere"?
>>>>>>>>>>> There are no messages in the timeline? Tagged messages?
>>>>>>>>>>>
>>>>>>>>>>> D.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jul 28, 2011 at 8:21 PM, Jack
>>>>>>>>>>> Park<ja...@topicquests.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I returned to 1.2 and found, now, that it remembers the tags,
>>>>>>>>>>>> but
>>>>>>>>>>>> not any
>>>>>>>>>>>> of
>>>>>>>>>>>> the messages. In fact, no messages sent from it show up
>>>>>>>>>>>> anywhere.
>>>>>>>>>>>> Time
>>>>>>>>>>>> for
>>>>>>>>>>>> some coffee, I suppose.
>>>>>>>>>>>>
>>>>>>>>>>>> On 7/28/2011 11:10 AM, Richard Hirsch wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I just tried it on the stax instance and added the user
>>>>>>>>>>>>> "acmebob"
>>>>>>>>>>>>> to a
>>>>>>>>>>>>> pool and it worked.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding the "Edit" button for pools. Right now it is not
>>>>>>>>>>>>> correctly
>>>>>>>>>>>>> labeled, because you can just add users. But we plan to be able
>>>>>>>>>>>>> to
>>>>>>>>>>>>> change other parts of the group (rename, etc) and then
>>>>>>>>>>>>> hopefully
>>>>>>>>>>>>> the
>>>>>>>>>>>>> "edit" label will be correct.
>>>>>>>>>>>>>
>>>>>>>>>>>>> D.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jul 28, 2011 at 8:06 PM, Jack
>>>>>>>>>>>>> Park<ja...@topicquests.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Where the messages never stop!
>>>>>>>>>>>>>> Went back to 1.2 with what I learned playing with 1.3.
>>>>>>>>>>>>>> 1.2 lets me add "joe" as a user to my pool.
>>>>>>>>>>>>>> So, I suspect 1.3 is broken in terms of adding users to pools.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Jack
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 7/28/2011 10:59 AM, Jack Park wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Slight update. What I know about pools came from 1.2. In 1.3,
>>>>>>>>>>>>>>> pools
>>>>>>>>>>>>>>> behave a bit differently, or so it seems. I found that the
>>>>>>>>>>>>>>> (misnamed?)
>>>>>>>>>>>>>>> Edit button lets me add a member. I tried adding the other
>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>> ("joe").
>>>>>>>>>>>>>>> Clicking add user does nothing. "joe" is a real user visible
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> user's list.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I hope to learn what you can do when you have a pool. Can you
>>>>>>>>>>>>>>> assign
>>>>>>>>>>>>>>> messages to the pool? Can pools be private?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Jack
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 7/28/2011 10:39 AM, Jack Park wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Now that I've got your attention, may I ask for links to
>>>>>>>>>>>>>>>> explanations
>>>>>>>>>>>>>>>> about pools and, while we're at it, tokens? I have some
>>>>>>>>>>>>>>>> hints
>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>> tokens, but absolutely no clue about pools. I created one,
>>>>>>>>>>>>>>>> thinking
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> might be a "group" (a feature I want), but when I came in as
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> different
>>>>>>>>>>>>>>>> user, I was not able to "join" that pool.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>> Jack
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>
>>>
>
Re: Scalability was Re: What are pools? was Re: [VOTE] Approve the
release of apache-esme-1.3
Posted by Richard Hirsch <hi...@gmail.com>.
We use MySQL for the CloudBees instance. It is no problem since ESME
uses the J2EE container to find its javax.sql.DataSource.
D.
On Thu, Aug 4, 2011 at 10:37 AM, Ethan Jewett <es...@gmail.com> wrote:
> Hi Jack,
>
> Here is an example of setting up a Lift app with PostgreSQL. The same
> approach would work for MySQL:
> http://www.assembla.com/spaces/liftweb/wiki/Set_Up_Jetty_and_PostgreSQL
>
> ESME currently works in terms of large-ish memory models and load
> balancers, if you really need it. But you should be able to get a
> single application server up to at least 10s of thousands of users, so
> the need for a load-balancer setup is purely theoretical at this
> point.
>
> Regarding your scenario, the point about ESME is that it is designed
> so that very few operations will result in database reads. The history
> of timelines is not easily accessible, and new messages are inserted
> into every subscriber's mailbox at the time of message creation
> (rather than on read). Yes, if someone goes back in history via search
> or another mechanism, we'll need to read the database once, but that
> is a read on the single key field (the message ID) so it should be
> very fast. And the next time that message pops up in search on the
> same application server, it is already cached so no DB read is
> necessary.
>
> We do have a bit of an in-depth explanation of why avoiding the
> database is a good idea for messaging applications on our wiki. You
> can read it here.
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=9373341
>
> Cheers,
> Ethan
>
>
>
> On Wed, Aug 3, 2011 at 9:21 PM, Jack Park <ja...@topicquests.org> wrote:
>> Thanks, Ethan.
>> I'd like to know more about using MySQL.
>> Indeed, I'd like to know more about other classes of databases, e.g. graph
>> databases.
>>
>> Still, while hit rate is one aspect of scalability, when you talk about
>> "safely cached", I wonder about the combinatoric aspect of scalability, say,
>> a million users each with timelines in the hundreds to thousands long, and
>> all the entailed message objects. Are you thinking in terms of, say,
>> external memcached solutions, or just large memory models and load
>> balancers?
>>
>> Curious minds...
>> Thanks
>> Jack
>>
>> On 8/3/2011 4:56 AM, Ethan Jewett wrote:
>>>
>>> Hi Jack,
>>>
>>> It should be able to run on any Java application server with any
>>> database via JNDI or with any JDBC database by providing JDBC
>>> connection information in a .props file. I believe that on Stax (now
>>> CloudBees) we are using a MySQL database.
>>>
>>> Regarding scalability, the design of ESME is such that it doesn't rely
>>> much on the database for scalability. Operations should generally
>>> happen on data in memory, and there shouldn't be access of the
>>> database on most operations. We can do this because the data model has
>>> messages as immutable objects (you can't change or delete them), so
>>> they can be safely cached and we can rest assured that they will never
>>> be out of sync with the database.
>>>
>>> I'm fairly sure we did some performance tests a while ago that showed
>>> ESME was able to support thousands of concurrent users on a fairly
>>> small server without trouble, but I can't currently find that
>>> documentation. Maybe someone else knows where it ran off to? Seems
>>> like something we should have on a our website (and a test that we
>>> should redo at some point).
>>>
>>> Cheers,
>>> Ethan
>>>
>>
>
Re: Scalability was Re: What are pools? was Re: [VOTE] Approve the
release of apache-esme-1.3
Posted by Ethan Jewett <es...@gmail.com>.
Hi Jack,
Here is an example of setting up a Lift app with PostgreSQL. The same
approach would work for MySQL:
http://www.assembla.com/spaces/liftweb/wiki/Set_Up_Jetty_and_PostgreSQL
ESME currently works in terms of large-ish memory models and load
balancers, if you really need it. But you should be able to get a
single application server up to at least 10s of thousands of users, so
the need for a load-balancer setup is purely theoretical at this
point.
Regarding your scenario, the point about ESME is that it is designed
so that very few operations will result in database reads. The history
of timelines is not easily accessible, and new messages are inserted
into every subscriber's mailbox at the time of message creation
(rather than on read). Yes, if someone goes back in history via search
or another mechanism, we'll need to read the database once, but that
is a read on the single key field (the message ID) so it should be
very fast. And the next time that message pops up in search on the
same application server, it is already cached so no DB read is
necessary.
We do have a bit of an in-depth explanation of why avoiding the
database is a good idea for messaging applications on our wiki. You
can read it here.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=9373341
Cheers,
Ethan
On Wed, Aug 3, 2011 at 9:21 PM, Jack Park <ja...@topicquests.org> wrote:
> Thanks, Ethan.
> I'd like to know more about using MySQL.
> Indeed, I'd like to know more about other classes of databases, e.g. graph
> databases.
>
> Still, while hit rate is one aspect of scalability, when you talk about
> "safely cached", I wonder about the combinatoric aspect of scalability, say,
> a million users each with timelines in the hundreds to thousands long, and
> all the entailed message objects. Are you thinking in terms of, say,
> external memcached solutions, or just large memory models and load
> balancers?
>
> Curious minds...
> Thanks
> Jack
>
> On 8/3/2011 4:56 AM, Ethan Jewett wrote:
>>
>> Hi Jack,
>>
>> It should be able to run on any Java application server with any
>> database via JNDI or with any JDBC database by providing JDBC
>> connection information in a .props file. I believe that on Stax (now
>> CloudBees) we are using a MySQL database.
>>
>> Regarding scalability, the design of ESME is such that it doesn't rely
>> much on the database for scalability. Operations should generally
>> happen on data in memory, and there shouldn't be access of the
>> database on most operations. We can do this because the data model has
>> messages as immutable objects (you can't change or delete them), so
>> they can be safely cached and we can rest assured that they will never
>> be out of sync with the database.
>>
>> I'm fairly sure we did some performance tests a while ago that showed
>> ESME was able to support thousands of concurrent users on a fairly
>> small server without trouble, but I can't currently find that
>> documentation. Maybe someone else knows where it ran off to? Seems
>> like something we should have on a our website (and a test that we
>> should redo at some point).
>>
>> Cheers,
>> Ethan
>>
>
Re: Scalability was Re: What are pools? was Re: [VOTE] Approve the
release of apache-esme-1.3
Posted by Vladimir Ivanov <le...@gmail.com>.
Correct,
There is no out-of-the-box Lift module for any graph database AFAIK. I've
found couple of threads on Lift Google Group about Neo4J and OrientDB:
http://groups.google.com/group/liftweb/browse_thread/thread/e73579ed220ec4d3/55b891f0354e8440?lnk=gst&q=neo4j#55b891f0354e8440
,
former has incompatible license with Lift's one, there's no any info about
latter and Lift compatibility.
Vladimir
2011/8/4 Jack Park <ja...@topicquests.org>
> Looking in the distribution, I see MongoDB, and CouchDB, plus general db
> paths. Looking through code, I've seen Mongo and Couch Lift projects at
> github, and have code for "vendors" like Derby and H2. I've not seen a mysql
> vendor yet, but guess (without yet trying) that you just insert the right
> class path and URL to use MySQL.
>
> I've not seen any other nosql projects; am interested in graph databases,
> e.g. as driven by tinkerpop DSLs (which may soon migrate to scala), or
> triplestores. Dream on!
>
> Jack
>
>
> On 8/3/2011 4:05 PM, Vladimir Ivanov wrote:
>
>> Ethan,
>>
>> Correct me if I'm wrong, but at the moment ESME should be able to run on
>> any
>> *relational* database due to all entities are stored/retreived via Lift
>> Mapper persistence layer which implies traditional relational store. It's
>> absolutly possible to use Lift Record persistence layer though which is
>> intended for NoSQL databases, such as MongoDB.
>>
>> Vladimir
>>
>> 2011/8/3 Jack Park<ja...@topicquests.org>
>>
>> Thanks, Ethan.
>>> I'd like to know more about using MySQL.
>>> Indeed, I'd like to know more about other classes of databases, e.g.
>>> graph
>>> databases.
>>>
>>> Still, while hit rate is one aspect of scalability, when you talk about
>>> "safely cached", I wonder about the combinatoric aspect of scalability,
>>> say,
>>> a million users each with timelines in the hundreds to thousands long,
>>> and
>>> all the entailed message objects. Are you thinking in terms of, say,
>>> external memcached solutions, or just large memory models and load
>>> balancers?
>>>
>>> Curious minds...
>>> Thanks
>>> Jack
>>>
>>>
>>> On 8/3/2011 4:56 AM, Ethan Jewett wrote:
>>>
>>> Hi Jack,
>>>>
>>>> It should be able to run on any Java application server with any
>>>> database via JNDI or with any JDBC database by providing JDBC
>>>> connection information in a .props file. I believe that on Stax (now
>>>> CloudBees) we are using a MySQL database.
>>>>
>>>> Regarding scalability, the design of ESME is such that it doesn't rely
>>>> much on the database for scalability. Operations should generally
>>>> happen on data in memory, and there shouldn't be access of the
>>>> database on most operations. We can do this because the data model has
>>>> messages as immutable objects (you can't change or delete them), so
>>>> they can be safely cached and we can rest assured that they will never
>>>> be out of sync with the database.
>>>>
>>>> I'm fairly sure we did some performance tests a while ago that showed
>>>> ESME was able to support thousands of concurrent users on a fairly
>>>> small server without trouble, but I can't currently find that
>>>> documentation. Maybe someone else knows where it ran off to? Seems
>>>> like something we should have on a our website (and a test that we
>>>> should redo at some point).
>>>>
>>>> Cheers,
>>>> Ethan
>>>>
>>>>
>>>>
>>
>>
--
Best Regards,
Vladimir Ivanov
Re: Scalability was Re: What are pools? was Re: [VOTE] Approve the
release of apache-esme-1.3
Posted by Jack Park <ja...@topicquests.org>.
Looking in the distribution, I see MongoDB, and CouchDB, plus general db
paths. Looking through code, I've seen Mongo and Couch Lift projects at
github, and have code for "vendors" like Derby and H2. I've not seen a
mysql vendor yet, but guess (without yet trying) that you just insert
the right class path and URL to use MySQL.
I've not seen any other nosql projects; am interested in graph
databases, e.g. as driven by tinkerpop DSLs (which may soon migrate to
scala), or triplestores. Dream on!
Jack
On 8/3/2011 4:05 PM, Vladimir Ivanov wrote:
> Ethan,
>
> Correct me if I'm wrong, but at the moment ESME should be able to run on any
> *relational* database due to all entities are stored/retreived via Lift
> Mapper persistence layer which implies traditional relational store. It's
> absolutly possible to use Lift Record persistence layer though which is
> intended for NoSQL databases, such as MongoDB.
>
> Vladimir
>
> 2011/8/3 Jack Park<ja...@topicquests.org>
>
>> Thanks, Ethan.
>> I'd like to know more about using MySQL.
>> Indeed, I'd like to know more about other classes of databases, e.g. graph
>> databases.
>>
>> Still, while hit rate is one aspect of scalability, when you talk about
>> "safely cached", I wonder about the combinatoric aspect of scalability, say,
>> a million users each with timelines in the hundreds to thousands long, and
>> all the entailed message objects. Are you thinking in terms of, say,
>> external memcached solutions, or just large memory models and load
>> balancers?
>>
>> Curious minds...
>> Thanks
>> Jack
>>
>>
>> On 8/3/2011 4:56 AM, Ethan Jewett wrote:
>>
>>> Hi Jack,
>>>
>>> It should be able to run on any Java application server with any
>>> database via JNDI or with any JDBC database by providing JDBC
>>> connection information in a .props file. I believe that on Stax (now
>>> CloudBees) we are using a MySQL database.
>>>
>>> Regarding scalability, the design of ESME is such that it doesn't rely
>>> much on the database for scalability. Operations should generally
>>> happen on data in memory, and there shouldn't be access of the
>>> database on most operations. We can do this because the data model has
>>> messages as immutable objects (you can't change or delete them), so
>>> they can be safely cached and we can rest assured that they will never
>>> be out of sync with the database.
>>>
>>> I'm fairly sure we did some performance tests a while ago that showed
>>> ESME was able to support thousands of concurrent users on a fairly
>>> small server without trouble, but I can't currently find that
>>> documentation. Maybe someone else knows where it ran off to? Seems
>>> like something we should have on a our website (and a test that we
>>> should redo at some point).
>>>
>>> Cheers,
>>> Ethan
>>>
>>>
>
>
Re: Scalability was Re: What are pools? was Re: [VOTE] Approve the
release of apache-esme-1.3
Posted by Ethan Jewett <es...@gmail.com>.
Hi Vladimir,
I think that's correct, since ESME uses Mapper it will run on any
database that Mapper supports. I believe that this is dependent on
access to the database using a driver that provides a JDBC interface,
but I'm not sure. I think that all databases that Mapper supports are
relational, though I suppose you could try using a JDBC driver for a
non-relational database (like HBQL for hBase -
http://www.hbql.com/examples/jdbc.html).
Cheers,
Ethan
On Thu, Aug 4, 2011 at 1:05 AM, Vladimir Ivanov <le...@gmail.com> wrote:
> Ethan,
>
> Correct me if I'm wrong, but at the moment ESME should be able to run on any
> *relational* database due to all entities are stored/retreived via Lift
> Mapper persistence layer which implies traditional relational store. It's
> absolutly possible to use Lift Record persistence layer though which is
> intended for NoSQL databases, such as MongoDB.
>
> Vladimir
>
> 2011/8/3 Jack Park <ja...@topicquests.org>
>
>> Thanks, Ethan.
>> I'd like to know more about using MySQL.
>> Indeed, I'd like to know more about other classes of databases, e.g. graph
>> databases.
>>
>> Still, while hit rate is one aspect of scalability, when you talk about
>> "safely cached", I wonder about the combinatoric aspect of scalability, say,
>> a million users each with timelines in the hundreds to thousands long, and
>> all the entailed message objects. Are you thinking in terms of, say,
>> external memcached solutions, or just large memory models and load
>> balancers?
>>
>> Curious minds...
>> Thanks
>> Jack
>>
>>
>> On 8/3/2011 4:56 AM, Ethan Jewett wrote:
>>
>>> Hi Jack,
>>>
>>> It should be able to run on any Java application server with any
>>> database via JNDI or with any JDBC database by providing JDBC
>>> connection information in a .props file. I believe that on Stax (now
>>> CloudBees) we are using a MySQL database.
>>>
>>> Regarding scalability, the design of ESME is such that it doesn't rely
>>> much on the database for scalability. Operations should generally
>>> happen on data in memory, and there shouldn't be access of the
>>> database on most operations. We can do this because the data model has
>>> messages as immutable objects (you can't change or delete them), so
>>> they can be safely cached and we can rest assured that they will never
>>> be out of sync with the database.
>>>
>>> I'm fairly sure we did some performance tests a while ago that showed
>>> ESME was able to support thousands of concurrent users on a fairly
>>> small server without trouble, but I can't currently find that
>>> documentation. Maybe someone else knows where it ran off to? Seems
>>> like something we should have on a our website (and a test that we
>>> should redo at some point).
>>>
>>> Cheers,
>>> Ethan
>>>
>>>
>
>
> --
> Best Regards,
> Vladimir Ivanov
>
Re: Scalability was Re: What are pools? was Re: [VOTE] Approve the
release of apache-esme-1.3
Posted by Vladimir Ivanov <le...@gmail.com>.
Ethan,
Correct me if I'm wrong, but at the moment ESME should be able to run on any
*relational* database due to all entities are stored/retreived via Lift
Mapper persistence layer which implies traditional relational store. It's
absolutly possible to use Lift Record persistence layer though which is
intended for NoSQL databases, such as MongoDB.
Vladimir
2011/8/3 Jack Park <ja...@topicquests.org>
> Thanks, Ethan.
> I'd like to know more about using MySQL.
> Indeed, I'd like to know more about other classes of databases, e.g. graph
> databases.
>
> Still, while hit rate is one aspect of scalability, when you talk about
> "safely cached", I wonder about the combinatoric aspect of scalability, say,
> a million users each with timelines in the hundreds to thousands long, and
> all the entailed message objects. Are you thinking in terms of, say,
> external memcached solutions, or just large memory models and load
> balancers?
>
> Curious minds...
> Thanks
> Jack
>
>
> On 8/3/2011 4:56 AM, Ethan Jewett wrote:
>
>> Hi Jack,
>>
>> It should be able to run on any Java application server with any
>> database via JNDI or with any JDBC database by providing JDBC
>> connection information in a .props file. I believe that on Stax (now
>> CloudBees) we are using a MySQL database.
>>
>> Regarding scalability, the design of ESME is such that it doesn't rely
>> much on the database for scalability. Operations should generally
>> happen on data in memory, and there shouldn't be access of the
>> database on most operations. We can do this because the data model has
>> messages as immutable objects (you can't change or delete them), so
>> they can be safely cached and we can rest assured that they will never
>> be out of sync with the database.
>>
>> I'm fairly sure we did some performance tests a while ago that showed
>> ESME was able to support thousands of concurrent users on a fairly
>> small server without trouble, but I can't currently find that
>> documentation. Maybe someone else knows where it ran off to? Seems
>> like something we should have on a our website (and a test that we
>> should redo at some point).
>>
>> Cheers,
>> Ethan
>>
>>
--
Best Regards,
Vladimir Ivanov
Re: Scalability was Re: What are pools? was Re: [VOTE] Approve the
release of apache-esme-1.3
Posted by Jack Park <ja...@topicquests.org>.
Thanks, Ethan.
I'd like to know more about using MySQL.
Indeed, I'd like to know more about other classes of databases, e.g.
graph databases.
Still, while hit rate is one aspect of scalability, when you talk about
"safely cached", I wonder about the combinatoric aspect of scalability,
say, a million users each with timelines in the hundreds to thousands
long, and all the entailed message objects. Are you thinking in terms
of, say, external memcached solutions, or just large memory models and
load balancers?
Curious minds...
Thanks
Jack
On 8/3/2011 4:56 AM, Ethan Jewett wrote:
> Hi Jack,
>
> It should be able to run on any Java application server with any
> database via JNDI or with any JDBC database by providing JDBC
> connection information in a .props file. I believe that on Stax (now
> CloudBees) we are using a MySQL database.
>
> Regarding scalability, the design of ESME is such that it doesn't rely
> much on the database for scalability. Operations should generally
> happen on data in memory, and there shouldn't be access of the
> database on most operations. We can do this because the data model has
> messages as immutable objects (you can't change or delete them), so
> they can be safely cached and we can rest assured that they will never
> be out of sync with the database.
>
> I'm fairly sure we did some performance tests a while ago that showed
> ESME was able to support thousands of concurrent users on a fairly
> small server without trouble, but I can't currently find that
> documentation. Maybe someone else knows where it ran off to? Seems
> like something we should have on a our website (and a test that we
> should redo at some point).
>
> Cheers,
> Ethan
>