You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@wicket.apache.org by Eelco Hillenius <ee...@gmail.com> on 2007/12/15 07:44:23 UTC
Wicket Cluster as core projects?
Hi,
Matej developed some cluster code that is optimized for Wicket (it has
for instance a clustered page store) + Jetty and that uses Tomcat
tribes for cluster communication. I've tweaked (mainly separated it in
3 projects to make the dependencies work) and tested it with
Teachscape. It looks good enough to be a 1.0 release.
If people don't object, I'd like to make these projects part of our
core projects. I definitively want support this project, since we plan
to use it for Teachscape soon (on top of additional clustering of for
instance Lucene and JMX, for which we might use Terracotta). The
licenses are not a problem (Jetty and Tomcat Tribes mainly), and the
ASF headers and tests for it are all in place.
Clustering support is listed here:
http://cwiki.apache.org/confluence/display/WICKET/Wicket+1.4+Wish+List
Though that says: 'enabled by default', which I don't think we should
do. The default is fine, and you can cluster your Wicket app easy
enough (without changes to your code) by using this project and/ or
Terracotta (or tradional clustering if you are fine with using the
older session store). Enabling it by default would mean more
dependencies, which I don't think we should do. I'd also like to
stress that these projects don't affect other wicket projects in any
way.
If no-one objects to making this a core project, I'd also like to turn
it on for the example project. If there is just one node in the
cluster, it will still run without any problems, so it wouldn't
endanger anything or make deployment more difficult.
WDYT?
Eelco
Re: Wicket Cluster as core projects?
Posted by Matej Knopp <ma...@gmail.com>.
On Dec 15, 2007 11:20 AM, Johan Compagner <jc...@gmail.com> wrote:
> Do we get a call on all the nodes if you add an object in the session in 1 node?
>
>
> On 12/15/07, Igor Vaynberg <ig...@gmail.com> wrote:
> > i guess thats my point, why do we need a message-passing framework as
> > part of core? its outside the scope of a web app framework.
> >
> > i guess what i was hoping for, although i dont know if its possible,
> > is installing a httpsessionattributelistener or something similar that
> > can monitor the attributes in the session and then on the attribute
> > changed we would check if its a page from another node and save it to
> > disk. that way we piggy-back on whatever clustering mechanism the
> > servlet container uses - we dont need to have our own parallel
> > messaging framework to replicate the pages. like i said, dont know how
> > feasible that is, matej should know more.
That's kinda complicated as you need to have synchronized session
(same session id) on all nodes, so it's not likely you can do
something like this in a container agnostic way.
> >
> > -igor
> >
> >
> > On Dec 14, 2007 11:09 PM, Eelco Hillenius <ee...@gmail.com> wrote:
> > >
> > > On Dec 14, 2007 11:07 PM, Eelco Hillenius <ee...@gmail.com>
> > wrote:
> > > > On Dec 14, 2007 10:57 PM, <ig...@gmail.com> wrote:
> > > > > I don't know if these should go into core just yet. The cluster
> > > > > support I put on the wiki was going to work differently without any
> > > > > external libs. Basically each node would write the replicated current
> > > > > page of the other node to disk so each node has all page of all other
> > > > > nodes on disk. That's what I meant by transparent. Marek?
> > > >
> > > > That's not a configuration I would prefer, but can't we fold that into
> > > > the same project? Did you start from scratch? It would be unfortunate
> > > > if we only have wicket-stuff projects for this.
> > >
> > > Btw, page map replication is one optional part (project); the core
> > > mainly exists for generic message passing that is used for normal
> > > session clustering. That only imho is worth having a project for.
> > >
> > > Eelco
> > >
> >
>
Re: Wicket Cluster as core projects?
Posted by Johan Compagner <jc...@gmail.com>.
Do we get a call on all the nodes if you add an object in the session in 1 node?
On 12/15/07, Igor Vaynberg <ig...@gmail.com> wrote:
> i guess thats my point, why do we need a message-passing framework as
> part of core? its outside the scope of a web app framework.
>
> i guess what i was hoping for, although i dont know if its possible,
> is installing a httpsessionattributelistener or something similar that
> can monitor the attributes in the session and then on the attribute
> changed we would check if its a page from another node and save it to
> disk. that way we piggy-back on whatever clustering mechanism the
> servlet container uses - we dont need to have our own parallel
> messaging framework to replicate the pages. like i said, dont know how
> feasible that is, matej should know more.
>
> -igor
>
>
> On Dec 14, 2007 11:09 PM, Eelco Hillenius <ee...@gmail.com> wrote:
> >
> > On Dec 14, 2007 11:07 PM, Eelco Hillenius <ee...@gmail.com>
> wrote:
> > > On Dec 14, 2007 10:57 PM, <ig...@gmail.com> wrote:
> > > > I don't know if these should go into core just yet. The cluster
> > > > support I put on the wiki was going to work differently without any
> > > > external libs. Basically each node would write the replicated current
> > > > page of the other node to disk so each node has all page of all other
> > > > nodes on disk. That's what I meant by transparent. Marek?
> > >
> > > That's not a configuration I would prefer, but can't we fold that into
> > > the same project? Did you start from scratch? It would be unfortunate
> > > if we only have wicket-stuff projects for this.
> >
> > Btw, page map replication is one optional part (project); the core
> > mainly exists for generic message passing that is used for normal
> > session clustering. That only imho is worth having a project for.
> >
> > Eelco
> >
>
Re: Wicket Cluster as core projects?
Posted by Eelco Hillenius <ee...@gmail.com>.
On Dec 14, 2007 11:33 PM, Igor Vaynberg <ig...@gmail.com> wrote:
> i guess thats my point, why do we need a message-passing framework as
> part of core? its outside the scope of a web app framework.
>
> i guess what i was hoping for, although i dont know if its possible,
> is installing a httpsessionattributelistener or something similar that
> can monitor the attributes in the session and then on the attribute
> changed we would check if its a page from another node and save it to
> disk. that way we piggy-back on whatever clustering mechanism the
> servlet container uses - we dont need to have our own parallel
> messaging framework to replicate the pages. like i said, dont know how
> feasible that is, matej should know more.
But one thing that is nice about this code I have here is that it
automatically discovers node, synchronizes new ones, etc. That's
exactly the stuff that you don't want to write yourself.
But, ok, let's hear from Matej :-)
Eeco
Re: Wicket Cluster as core projects?
Posted by Igor Vaynberg <ig...@gmail.com>.
i guess thats my point, why do we need a message-passing framework as
part of core? its outside the scope of a web app framework.
i guess what i was hoping for, although i dont know if its possible,
is installing a httpsessionattributelistener or something similar that
can monitor the attributes in the session and then on the attribute
changed we would check if its a page from another node and save it to
disk. that way we piggy-back on whatever clustering mechanism the
servlet container uses - we dont need to have our own parallel
messaging framework to replicate the pages. like i said, dont know how
feasible that is, matej should know more.
-igor
On Dec 14, 2007 11:09 PM, Eelco Hillenius <ee...@gmail.com> wrote:
>
> On Dec 14, 2007 11:07 PM, Eelco Hillenius <ee...@gmail.com> wrote:
> > On Dec 14, 2007 10:57 PM, <ig...@gmail.com> wrote:
> > > I don't know if these should go into core just yet. The cluster
> > > support I put on the wiki was going to work differently without any
> > > external libs. Basically each node would write the replicated current
> > > page of the other node to disk so each node has all page of all other
> > > nodes on disk. That's what I meant by transparent. Marek?
> >
> > That's not a configuration I would prefer, but can't we fold that into
> > the same project? Did you start from scratch? It would be unfortunate
> > if we only have wicket-stuff projects for this.
>
> Btw, page map replication is one optional part (project); the core
> mainly exists for generic message passing that is used for normal
> session clustering. That only imho is worth having a project for.
>
> Eelco
>
Re: Wicket Cluster as core projects?
Posted by Eelco Hillenius <ee...@gmail.com>.
A final remark: I have this ready now, and it works. I only have to
submit it, and it builds with maven as part of our projects, and
people can immediately start playing with it. A new initiative would
probably take some time. I can live with these projects just being
wicket-stuff projects. I just feel this would be a shame, because
Matej did a pretty good job on this (and thanks Teachscape for letting
him work on it for a couple of weeks). If you (Igor) feel that this
project isn't good or you'll do ten times better, that's ok, but
you'll probably have to live with this being a wicket-stuff project as
well. What I'm saying is that the ideal situation would be to fold
both ideas into one project (at least building on one core project),
and commit what I have now so that it can be built upon iteratively.
Cheers,
Eelco
Re: Wicket Cluster as core projects?
Posted by Eelco Hillenius <ee...@gmail.com>.
On Dec 14, 2007 11:07 PM, Eelco Hillenius <ee...@gmail.com> wrote:
> On Dec 14, 2007 10:57 PM, <ig...@gmail.com> wrote:
> > I don't know if these should go into core just yet. The cluster
> > support I put on the wiki was going to work differently without any
> > external libs. Basically each node would write the replicated current
> > page of the other node to disk so each node has all page of all other
> > nodes on disk. That's what I meant by transparent. Marek?
>
> That's not a configuration I would prefer, but can't we fold that into
> the same project? Did you start from scratch? It would be unfortunate
> if we only have wicket-stuff projects for this.
Btw, page map replication is one optional part (project); the core
mainly exists for generic message passing that is used for normal
session clustering. That only imho is worth having a project for.
Eelco
Re: Wicket Cluster as core projects?
Posted by Eelco Hillenius <ee...@gmail.com>.
On Dec 14, 2007 10:57 PM, <ig...@gmail.com> wrote:
> I don't know if these should go into core just yet. The cluster
> support I put on the wiki was going to work differently without any
> external libs. Basically each node would write the replicated current
> page of the other node to disk so each node has all page of all other
> nodes on disk. That's what I meant by transparent. Marek?
That's not a configuration I would prefer, but can't we fold that into
the same project? Did you start from scratch? It would be unfortunate
if we only have wicket-stuff projects for this.
Eelco
Re: Wicket Cluster as core projects?
Posted by ig...@gmail.com.
s/marek/matej/ stupid iPhone!
-igor
On 12/14/07, igor.vaynberg@gmail.com <ig...@gmail.com> wrote:
> I don't know if these should go into core just yet. The cluster
> support I put on the wiki was going to work differently without any
> external libs. Basically each node would write the replicated current
> page of the other node to disk so each node has all page of all other
> nodes on disk. That's what I meant by transparent. Marek?
>
> -igor
>
> On 12/14/07, Eelco Hillenius <ee...@gmail.com> wrote:
> > Hi,
> >
> > Matej developed some cluster code that is optimized for Wicket (it has
> > for instance a clustered page store) + Jetty and that uses Tomcat
> > tribes for cluster communication. I've tweaked (mainly separated it in
> > 3 projects to make the dependencies work) and tested it with
> > Teachscape. It looks good enough to be a 1.0 release.
> >
> > If people don't object, I'd like to make these projects part of our
> > core projects. I definitively want support this project, since we plan
> > to use it for Teachscape soon (on top of additional clustering of for
> > instance Lucene and JMX, for which we might use Terracotta). The
> > licenses are not a problem (Jetty and Tomcat Tribes mainly), and the
> > ASF headers and tests for it are all in place.
> >
> > Clustering support is listed here:
> > http://cwiki.apache.org/confluence/display/WICKET/Wicket+1.4+Wish+List
> >
> > Though that says: 'enabled by default', which I don't think we should
> > do. The default is fine, and you can cluster your Wicket app easy
> > enough (without changes to your code) by using this project and/ or
> > Terracotta (or tradional clustering if you are fine with using the
> > older session store). Enabling it by default would mean more
> > dependencies, which I don't think we should do. I'd also like to
> > stress that these projects don't affect other wicket projects in any
> > way.
> >
> > If no-one objects to making this a core project, I'd also like to turn
> > it on for the example project. If there is just one node in the
> > cluster, it will still run without any problems, so it wouldn't
> > endanger anything or make deployment more difficult.
> >
> > WDYT?
> >
> > Eelco
> >
>
Re: Wicket Cluster as core projects?
Posted by ig...@gmail.com.
I don't know if these should go into core just yet. The cluster
support I put on the wiki was going to work differently without any
external libs. Basically each node would write the replicated current
page of the other node to disk so each node has all page of all other
nodes on disk. That's what I meant by transparent. Marek?
-igor
On 12/14/07, Eelco Hillenius <ee...@gmail.com> wrote:
> Hi,
>
> Matej developed some cluster code that is optimized for Wicket (it has
> for instance a clustered page store) + Jetty and that uses Tomcat
> tribes for cluster communication. I've tweaked (mainly separated it in
> 3 projects to make the dependencies work) and tested it with
> Teachscape. It looks good enough to be a 1.0 release.
>
> If people don't object, I'd like to make these projects part of our
> core projects. I definitively want support this project, since we plan
> to use it for Teachscape soon (on top of additional clustering of for
> instance Lucene and JMX, for which we might use Terracotta). The
> licenses are not a problem (Jetty and Tomcat Tribes mainly), and the
> ASF headers and tests for it are all in place.
>
> Clustering support is listed here:
> http://cwiki.apache.org/confluence/display/WICKET/Wicket+1.4+Wish+List
>
> Though that says: 'enabled by default', which I don't think we should
> do. The default is fine, and you can cluster your Wicket app easy
> enough (without changes to your code) by using this project and/ or
> Terracotta (or tradional clustering if you are fine with using the
> older session store). Enabling it by default would mean more
> dependencies, which I don't think we should do. I'd also like to
> stress that these projects don't affect other wicket projects in any
> way.
>
> If no-one objects to making this a core project, I'd also like to turn
> it on for the example project. If there is just one node in the
> cluster, it will still run without any problems, so it wouldn't
> endanger anything or make deployment more difficult.
>
> WDYT?
>
> Eelco
>
Re: Wicket Cluster as core projects?
Posted by "C. Bergström" <cb...@netsyncro.com>.
On Sat, 2007-12-15 at 10:43 +0100, Martijn Dashorst wrote:
> AFAIK, memcached java library support is GPL software so that is a no-go.
Looks to be MIT and there's a successor or they just moved repos..
http://bleu.west.spy.net/~dustin/projects/memcached/
hg clone http://hg.west.spy.net/hg/memcached/
I like Eelco's idea for sure.. I'd be interested to see it in action..
How many servers has it been tested on.. Any test results which show how
many tps (transactions per second) per node across the cluster... etc
etc.. What if any are the caveats...
./C
Re: Wicket Cluster as core projects?
Posted by Martijn Dashorst <ma...@gmail.com>.
AFAIK, memcached java library support is GPL software so that is a no-go.
Martijn
On Dec 15, 2007 10:13 AM, C. Bergström <cb...@netsyncro.com> wrote:
>
> On Fri, 2007-12-14 at 22:44 -0800, Eelco Hillenius wrote:
> > Hi,
> >
> > Matej developed some cluster code that is optimized for Wicket (it has
> > for instance a clustered page store) + Jetty and that uses Tomcat
> > tribes for cluster communication. I've tweaked (mainly separated it in
> > 3 projects to make the dependencies work) and tested it with
> > Teachscape. It looks good enough to be a 1.0 release.
> >
> > If people don't object, I'd like to make these projects part of our
> > core projects. I definitively want support this project, since we plan
> > to use it for Teachscape soon (on top of additional clustering of for
> > instance Lucene and JMX, for which we might use Terracotta). The
> > licenses are not a problem (Jetty and Tomcat Tribes mainly), and the
> > ASF headers and tests for it are all in place.
> >
>
> It sounds very interesting and while I like the idea..
>
> 1) Terracotta license is a very liberal badgeware
> 2) Why not work memcached in this somehow..
> a. It has built-in HA/failover/distribution
> b. It's blazing fast and efficient at caching things
> c. Has an active and growing community
> d. Already in production for sites like facebook, myspace,
> or hyves
> for the dutch on the list..
>
>
> I haven't worked with the java client connector, but seems to be
> maintained and a small example can be found here. Currently builds with
> Ant the best I can tell and probably not too difficult to mavenize.
> http://www.whalin.com/memcached/HOWTO.txt
>
> Anywho.. great work Eelco/Igor/Mataj and just trying to add another
> perspective..
>
>
--
Buy Wicket in Action: http://manning.com/dashorst
Apache Wicket 1.3.0-rc1 is released
Get it now: http://www.apache.org/dyn/closer.cgi/wicket/1.3.0-rc1/
Re: Wicket Cluster as core projects?
Posted by Eelco Hillenius <ee...@gmail.com>.
On Dec 16, 2007 10:27 PM, Igor Vaynberg <ig...@gmail.com> wrote:
> hm, looks like you removed them in rev 3383
Duh! Back again now.
Eelco
Re: Wicket Cluster as core projects?
Posted by Igor Vaynberg <ig...@gmail.com>.
hm, looks like you removed them in rev 3383
-igor
On Dec 16, 2007 10:26 PM, Igor Vaynberg <ig...@gmail.com> wrote:
> you sure you checked it in? url is no good for me...
>
> -igor
>
>
>
> On Dec 16, 2007 10:05 PM, Eelco Hillenius <ee...@gmail.com> wrote:
> > > Ok, that's fine. I'm gonna keep it in Teachscape. Maybe when I have
> > > spare time I'll put it in wicket-stuff, but right now that isn't worth
> > > the extra hassle for me (I already spent half of today on this). The
> > > nbr1 priority is to have this working well for Teachscape, so that's
> > > just what I'll focus on.
> >
> > I made some time, and the three projects are now in
> > http://wicket-stuff.svn.sourceforge.net/svnroot/wicket-stuff/trunk/wicket-cluster.
> > You need to check out that project to build the three sub projects
> > (like Wicket). They follow the same versioning as Wicket. They don't
> > build on Bamboo yet, but I'm planning to do that (or if someone else
> > would like to set that up, that'd be great).
> >
> > Eelco
> >
>
Re: Wicket Cluster as core projects?
Posted by Igor Vaynberg <ig...@gmail.com>.
you sure you checked it in? url is no good for me...
-igor
On Dec 16, 2007 10:05 PM, Eelco Hillenius <ee...@gmail.com> wrote:
> > Ok, that's fine. I'm gonna keep it in Teachscape. Maybe when I have
> > spare time I'll put it in wicket-stuff, but right now that isn't worth
> > the extra hassle for me (I already spent half of today on this). The
> > nbr1 priority is to have this working well for Teachscape, so that's
> > just what I'll focus on.
>
> I made some time, and the three projects are now in
> http://wicket-stuff.svn.sourceforge.net/svnroot/wicket-stuff/trunk/wicket-cluster.
> You need to check out that project to build the three sub projects
> (like Wicket). They follow the same versioning as Wicket. They don't
> build on Bamboo yet, but I'm planning to do that (or if someone else
> would like to set that up, that'd be great).
>
> Eelco
>
Re: Wicket Cluster as core projects?
Posted by Eelco Hillenius <ee...@gmail.com>.
> Ok, that's fine. I'm gonna keep it in Teachscape. Maybe when I have
> spare time I'll put it in wicket-stuff, but right now that isn't worth
> the extra hassle for me (I already spent half of today on this). The
> nbr1 priority is to have this working well for Teachscape, so that's
> just what I'll focus on.
I made some time, and the three projects are now in
http://wicket-stuff.svn.sourceforge.net/svnroot/wicket-stuff/trunk/wicket-cluster.
You need to check out that project to build the three sub projects
(like Wicket). They follow the same versioning as Wicket. They don't
build on Bamboo yet, but I'm planning to do that (or if someone else
would like to set that up, that'd be great).
Eelco
Re: Wicket Cluster as core projects?
Posted by Eelco Hillenius <ee...@gmail.com>.
On Dec 15, 2007 1:42 AM, Matej Knopp <ma...@gmail.com> wrote:
> Err, I haven't went through the whole thread, but I don't think the
> wicket-jetty-cluster is a right thing to put in core.
> The pagestore can/will be clustered without any dependencies on
> servlet container. And the wicket-jetty-cluster is mostly a session
> replication for jetty, there is minimal amount of wicket related
> stuff.
Well, it is optimized for Wicket, even if built on a wicket (and
server) independent layer. You weren't planning on making this a full
fledged independent project anytime soon?
> So I'm not really eager to put this into core.
Ok, that's fine. I'm gonna keep it in Teachscape. Maybe when I have
spare time I'll put it in wicket-stuff, but right now that isn't worth
the extra hassle for me (I already spent half of today on this). The
nbr1 priority is to have this working well for Teachscape, so that's
just what I'll focus on.
Eelco
Re: Wicket Cluster as core projects?
Posted by Matej Knopp <ma...@gmail.com>.
Err, I haven't went through the whole thread, but I don't think the
wicket-jetty-cluster is a right thing to put in core.
The pagestore can/will be clustered without any dependencies on
servlet container. And the wicket-jetty-cluster is mostly a session
replication for jetty, there is minimal amount of wicket related
stuff.
So I'm not really eager to put this into core.
-Matej
On Dec 15, 2007 10:13 AM, C. Bergström <cb...@netsyncro.com> wrote:
>
> On Fri, 2007-12-14 at 22:44 -0800, Eelco Hillenius wrote:
> > Hi,
> >
> > Matej developed some cluster code that is optimized for Wicket (it has
> > for instance a clustered page store) + Jetty and that uses Tomcat
> > tribes for cluster communication. I've tweaked (mainly separated it in
> > 3 projects to make the dependencies work) and tested it with
> > Teachscape. It looks good enough to be a 1.0 release.
> >
> > If people don't object, I'd like to make these projects part of our
> > core projects. I definitively want support this project, since we plan
> > to use it for Teachscape soon (on top of additional clustering of for
> > instance Lucene and JMX, for which we might use Terracotta). The
> > licenses are not a problem (Jetty and Tomcat Tribes mainly), and the
> > ASF headers and tests for it are all in place.
> >
>
> It sounds very interesting and while I like the idea..
>
> 1) Terracotta license is a very liberal badgeware
> 2) Why not work memcached in this somehow..
> a. It has built-in HA/failover/distribution
> b. It's blazing fast and efficient at caching things
> c. Has an active and growing community
> d. Already in production for sites like facebook, myspace, or hyves
> for the dutch on the list..
>
>
> I haven't worked with the java client connector, but seems to be
> maintained and a small example can be found here. Currently builds with
> Ant the best I can tell and probably not too difficult to mavenize.
> http://www.whalin.com/memcached/HOWTO.txt
>
> Anywho.. great work Eelco/Igor/Mataj and just trying to add another
> perspective..
>
>
Re: Wicket Cluster as core projects?
Posted by "C. Bergström" <cb...@netsyncro.com>.
On Fri, 2007-12-14 at 22:44 -0800, Eelco Hillenius wrote:
> Hi,
>
> Matej developed some cluster code that is optimized for Wicket (it has
> for instance a clustered page store) + Jetty and that uses Tomcat
> tribes for cluster communication. I've tweaked (mainly separated it in
> 3 projects to make the dependencies work) and tested it with
> Teachscape. It looks good enough to be a 1.0 release.
>
> If people don't object, I'd like to make these projects part of our
> core projects. I definitively want support this project, since we plan
> to use it for Teachscape soon (on top of additional clustering of for
> instance Lucene and JMX, for which we might use Terracotta). The
> licenses are not a problem (Jetty and Tomcat Tribes mainly), and the
> ASF headers and tests for it are all in place.
>
It sounds very interesting and while I like the idea..
1) Terracotta license is a very liberal badgeware
2) Why not work memcached in this somehow..
a. It has built-in HA/failover/distribution
b. It's blazing fast and efficient at caching things
c. Has an active and growing community
d. Already in production for sites like facebook, myspace, or hyves
for the dutch on the list..
I haven't worked with the java client connector, but seems to be
maintained and a small example can be found here. Currently builds with
Ant the best I can tell and probably not too difficult to mavenize.
http://www.whalin.com/memcached/HOWTO.txt
Anywho.. great work Eelco/Igor/Mataj and just trying to add another
perspective..
Re: Wicket Cluster as core projects?
Posted by Martijn Dashorst <ma...@gmail.com>.
On Dec 15, 2007 7:44 AM, Eelco Hillenius <ee...@gmail.com> wrote:
> WDYT?
>
Only after 1.3 has been made final. though the code is 1.0 worthy,
additional bugs will be discovered and it will drain our already limited
resources for 1.3.
We also need to make sure no external contribs have been done to the code
base, i.e. supply a svn log on the project before we commit it into our code
base.
Martijn
--
Buy Wicket in Action: http://manning.com/dashorst
Apache Wicket 1.3.0-rc1 is released
Get it now: http://www.apache.org/dyn/closer.cgi/wicket/1.3.0-rc1/