You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by Tommaso Teofili <to...@gmail.com> on 2013/10/31 15:41:15 UTC
[PROPOSAL] Sling Replication module
Hi all,
In the last weeks I've been working on an Apache Sling based replication
module and I'd like to donate it to the project so I'm wondering if that
may sound interesting for the community.
Its main purpose is to "replicate" data between Sling instances.
Some design concepts :
- Replication agents are the responsible units for the replication of data
to a certain endpoint
- Each replication agent is an OSGi service
- Configuration of agents happens via OSGi configuration (e.g. by adding
nodes of type sling:OsgiConfig with certain properties as JSoN files)
- Configuration of agents include: endpoint, instance authentication,
transport implementation, binary format for data to be exchanged
- Each agent manages one or more queues of data packages to be replicated
Main use case is master / slave architectures where data gets created on
the master and need to be replicated on the slaves which are publicly
available, or viceversa where data gets created by the user and needs to be
consistently replicated in the other instances of the architecture.
The current implementation is still a work in progress so don't expect it
to be full feature but maybe it's worth having it among the extensions or
either in a sandbox space.
I can create a Jira issue and add the code as a patch / zip file if there's
interest.
Looking forward to hear what you think,
Regards,
Tommaso
Re: [PROPOSAL] Sling Replication module
Posted by Carsten Ziegeler <cz...@apache.org>.
Hi Tommaso,
I think this sounds fantastic - I suggest you create this Jira issue and
attach the code, so everyone can have a look at it.
Regards
Carsten
2013/10/31 Tommaso Teofili <to...@gmail.com>
> Hi all,
>
> In the last weeks I've been working on an Apache Sling based replication
> module and I'd like to donate it to the project so I'm wondering if that
> may sound interesting for the community.
>
> Its main purpose is to "replicate" data between Sling instances.
>
> Some design concepts :
> - Replication agents are the responsible units for the replication of data
> to a certain endpoint
> - Each replication agent is an OSGi service
> - Configuration of agents happens via OSGi configuration (e.g. by adding
> nodes of type sling:OsgiConfig with certain properties as JSoN files)
> - Configuration of agents include: endpoint, instance authentication,
> transport implementation, binary format for data to be exchanged
> - Each agent manages one or more queues of data packages to be replicated
>
> Main use case is master / slave architectures where data gets created on
> the master and need to be replicated on the slaves which are publicly
> available, or viceversa where data gets created by the user and needs to be
> consistently replicated in the other instances of the architecture.
>
> The current implementation is still a work in progress so don't expect it
> to be full feature but maybe it's worth having it among the extensions or
> either in a sandbox space.
>
> I can create a Jira issue and add the code as a patch / zip file if there's
> interest.
> Looking forward to hear what you think,
> Regards,
> Tommaso
>
--
Carsten Ziegeler
cziegeler@apache.org
Re: [PROPOSAL] Sling Replication module
Posted by Tobias Bocanegra <tr...@day.com>.
Thanks Tommaso.
I browsed through the patch and added some comments to SLING-3223.
Regards, Toby
On Tue, Nov 5, 2013 at 8:25 AM, Tommaso Teofili
<to...@gmail.com> wrote:
> I've attached the code as a patch to SLING-3223 [1] so that it can be
> reviewed.
>
> Given your review is positive, I think the next steps would be:
> 1. discuss / vote for acceptance
> 2. IP Clearance stuff (SGA + vote on general@incubator.a.o)
> 3. commit to trunk
>
> Thanks and regards,
> Tommaso
>
> [1] : https://issues.apache.org/jira/browse/SLING-3223
>
>
> 2013/11/4 Tommaso Teofili <to...@gmail.com>
>
>> Hi all,
>>
>> thanks for the positive feedback.
>> Answers to your questions below:
>>
>> > will this module work only between Sling instances, or will I be able
>> to make it work with external systems? I'm thinking of integration with
>> upstream proxy caches, like Varnish or Apache Traffic Server
>>
>> the current implementation is working on top of Sling, you may make it
>> work with external systems by customizing it (e.g. sending custom HTTP
>> requests to the cache to flush it) however I agree with Felix wrt using
>> replication for cache invalidation.
>>
>> > Is this thing similar to Apache Cellar from Karaf ?
>>
>> I may be wrong but it seems to me something like what Cellar provides can
>> be implemented with a combination of the discovery feature plus replication.
>>
>> I'll open a Jira issue and attach the code between today and tomorrow.
>> Regards,
>> Tommaso
>>
>>
>>
>> 2013/11/3 Felix Meschberger <fm...@adobe.com>
>>
>>> Hi
>>>
>>>
>>> Am 03.11.2013 um 02:54 schrieb Dragos Dascalita Haut <ddragosd@gmail.com
>>> >:
>>>
>>> > Interesting work Tommaso. Is this thing similar to Apache Cellar from
>>> Karaf ?
>>>
>>> Interesting pointer. Somehow I have the impression Cellar is more
>>> comparable to the Sling discovery feature ..
>>>
>>> Regards
>>> Felix
>>>
>>> >
>>> > Regards,
>>> > dragos dascalita haut
>>> >
>>> > Sent from my iPad
>>> >
>>> >> On Nov 2, 2013, at 2:37 PM, Robert Munteanu <ro...@apache.org>
>>> wrote:
>>> >>
>>> >> Hi Tommaso,
>>> >>
>>> >> On Thu, Oct 31, 2013 at 4:41 PM, Tommaso Teofili
>>> >> <to...@gmail.com> wrote:
>>> >>> Hi all,
>>> >>>
>>> >>> In the last weeks I've been working on an Apache Sling based
>>> replication
>>> >>> module and I'd like to donate it to the project so I'm wondering if
>>> that
>>> >>> may sound interesting for the community.
>>> >>>
>>> >>> Its main purpose is to "replicate" data between Sling instances.
>>> >>
>>> >> It sounds interesting indeed!
>>> >>
>>> >> A quick question - will this module work only between Sling instances,
>>> >> or will I be able to make it work with external systems? I'm thinking
>>> >> of integration with upstream proxy caches, like Varnish or Apache
>>> >> Traffic Server.
>>> >>
>>> >> Robert
>>> >>
>>> >>>
>>> >>> Some design concepts :
>>> >>> - Replication agents are the responsible units for the replication of
>>> data
>>> >>> to a certain endpoint
>>> >>> - Each replication agent is an OSGi service
>>> >>> - Configuration of agents happens via OSGi configuration (e.g. by
>>> adding
>>> >>> nodes of type sling:OsgiConfig with certain properties as JSoN files)
>>> >>> - Configuration of agents include: endpoint, instance authentication,
>>> >>> transport implementation, binary format for data to be exchanged
>>> >>> - Each agent manages one or more queues of data packages to be
>>> replicated
>>> >>>
>>> >>> Main use case is master / slave architectures where data gets created
>>> on
>>> >>> the master and need to be replicated on the slaves which are publicly
>>> >>> available, or viceversa where data gets created by the user and needs
>>> to be
>>> >>> consistently replicated in the other instances of the architecture.
>>> >>>
>>> >>> The current implementation is still a work in progress so don't
>>> expect it
>>> >>> to be full feature but maybe it's worth having it among the
>>> extensions or
>>> >>> either in a sandbox space.
>>> >>>
>>> >>> I can create a Jira issue and add the code as a patch / zip file if
>>> there's
>>> >>> interest.
>>> >>> Looking forward to hear what you think,
>>> >>> Regards,
>>> >>> Tommaso
>>>
>>>
>>
Re: [PROPOSAL] Sling Replication module
Posted by Tommaso Teofili <to...@gmail.com>.
I've attached the code as a patch to SLING-3223 [1] so that it can be
reviewed.
Given your review is positive, I think the next steps would be:
1. discuss / vote for acceptance
2. IP Clearance stuff (SGA + vote on general@incubator.a.o)
3. commit to trunk
Thanks and regards,
Tommaso
[1] : https://issues.apache.org/jira/browse/SLING-3223
2013/11/4 Tommaso Teofili <to...@gmail.com>
> Hi all,
>
> thanks for the positive feedback.
> Answers to your questions below:
>
> > will this module work only between Sling instances, or will I be able
> to make it work with external systems? I'm thinking of integration with
> upstream proxy caches, like Varnish or Apache Traffic Server
>
> the current implementation is working on top of Sling, you may make it
> work with external systems by customizing it (e.g. sending custom HTTP
> requests to the cache to flush it) however I agree with Felix wrt using
> replication for cache invalidation.
>
> > Is this thing similar to Apache Cellar from Karaf ?
>
> I may be wrong but it seems to me something like what Cellar provides can
> be implemented with a combination of the discovery feature plus replication.
>
> I'll open a Jira issue and attach the code between today and tomorrow.
> Regards,
> Tommaso
>
>
>
> 2013/11/3 Felix Meschberger <fm...@adobe.com>
>
>> Hi
>>
>>
>> Am 03.11.2013 um 02:54 schrieb Dragos Dascalita Haut <ddragosd@gmail.com
>> >:
>>
>> > Interesting work Tommaso. Is this thing similar to Apache Cellar from
>> Karaf ?
>>
>> Interesting pointer. Somehow I have the impression Cellar is more
>> comparable to the Sling discovery feature ..
>>
>> Regards
>> Felix
>>
>> >
>> > Regards,
>> > dragos dascalita haut
>> >
>> > Sent from my iPad
>> >
>> >> On Nov 2, 2013, at 2:37 PM, Robert Munteanu <ro...@apache.org>
>> wrote:
>> >>
>> >> Hi Tommaso,
>> >>
>> >> On Thu, Oct 31, 2013 at 4:41 PM, Tommaso Teofili
>> >> <to...@gmail.com> wrote:
>> >>> Hi all,
>> >>>
>> >>> In the last weeks I've been working on an Apache Sling based
>> replication
>> >>> module and I'd like to donate it to the project so I'm wondering if
>> that
>> >>> may sound interesting for the community.
>> >>>
>> >>> Its main purpose is to "replicate" data between Sling instances.
>> >>
>> >> It sounds interesting indeed!
>> >>
>> >> A quick question - will this module work only between Sling instances,
>> >> or will I be able to make it work with external systems? I'm thinking
>> >> of integration with upstream proxy caches, like Varnish or Apache
>> >> Traffic Server.
>> >>
>> >> Robert
>> >>
>> >>>
>> >>> Some design concepts :
>> >>> - Replication agents are the responsible units for the replication of
>> data
>> >>> to a certain endpoint
>> >>> - Each replication agent is an OSGi service
>> >>> - Configuration of agents happens via OSGi configuration (e.g. by
>> adding
>> >>> nodes of type sling:OsgiConfig with certain properties as JSoN files)
>> >>> - Configuration of agents include: endpoint, instance authentication,
>> >>> transport implementation, binary format for data to be exchanged
>> >>> - Each agent manages one or more queues of data packages to be
>> replicated
>> >>>
>> >>> Main use case is master / slave architectures where data gets created
>> on
>> >>> the master and need to be replicated on the slaves which are publicly
>> >>> available, or viceversa where data gets created by the user and needs
>> to be
>> >>> consistently replicated in the other instances of the architecture.
>> >>>
>> >>> The current implementation is still a work in progress so don't
>> expect it
>> >>> to be full feature but maybe it's worth having it among the
>> extensions or
>> >>> either in a sandbox space.
>> >>>
>> >>> I can create a Jira issue and add the code as a patch / zip file if
>> there's
>> >>> interest.
>> >>> Looking forward to hear what you think,
>> >>> Regards,
>> >>> Tommaso
>>
>>
>
Re: [PROPOSAL] Sling Replication module
Posted by Tommaso Teofili <to...@gmail.com>.
Hi all,
thanks for the positive feedback.
Answers to your questions below:
> will this module work only between Sling instances, or will I be able to
make it work with external systems? I'm thinking of integration with
upstream proxy caches, like Varnish or Apache Traffic Server
the current implementation is working on top of Sling, you may make it work
with external systems by customizing it (e.g. sending custom HTTP requests
to the cache to flush it) however I agree with Felix wrt using replication
for cache invalidation.
> Is this thing similar to Apache Cellar from Karaf ?
I may be wrong but it seems to me something like what Cellar provides can
be implemented with a combination of the discovery feature plus replication.
I'll open a Jira issue and attach the code between today and tomorrow.
Regards,
Tommaso
2013/11/3 Felix Meschberger <fm...@adobe.com>
> Hi
>
>
> Am 03.11.2013 um 02:54 schrieb Dragos Dascalita Haut <dd...@gmail.com>:
>
> > Interesting work Tommaso. Is this thing similar to Apache Cellar from
> Karaf ?
>
> Interesting pointer. Somehow I have the impression Cellar is more
> comparable to the Sling discovery feature ..
>
> Regards
> Felix
>
> >
> > Regards,
> > dragos dascalita haut
> >
> > Sent from my iPad
> >
> >> On Nov 2, 2013, at 2:37 PM, Robert Munteanu <ro...@apache.org> wrote:
> >>
> >> Hi Tommaso,
> >>
> >> On Thu, Oct 31, 2013 at 4:41 PM, Tommaso Teofili
> >> <to...@gmail.com> wrote:
> >>> Hi all,
> >>>
> >>> In the last weeks I've been working on an Apache Sling based
> replication
> >>> module and I'd like to donate it to the project so I'm wondering if
> that
> >>> may sound interesting for the community.
> >>>
> >>> Its main purpose is to "replicate" data between Sling instances.
> >>
> >> It sounds interesting indeed!
> >>
> >> A quick question - will this module work only between Sling instances,
> >> or will I be able to make it work with external systems? I'm thinking
> >> of integration with upstream proxy caches, like Varnish or Apache
> >> Traffic Server.
> >>
> >> Robert
> >>
> >>>
> >>> Some design concepts :
> >>> - Replication agents are the responsible units for the replication of
> data
> >>> to a certain endpoint
> >>> - Each replication agent is an OSGi service
> >>> - Configuration of agents happens via OSGi configuration (e.g. by
> adding
> >>> nodes of type sling:OsgiConfig with certain properties as JSoN files)
> >>> - Configuration of agents include: endpoint, instance authentication,
> >>> transport implementation, binary format for data to be exchanged
> >>> - Each agent manages one or more queues of data packages to be
> replicated
> >>>
> >>> Main use case is master / slave architectures where data gets created
> on
> >>> the master and need to be replicated on the slaves which are publicly
> >>> available, or viceversa where data gets created by the user and needs
> to be
> >>> consistently replicated in the other instances of the architecture.
> >>>
> >>> The current implementation is still a work in progress so don't expect
> it
> >>> to be full feature but maybe it's worth having it among the extensions
> or
> >>> either in a sandbox space.
> >>>
> >>> I can create a Jira issue and add the code as a patch / zip file if
> there's
> >>> interest.
> >>> Looking forward to hear what you think,
> >>> Regards,
> >>> Tommaso
>
>
Re: [PROPOSAL] Sling Replication module
Posted by Felix Meschberger <fm...@adobe.com>.
Hi
Am 03.11.2013 um 02:54 schrieb Dragos Dascalita Haut <dd...@gmail.com>:
> Interesting work Tommaso. Is this thing similar to Apache Cellar from Karaf ?
Interesting pointer. Somehow I have the impression Cellar is more comparable to the Sling discovery feature ..
Regards
Felix
>
> Regards,
> dragos dascalita haut
>
> Sent from my iPad
>
>> On Nov 2, 2013, at 2:37 PM, Robert Munteanu <ro...@apache.org> wrote:
>>
>> Hi Tommaso,
>>
>> On Thu, Oct 31, 2013 at 4:41 PM, Tommaso Teofili
>> <to...@gmail.com> wrote:
>>> Hi all,
>>>
>>> In the last weeks I've been working on an Apache Sling based replication
>>> module and I'd like to donate it to the project so I'm wondering if that
>>> may sound interesting for the community.
>>>
>>> Its main purpose is to "replicate" data between Sling instances.
>>
>> It sounds interesting indeed!
>>
>> A quick question - will this module work only between Sling instances,
>> or will I be able to make it work with external systems? I'm thinking
>> of integration with upstream proxy caches, like Varnish or Apache
>> Traffic Server.
>>
>> Robert
>>
>>>
>>> Some design concepts :
>>> - Replication agents are the responsible units for the replication of data
>>> to a certain endpoint
>>> - Each replication agent is an OSGi service
>>> - Configuration of agents happens via OSGi configuration (e.g. by adding
>>> nodes of type sling:OsgiConfig with certain properties as JSoN files)
>>> - Configuration of agents include: endpoint, instance authentication,
>>> transport implementation, binary format for data to be exchanged
>>> - Each agent manages one or more queues of data packages to be replicated
>>>
>>> Main use case is master / slave architectures where data gets created on
>>> the master and need to be replicated on the slaves which are publicly
>>> available, or viceversa where data gets created by the user and needs to be
>>> consistently replicated in the other instances of the architecture.
>>>
>>> The current implementation is still a work in progress so don't expect it
>>> to be full feature but maybe it's worth having it among the extensions or
>>> either in a sandbox space.
>>>
>>> I can create a Jira issue and add the code as a patch / zip file if there's
>>> interest.
>>> Looking forward to hear what you think,
>>> Regards,
>>> Tommaso
Re: [PROPOSAL] Sling Replication module
Posted by Dragos Dascalita Haut <dd...@gmail.com>.
Interesting work Tommaso. Is this thing similar to Apache Cellar from Karaf ?
Regards,
dragos dascalita haut
Sent from my iPad
> On Nov 2, 2013, at 2:37 PM, Robert Munteanu <ro...@apache.org> wrote:
>
> Hi Tommaso,
>
> On Thu, Oct 31, 2013 at 4:41 PM, Tommaso Teofili
> <to...@gmail.com> wrote:
>> Hi all,
>>
>> In the last weeks I've been working on an Apache Sling based replication
>> module and I'd like to donate it to the project so I'm wondering if that
>> may sound interesting for the community.
>>
>> Its main purpose is to "replicate" data between Sling instances.
>
> It sounds interesting indeed!
>
> A quick question - will this module work only between Sling instances,
> or will I be able to make it work with external systems? I'm thinking
> of integration with upstream proxy caches, like Varnish or Apache
> Traffic Server.
>
> Robert
>
>>
>> Some design concepts :
>> - Replication agents are the responsible units for the replication of data
>> to a certain endpoint
>> - Each replication agent is an OSGi service
>> - Configuration of agents happens via OSGi configuration (e.g. by adding
>> nodes of type sling:OsgiConfig with certain properties as JSoN files)
>> - Configuration of agents include: endpoint, instance authentication,
>> transport implementation, binary format for data to be exchanged
>> - Each agent manages one or more queues of data packages to be replicated
>>
>> Main use case is master / slave architectures where data gets created on
>> the master and need to be replicated on the slaves which are publicly
>> available, or viceversa where data gets created by the user and needs to be
>> consistently replicated in the other instances of the architecture.
>>
>> The current implementation is still a work in progress so don't expect it
>> to be full feature but maybe it's worth having it among the extensions or
>> either in a sandbox space.
>>
>> I can create a Jira issue and add the code as a patch / zip file if there's
>> interest.
>> Looking forward to hear what you think,
>> Regards,
>> Tommaso
Re: [PROPOSAL] Sling Replication module
Posted by Felix Meschberger <fm...@adobe.com>.
Hi
Am 02.11.2013 um 22:37 schrieb Robert Munteanu <ro...@apache.org>:
> Hi Tommaso,
>
> On Thu, Oct 31, 2013 at 4:41 PM, Tommaso Teofili
> <to...@gmail.com> wrote:
>> Hi all,
>>
>> In the last weeks I've been working on an Apache Sling based replication
>> module and I'd like to donate it to the project so I'm wondering if that
>> may sound interesting for the community.
>>
>> Its main purpose is to "replicate" data between Sling instances.
>
> It sounds interesting indeed!
>
> A quick question - will this module work only between Sling instances,
> or will I be able to make it work with external systems? I'm thinking
> of integration with upstream proxy caches, like Varnish or Apache
> Traffic Server.
Hmm, this is goind on a tangent, but … I don’t think „replication“ is the correct mechanism to invalidate proxy caches.
Regards
Felix
>
> Robert
>
>>
>> Some design concepts :
>> - Replication agents are the responsible units for the replication of data
>> to a certain endpoint
>> - Each replication agent is an OSGi service
>> - Configuration of agents happens via OSGi configuration (e.g. by adding
>> nodes of type sling:OsgiConfig with certain properties as JSoN files)
>> - Configuration of agents include: endpoint, instance authentication,
>> transport implementation, binary format for data to be exchanged
>> - Each agent manages one or more queues of data packages to be replicated
>>
>> Main use case is master / slave architectures where data gets created on
>> the master and need to be replicated on the slaves which are publicly
>> available, or viceversa where data gets created by the user and needs to be
>> consistently replicated in the other instances of the architecture.
>>
>> The current implementation is still a work in progress so don't expect it
>> to be full feature but maybe it's worth having it among the extensions or
>> either in a sandbox space.
>>
>> I can create a Jira issue and add the code as a patch / zip file if there's
>> interest.
>> Looking forward to hear what you think,
>> Regards,
>> Tommaso
Re: [PROPOSAL] Sling Replication module
Posted by Robert Munteanu <ro...@apache.org>.
Hi Tommaso,
On Thu, Oct 31, 2013 at 4:41 PM, Tommaso Teofili
<to...@gmail.com> wrote:
> Hi all,
>
> In the last weeks I've been working on an Apache Sling based replication
> module and I'd like to donate it to the project so I'm wondering if that
> may sound interesting for the community.
>
> Its main purpose is to "replicate" data between Sling instances.
It sounds interesting indeed!
A quick question - will this module work only between Sling instances,
or will I be able to make it work with external systems? I'm thinking
of integration with upstream proxy caches, like Varnish or Apache
Traffic Server.
Robert
>
> Some design concepts :
> - Replication agents are the responsible units for the replication of data
> to a certain endpoint
> - Each replication agent is an OSGi service
> - Configuration of agents happens via OSGi configuration (e.g. by adding
> nodes of type sling:OsgiConfig with certain properties as JSoN files)
> - Configuration of agents include: endpoint, instance authentication,
> transport implementation, binary format for data to be exchanged
> - Each agent manages one or more queues of data packages to be replicated
>
> Main use case is master / slave architectures where data gets created on
> the master and need to be replicated on the slaves which are publicly
> available, or viceversa where data gets created by the user and needs to be
> consistently replicated in the other instances of the architecture.
>
> The current implementation is still a work in progress so don't expect it
> to be full feature but maybe it's worth having it among the extensions or
> either in a sandbox space.
>
> I can create a Jira issue and add the code as a patch / zip file if there's
> interest.
> Looking forward to hear what you think,
> Regards,
> Tommaso