You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openjpa.apache.org by Francesco Chicchiriccò <il...@apache.org> on 2012/05/30 12:09:00 UTC
Experimenting with MS SQL Azure federations
Hi all,
one of my company (Tirasa) customers asked Fabio Martelli (in CC) and me
to start exploring the integration between OpenJPA and MS SQL Azure
federations.
After some work we came out with [1], a sample webapp with persistence
layer (managed by OpenJPA) on MS SQL Azure; moreover, this sample code
makes profit of federations [2], a way to perform table partition, close
to the "sharding" concept.
SQL Azure federations were initially considered as mappable to OpenJPA
slices: in fact, both refers to 'sharding', however with slight
different meaning.
OpenJPA slices provide their own mechanism to handle at application
level distributed queries, data distribution and data replication1; SQL
Azure federations, instead, push such aspects to database configuration
level and empowers the ability to change some of these at runtime.
For example, with OpenJPA slices can be configured so that records from
table Person with field id value < 500 will be handled by slice1, while
the rest will be handled by slice2, where slice1 and slice2 are two
separated databases. This breaks the SQL Azure federations model,
because the Person table of this example can be dynamically split at
runtime, thus generating one or more federation members at persistence
level.
This implies that the actual federation member for a given record
* cannot be determined by configuration -- as OpenJPA slices require,
but only at runtime;
* cannot be chosen at application level but is enforced by persistence
level.
At the end, SQL Azure federations provide an additional scalability
layer and can then be seen as complementary to OpenJPA slices, rather
then opposite.
If anyone is interested, we also wrote a simple wiki page [3] about how
to get our demo code up an running.
Any feedback is welcome.
Our customer is currently evaluating whether to extend this feasibility
study with a complete implementation and would also like us to
contribute such code back to OpenJPA community.
Fabio Martelli and I are already ASF committers and PPMC members at
Apache Syncope [4], IdM solution with persistence based on OpenJPA.
WDYT?
Regards.
[1] https://github.com/Tirasa/protojpasqlazure/
[2] http://msdn.microsoft.com/en-us/library/windowsazure/hh597452.aspx
[3] https://github.com/Tirasa/protojpasqlazure/wiki/Setup
[4] http://incubator.apache.org/syncope/
--
Francesco Chicchiriccò
ASF Member, Apache Cocoon PMC and Apache Syncope PPMC Member
http://people.apache.org/~ilgrosso/
Re: Experimenting with MS SQL Azure federations
Posted by Francesco Chicchiriccò <il...@apache.org>.
On 14/08/2012 12.30, Mark Struberg wrote:
> All this stuff is pluggable and only uses the SPI, right?
Correct: you can take a look at the current feasibility study
implementation [1] source code for this; anyway, we plan to keep this
also for the upcoming full implementation.
I can think that - maybe - we'd need to introduce some more SPI
somewhere in order to have a cleaner integration, but I cannot be 100%
sure at the moment.
> Are there comparable features needed for other NoSQL/bigdata storages as well?
Hum, I have very poor knowledge about such topics...
Regards.
> ----- Original Message -----
>> From: Francesco Chicchiriccò <il...@apache.org>
>> To: dev@openjpa.apache.org
>> Cc:
>> Sent: Monday, August 13, 2012 4:18 PM
>> Subject: Re: Experimenting with MS SQL Azure federations
>>
>> Hi all,
>> anyone interested in this topic?
>>
>> After the initial feasibility study, our customer is now about to
>> commission the whole implementation.
>>
>> We can release this under AL 2.0 and put it somewhere (apache-extras,
>> github), but if possible we would like to involve the OpenJPA community
>> in this, possibly contributing to mainstream OpenJPA code & doc.
>>
>> WDYT?
>>
>> On 30/05/2012 12.09, Francesco Chicchiriccò wrote:
>>> Hi all,
>>> one of my company (Tirasa) customers asked Fabio Martelli (in CC) and me
>>> to start exploring the integration between OpenJPA and MS SQL Azure
>>> federations.
>>>
>>> After some work we came out with [1], a sample webapp with persistence
>>> layer (managed by OpenJPA) on MS SQL Azure; moreover, this sample code
>>> makes profit of federations [2], a way to perform table partition, close
>>> to the "sharding" concept.
>>>
>>> SQL Azure federations were initially considered as mappable to OpenJPA
>>> slices: in fact, both refers to 'sharding', however with slight
>>> different meaning.
>>> OpenJPA slices provide their own mechanism to handle at application
>>> level distributed queries, data distribution and data replication1; SQL
>>> Azure federations, instead, push such aspects to database configuration
>>> level and empowers the ability to change some of these at runtime.
>>>
>>> For example, with OpenJPA slices can be configured so that records from
>>> table Person with field id value < 500 will be handled by slice1, while
>>> the rest will be handled by slice2, where slice1 and slice2 are two
>>> separated databases. This breaks the SQL Azure federations model,
>>> because the Person table of this example can be dynamically split at
>>> runtime, thus generating one or more federation members at persistence
>>> level.
>>>
>>> This implies that the actual federation member for a given record
>>> * cannot be determined by configuration -- as OpenJPA slices require,
>>> but only at runtime;
>>> * cannot be chosen at application level but is enforced by persistence
>>> level.
>>>
>>> At the end, SQL Azure federations provide an additional scalability
>>> layer and can then be seen as complementary to OpenJPA slices, rather
>>> then opposite.
>>>
>>> If anyone is interested, we also wrote a simple wiki page [3] about how
>>> to get our demo code up an running.
>>> Any feedback is welcome.
>>>
>>> Our customer is currently evaluating whether to extend this feasibility
>>> study with a complete implementation and would also like us to
>>> contribute such code back to OpenJPA community.
>>> Fabio Martelli and I are already ASF committers and PPMC members at
>>> Apache Syncope [4], IdM solution with persistence based on OpenJPA.
>>>
>>> WDYT?
>>>
>>> Regards.
>>>
>>> [1] https://github.com/Tirasa/protojpasqlazure/
>>> [2] http://msdn.microsoft.com/en-us/library/windowsazure/hh597452.aspx
>>> [3] https://github.com/Tirasa/protojpasqlazure/wiki/Setup
>>> [4] http://incubator.apache.org/syncope/
--
Francesco Chicchiriccò
ASF Member, Apache Cocoon PMC and Apache Syncope PPMC Member
http://people.apache.org/~ilgrosso/
Re: Experimenting with MS SQL Azure federations
Posted by Mark Struberg <st...@yahoo.de>.
All this stuff is pluggable and only uses the SPI, right?
Are there comparable features needed for other NoSQL/bigdata storages as well?
LieGrue,
strub
----- Original Message -----
> From: Francesco Chicchiriccò <il...@apache.org>
> To: dev@openjpa.apache.org
> Cc:
> Sent: Monday, August 13, 2012 4:18 PM
> Subject: Re: Experimenting with MS SQL Azure federations
>
> Hi all,
> anyone interested in this topic?
>
> After the initial feasibility study, our customer is now about to
> commission the whole implementation.
>
> We can release this under AL 2.0 and put it somewhere (apache-extras,
> github), but if possible we would like to involve the OpenJPA community
> in this, possibly contributing to mainstream OpenJPA code & doc.
>
> WDYT?
>
> On 30/05/2012 12.09, Francesco Chicchiriccò wrote:
>> Hi all,
>> one of my company (Tirasa) customers asked Fabio Martelli (in CC) and me
>> to start exploring the integration between OpenJPA and MS SQL Azure
>> federations.
>>
>> After some work we came out with [1], a sample webapp with persistence
>> layer (managed by OpenJPA) on MS SQL Azure; moreover, this sample code
>> makes profit of federations [2], a way to perform table partition, close
>> to the "sharding" concept.
>>
>> SQL Azure federations were initially considered as mappable to OpenJPA
>> slices: in fact, both refers to 'sharding', however with slight
>> different meaning.
>> OpenJPA slices provide their own mechanism to handle at application
>> level distributed queries, data distribution and data replication1; SQL
>> Azure federations, instead, push such aspects to database configuration
>> level and empowers the ability to change some of these at runtime.
>>
>> For example, with OpenJPA slices can be configured so that records from
>> table Person with field id value < 500 will be handled by slice1, while
>> the rest will be handled by slice2, where slice1 and slice2 are two
>> separated databases. This breaks the SQL Azure federations model,
>> because the Person table of this example can be dynamically split at
>> runtime, thus generating one or more federation members at persistence
>> level.
>>
>> This implies that the actual federation member for a given record
>> * cannot be determined by configuration -- as OpenJPA slices require,
>> but only at runtime;
>> * cannot be chosen at application level but is enforced by persistence
>> level.
>>
>> At the end, SQL Azure federations provide an additional scalability
>> layer and can then be seen as complementary to OpenJPA slices, rather
>> then opposite.
>>
>> If anyone is interested, we also wrote a simple wiki page [3] about how
>> to get our demo code up an running.
>> Any feedback is welcome.
>>
>> Our customer is currently evaluating whether to extend this feasibility
>> study with a complete implementation and would also like us to
>> contribute such code back to OpenJPA community.
>> Fabio Martelli and I are already ASF committers and PPMC members at
>> Apache Syncope [4], IdM solution with persistence based on OpenJPA.
>>
>> WDYT?
>>
>> Regards.
>>
>> [1] https://github.com/Tirasa/protojpasqlazure/
>> [2] http://msdn.microsoft.com/en-us/library/windowsazure/hh597452.aspx
>> [3] https://github.com/Tirasa/protojpasqlazure/wiki/Setup
>> [4] http://incubator.apache.org/syncope/
>
> --
> Francesco Chicchiriccò
>
> ASF Member, Apache Cocoon PMC and Apache Syncope PPMC Member
> http://people.apache.org/~ilgrosso/
>
Re: Experimenting with MS SQL Azure federations
Posted by Francesco Chicchiriccò <il...@apache.org>.
Hi all,
anyone interested in this topic?
After the initial feasibility study, our customer is now about to
commission the whole implementation.
We can release this under AL 2.0 and put it somewhere (apache-extras,
github), but if possible we would like to involve the OpenJPA community
in this, possibly contributing to mainstream OpenJPA code & doc.
WDYT?
On 30/05/2012 12.09, Francesco Chicchiriccò wrote:
> Hi all,
> one of my company (Tirasa) customers asked Fabio Martelli (in CC) and me
> to start exploring the integration between OpenJPA and MS SQL Azure
> federations.
>
> After some work we came out with [1], a sample webapp with persistence
> layer (managed by OpenJPA) on MS SQL Azure; moreover, this sample code
> makes profit of federations [2], a way to perform table partition, close
> to the "sharding" concept.
>
> SQL Azure federations were initially considered as mappable to OpenJPA
> slices: in fact, both refers to 'sharding', however with slight
> different meaning.
> OpenJPA slices provide their own mechanism to handle at application
> level distributed queries, data distribution and data replication1; SQL
> Azure federations, instead, push such aspects to database configuration
> level and empowers the ability to change some of these at runtime.
>
> For example, with OpenJPA slices can be configured so that records from
> table Person with field id value < 500 will be handled by slice1, while
> the rest will be handled by slice2, where slice1 and slice2 are two
> separated databases. This breaks the SQL Azure federations model,
> because the Person table of this example can be dynamically split at
> runtime, thus generating one or more federation members at persistence
> level.
>
> This implies that the actual federation member for a given record
> * cannot be determined by configuration -- as OpenJPA slices require,
> but only at runtime;
> * cannot be chosen at application level but is enforced by persistence
> level.
>
> At the end, SQL Azure federations provide an additional scalability
> layer and can then be seen as complementary to OpenJPA slices, rather
> then opposite.
>
> If anyone is interested, we also wrote a simple wiki page [3] about how
> to get our demo code up an running.
> Any feedback is welcome.
>
> Our customer is currently evaluating whether to extend this feasibility
> study with a complete implementation and would also like us to
> contribute such code back to OpenJPA community.
> Fabio Martelli and I are already ASF committers and PPMC members at
> Apache Syncope [4], IdM solution with persistence based on OpenJPA.
>
> WDYT?
>
> Regards.
>
> [1] https://github.com/Tirasa/protojpasqlazure/
> [2] http://msdn.microsoft.com/en-us/library/windowsazure/hh597452.aspx
> [3] https://github.com/Tirasa/protojpasqlazure/wiki/Setup
> [4] http://incubator.apache.org/syncope/
--
Francesco Chicchiriccò
ASF Member, Apache Cocoon PMC and Apache Syncope PPMC Member
http://people.apache.org/~ilgrosso/
Re: Experimenting with MS SQL Azure federations
Posted by Fabio Martelli <fa...@gmail.com>.
Hi Pinaki,
first of all thank you for your prompt reply and for references on OpenJPA Slice.
About what you say, please find my comments in-line.
Il giorno 30/mag/2012, alle ore 21.41, Pinaki Poddar ha scritto:
> Hi,
> I liked your concise description of Slice -- a more verbose description is
> available at [1].
>
> Slice is designed to handle federation at JPA/application layer and via
> Slice it is possible to support a multi-tenant environment using a set of
> databases that are *not* federated by themselves. In fact, Slice can work
> with the databases that are heterogeneous as well.
Right! ... and if I well understood, by specifying a distribution policy I can say to OpenJPA where read&write specific data ...
> For Azure or other federated databases, the relevance of Slice is diminished
> as sharding concerns are handled by database server itself. From a JPA
> application standpoint, it will see a federated database as a "single"
> database.
Sure, from a certain point of view I can agree.
Azure gives me the possibility to design and configure my federated SQL Azure database but data distribution must be handled at the application layer.
In fact, by using OpenJPA as is my application won't be able to see and manage all the federations but only the federation root database.
In this case I have no way to execute a statement on a particular federation but just on the root.
Further, things become more complicated when we consider that, usually, each federation is composed of several members.
This is the reason why, from my point of view, in order to have a OpenJPA based application with a federated SQL Azure db as a back-end we have to focus on:
1. how to distribute data among federations
2. how to distribute data among federation members
> So effectively, I see Slice and federated database as an either-or option,
> depending upon the design decision of which layer (database or JPA) is
> better suited to handle sharding. However, Slice does provide some extra
> features such as @Replicated types for master or reference data or query
> targeting.
>
> I would like to know from your experience with federated databases, how
> these features are supported by Azure.
I think that slices can help to specify a data distribution policy against federations.
I start from the following considerations:
1. federations are not so subject to changes; from my point of view, federations are design matter
1. unfortunately, federation members seems to be more dynamic (someone could choose to split members at runtime because a certain table is grown too much)
So, my idea is to consider a federation as a single database: in order to integrate the whole federated db we can use more slices, one for each federation.
Data distribution policy against federations will be configurable thanks to Slice features.
Data distribution policy against federation members will be managed at runtime reading from SQL Azures's sys tables member distribution for each federation.
I do think that by considering each federation as a single DB we can profit from all the features of Slice.
Please, let me know what you think.
Best regards,
F.
Re: Experimenting with MS SQL Azure federations
Posted by Pinaki Poddar <pp...@apache.org>.
Few references on Slice (missed in the previous post):
[1] http://www.ibm.com/developerworks/opensource/library/os-openjpa/
[2]
http://www.slideshare.net/pinaki.poddar/slice-for-distributed-persistence-javaone-2010
[3]
http://openjpa.apache.org/builds/latest/docs/docbook/manual.html#ref_guide_slice
-----
Pinaki Poddar
Chair, Apache OpenJPA Project
--
View this message in context: http://openjpa.208410.n2.nabble.com/Experimenting-with-MS-SQL-Azure-federations-tp7579852p7580061.html
Sent from the OpenJPA Developers mailing list archive at Nabble.com.
Re: Experimenting with MS SQL Azure federations
Posted by Pinaki Poddar <pp...@apache.org>.
Hi,
I liked your concise description of Slice -- a more verbose description is
available at [1].
Slice is designed to handle federation at JPA/application layer and via
Slice it is possible to support a multi-tenant environment using a set of
databases that are *not* federated by themselves. In fact, Slice can work
with the databases that are heterogeneous as well.
For Azure or other federated databases, the relevance of Slice is diminished
as sharding concerns are handled by database server itself. From a JPA
application standpoint, it will see a federated database as a "single"
database.
So effectively, I see Slice and federated database as an either-or option,
depending upon the design decision of which layer (database or JPA) is
better suited to handle sharding. However, Slice does provide some extra
features such as @Replicated types for master or reference data or query
targeting.
I would like to know from your experience with federated databases, how
these features are supported by Azure.
Regards --
-----
Pinaki Poddar
Chair, Apache OpenJPA Project
--
View this message in context: http://openjpa.208410.n2.nabble.com/Experimenting-with-MS-SQL-Azure-federations-tp7579852p7580058.html
Sent from the OpenJPA Developers mailing list archive at Nabble.com.