You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by Martin Harris <ma...@cloudsoftcorp.com> on 2014/05/14 11:36:12 UTC
Re: [brooklyn-dev] Object store persistence
How frequently is the object store written to / read from?
On 14 May 2014 08:50, Andrea Turli <an...@cloudsoftcorp.com> wrote:
> Hi,
>
> I'm thinking about adding support to persist to generic object store.
> The basic idea is to use the jclouds blobStore abstraction and give users
> the ability to specify a `persistenceLocation` that can is a location spec,
> where they can specify the cloud provider they want to use.
> By default, If `persistenceLocation` is not specified the `persistenceDir`
> will be created on the local filesystem at the desired path (as it is now).
> If `persistenceLocation` is specified, `persistenceDir` will be used as
> relativePath inside the cloud providers
> (s3:/path/specified/by/persistenceDir), if available, otherwise the
> serialized brooklyn objects will be created in the root folder of the
> account.
>
> Long-term, it would be nice to add a jsonSerializer to support nosql
> technologies that support only json documents.
>
> Any thoughts?
>
> Best,
> Andrea
>
> --
> You received this message because you are subscribed to the Google Groups
> "brooklyn-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to brooklyn-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
Martin Harris
Lead Software Engineer
Cloudsoft Corporation Ltd
www.cloudsoftcorp.com
Mobile: +44 (0)7989 047-855
Re: [brooklyn-dev] Object store persistence
Posted by Aled Sage <al...@gmail.com>.
Hi Andrea,
The other time that affects write-frequency is in
RebindManagerImpl.periodicPersistPeriod. This is the major one as it
affects the maximum frequency that (changed) entities/locations will be
persisted.
---
For some code pointers into hooking in the blob-store persistence...
See BrooklynMementoPersisterToMultiFile for where we control writing to
and reading from the file system.
I suspect this could be generalised so that it isn't just about files.
It delegates to... MementoFileWriter. This is the file-specific piece
that you'd want to replace. However, it also includes generic logic to
handle concurrency: if asked to write,write,write then it handles
queuing up these writes and discarding the middle write if the new
request comes in before the middle one has started (etc). It also
handles concurrent calls to write/delete to ensure that the last-update
wins.
---
Rather than completely duplicating these classes, it's probably worth
starting with an interface such as:
public interface ObjectStore {
public interface ObjectAccessor<T> {
void writeAsync(T val);
void deleteAsync();
T read();
}
/**
* For reading/writing data to the given path.
* For example, if a file-based ObjectStore is configured to
write to file://path/to/root/
* then subPath=abc would read/write data to
file://path/to/root/abc
*/
ObjectAccessor newAccessor(String subPath);
/**
* Lists the paths of objects contained at the given path.
* For example, if a file-based ObjectStore is configured to
write to file://path/to/root/
* then parentSubPath=entities would return the contents of
/path/to/root/entities/, such as
* [entities/e1, entities/e2, entities/e3].
* The returned paths values are usable in calls to {@link
#newAccessor(String)}.
*/
List<String> listContents(String parentSubPath);
/**
* Makes a copy of all objects under parentSubPath.
* For example, if a file-based ObjectStore is configured to
write to file://path/to/root/
* then parentSubPath="abc", backupSubPath="abc.bak" would be
the same as:
* `cp -R /path/to/root/abc/* /path/to/root/abc.bak/`
*/
void backupContents(String parentSubPath, backupSubPath);
/**
* Closes all resources used by this ObjectStore. No subsequent
calls should be made to the ObjectStore;
* behaviour of such calls is undefined but likely to throw
exceptions.
*/
void close();
}
The BrooklynMementoPersisterToMultiFile could then be refactored to use
this, and the existing file-based code refactored to implement that
interface.
Does that make sense to you?
Thoughts?
Aled
On 14/05/2014 10:52, Andrea Turli wrote:
> Hi Martin,
>
> for example, HighAvailabilityManagerImpl has a pollingTask that polls
> every 5 seconds by default, so I think this will be the read frequency.
>
> Does it answer your question?
>
> Andrea
>
>
> On 14 May 2014 11:36, Martin Harris <martin.harris@cloudsoftcorp.com
> <ma...@cloudsoftcorp.com>> wrote:
>
> How frequently is the object store written to / read from?
>
>
> On 14 May 2014 08:50, Andrea Turli <andrea.turli@cloudsoftcorp.com
> <ma...@cloudsoftcorp.com>> wrote:
>
> Hi,
>
> I'm thinking about adding support to persist to generic object
> store.
> The basic idea is to use the jclouds blobStore abstraction and
> give users the ability to specify a `persistenceLocation` that
> can is a location spec, where they can specify the cloud
> provider they want to use.
> By default, If `persistenceLocation` is not specified the
> `persistenceDir` will be created on the local filesystem at
> the desired path (as it is now). If `persistenceLocation` is
> specified, `persistenceDir` will be used as relativePath
> inside the cloud providers
> (s3:/path/specified/by/persistenceDir), if available,
> otherwise the serialized brooklyn objects will be created in
> the root folder of the account.
>
> Long-term, it would be nice to add a jsonSerializer to support
> nosql technologies that support only json documents.
>
> Any thoughts?
>
> Best,
> Andrea
> --
> You received this message because you are subscribed to the
> Google Groups "brooklyn-dev" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to brooklyn-dev+unsubscribe@googlegroups.com
> <ma...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Martin Harris
> Lead Software Engineer
> Cloudsoft Corporation Ltd
> www.cloudsoftcorp.com <http://www.cloudsoftcorp.com>
> Mobile: +44 (0)7989 047-855 <tel:%2B44%20%280%297989%20047-855>
> --
> You received this message because you are subscribed to the Google
> Groups "brooklyn-dev" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to brooklyn-dev+unsubscribe@googlegroups.com
> <ma...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "brooklyn-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to brooklyn-dev+unsubscribe@googlegroups.com
> <ma...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.
Re: [brooklyn-dev] Object store persistence
Posted by Andrea Turli <an...@cloudsoftcorp.com>.
Hi Martin,
for example, HighAvailabilityManagerImpl has a pollingTask that polls every
5 seconds by default, so I think this will be the read frequency.
Does it answer your question?
Andrea
On 14 May 2014 11:36, Martin Harris <ma...@cloudsoftcorp.com> wrote:
> How frequently is the object store written to / read from?
>
>
> On 14 May 2014 08:50, Andrea Turli <an...@cloudsoftcorp.com> wrote:
>
>> Hi,
>>
>> I'm thinking about adding support to persist to generic object store.
>> The basic idea is to use the jclouds blobStore abstraction and give users
>> the ability to specify a `persistenceLocation` that can is a location spec,
>> where they can specify the cloud provider they want to use.
>> By default, If `persistenceLocation` is not specified the
>> `persistenceDir` will be created on the local filesystem at the desired
>> path (as it is now). If `persistenceLocation` is specified,
>> `persistenceDir` will be used as relativePath inside the cloud providers
>> (s3:/path/specified/by/persistenceDir), if available, otherwise the
>> serialized brooklyn objects will be created in the root folder of the
>> account.
>>
>> Long-term, it would be nice to add a jsonSerializer to support nosql
>> technologies that support only json documents.
>>
>> Any thoughts?
>>
>> Best,
>> Andrea
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "brooklyn-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to brooklyn-dev+unsubscribe@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Martin Harris
> Lead Software Engineer
> Cloudsoft Corporation Ltd
> www.cloudsoftcorp.com
> Mobile: +44 (0)7989 047-855
>
> --
> You received this message because you are subscribed to the Google Groups
> "brooklyn-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to brooklyn-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>