You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Goutam Tadi <gt...@pivotal.io> on 2017/05/01 19:19:29 UTC
Remove keys from a PartitionedRegion
Hi Team,
With +Bradford D Boyle <bb...@pivotal.io>
We are trying to remove a set of keys from a `PartitionedRegion`.
Currently, we execute a function with `onRegion()`. Inside the function, we
call `PartitionRegionHelper.getLocalPrimaryData()` and use the returned
region to execute `region.removeAll(keys)`.
The problem we are facing is that is slow. Is there a faster way to remove
a set of keys from a partitioned region?
We are considering using `getDataStore().getAllLocalBucketRegions()` to get
the set of `BucketRegion`s and then using a thread pool to remove the keys
in parallel. Are there alternative approaches?
Thanks,
Goutam Tadi.
--
Regards,
*Goutam Tadi.*
Re: Remove keys from a PartitionedRegion
Posted by Barry Oglesby <bo...@pivotal.io>.
Goutam,
What are you doing to get from the input id to the actual keys to delete?
Are you running a query? If so, how are you running it, and is it indexed?
Thanks,
Barry Oglesby
On Tue, May 2, 2017 at 11:02 AM, Goutam Tadi <gt...@pivotal.io> wrote:
> Hi Barry,
>
> 1) OptimizeForWrite returns true.
>
> 2)
> FunctionService.onRegion(event.getRegion()).withArgs(id).execute(
> DeleteCollectedKeysFunction.ID);
>
>
> idabove helps to retrieve the set of keys inside the DeleteCollectedKeys
> funciton.
>
> Thanks.
>
>
>
>
>
> On Mon, May 1, 2017 at 5:17 PM Barry Oglesby <bo...@pivotal.io> wrote:
>
> > Are you passing the keys as a filter or an argument? What does
> > optimizeForWrite in your function return?
> >
> > Thanks,
> > Barry Oglesby
> >
> >
> > On Mon, May 1, 2017 at 4:41 PM, Goutam Tadi <gt...@pivotal.io> wrote:
> >
> > > Hi Dan,
> > >
> > > Thanks for the reply.
> > > No, we are neither executing the function nor passing keys from a
> client.
> > > We are trying to remove a significant portion of the keys from a region
> > > (most, but not all) at once.
> > >
> > > Thanks.
> > >
> > > On Mon, May 1, 2017 at 2:40 PM Dan Smith <ds...@pivotal.io> wrote:
> > >
> > > > That seems like it should do things fairly quickly. Are you executing
> > the
> > > > function from a client? Did you find that using a function was
> actually
> > > > faster than just calling removeAll from the client? I think removeAll
> > > from
> > > > the client should send your keys in a single message, similar to your
> > > > function approach.
> > > >
> > > > How many keys are you trying to remove? If you have a really large
> > number
> > > > of keys, it might be better to batch up the keys. You could do
> multiple
> > > > removeAlls from the client, perhaps even in parallel.
> > > >
> > > > -Dan
> > > >
> > > > On Mon, May 1, 2017 at 12:19 PM, Goutam Tadi <gt...@pivotal.io>
> wrote:
> > > >
> > > > > Hi Team,
> > > > >
> > > > > With +Bradford D Boyle <bb...@pivotal.io>
> > > > >
> > > > > We are trying to remove a set of keys from a `PartitionedRegion`.
> > > > > Currently, we execute a function with `onRegion()`. Inside the
> > > function,
> > > > we
> > > > > call `PartitionRegionHelper.getLocalPrimaryData()` and use the
> > > returned
> > > > > region to execute `region.removeAll(keys)`.
> > > > >
> > > > > The problem we are facing is that is slow. Is there a faster way to
> > > > remove
> > > > > a set of keys from a partitioned region?
> > > > >
> > > > > We are considering using `getDataStore().
> getAllLocalBucketRegions()`
> > > to
> > > > > get
> > > > > the set of `BucketRegion`s and then using a thread pool to remove
> the
> > > > keys
> > > > > in parallel. Are there alternative approaches?
> > > > >
> > > > > Thanks,
> > > > > Goutam Tadi.
> > > > > --
> > > > > Regards,
> > > > > *Goutam Tadi.*
> > > > >
> > > >
> > > --
> > > Regards,
> > > *Goutam Tadi.*
> > >
> >
> --
> Regards,
> *Goutam Tadi.*
>
Re: Remove keys from a PartitionedRegion
Posted by Goutam Tadi <gt...@pivotal.io>.
Hi Barry,
1) OptimizeForWrite returns true.
2)
FunctionService.onRegion(event.getRegion()).withArgs(id).execute(DeleteCollectedKeysFunction.ID);
idabove helps to retrieve the set of keys inside the DeleteCollectedKeys
funciton.
Thanks.
On Mon, May 1, 2017 at 5:17 PM Barry Oglesby <bo...@pivotal.io> wrote:
> Are you passing the keys as a filter or an argument? What does
> optimizeForWrite in your function return?
>
> Thanks,
> Barry Oglesby
>
>
> On Mon, May 1, 2017 at 4:41 PM, Goutam Tadi <gt...@pivotal.io> wrote:
>
> > Hi Dan,
> >
> > Thanks for the reply.
> > No, we are neither executing the function nor passing keys from a client.
> > We are trying to remove a significant portion of the keys from a region
> > (most, but not all) at once.
> >
> > Thanks.
> >
> > On Mon, May 1, 2017 at 2:40 PM Dan Smith <ds...@pivotal.io> wrote:
> >
> > > That seems like it should do things fairly quickly. Are you executing
> the
> > > function from a client? Did you find that using a function was actually
> > > faster than just calling removeAll from the client? I think removeAll
> > from
> > > the client should send your keys in a single message, similar to your
> > > function approach.
> > >
> > > How many keys are you trying to remove? If you have a really large
> number
> > > of keys, it might be better to batch up the keys. You could do multiple
> > > removeAlls from the client, perhaps even in parallel.
> > >
> > > -Dan
> > >
> > > On Mon, May 1, 2017 at 12:19 PM, Goutam Tadi <gt...@pivotal.io> wrote:
> > >
> > > > Hi Team,
> > > >
> > > > With +Bradford D Boyle <bb...@pivotal.io>
> > > >
> > > > We are trying to remove a set of keys from a `PartitionedRegion`.
> > > > Currently, we execute a function with `onRegion()`. Inside the
> > function,
> > > we
> > > > call `PartitionRegionHelper.getLocalPrimaryData()` and use the
> > returned
> > > > region to execute `region.removeAll(keys)`.
> > > >
> > > > The problem we are facing is that is slow. Is there a faster way to
> > > remove
> > > > a set of keys from a partitioned region?
> > > >
> > > > We are considering using `getDataStore().getAllLocalBucketRegions()`
> > to
> > > > get
> > > > the set of `BucketRegion`s and then using a thread pool to remove the
> > > keys
> > > > in parallel. Are there alternative approaches?
> > > >
> > > > Thanks,
> > > > Goutam Tadi.
> > > > --
> > > > Regards,
> > > > *Goutam Tadi.*
> > > >
> > >
> > --
> > Regards,
> > *Goutam Tadi.*
> >
>
--
Regards,
*Goutam Tadi.*
Re: Remove keys from a PartitionedRegion
Posted by Barry Oglesby <bo...@pivotal.io>.
Are you passing the keys as a filter or an argument? What does
optimizeForWrite in your function return?
Thanks,
Barry Oglesby
On Mon, May 1, 2017 at 4:41 PM, Goutam Tadi <gt...@pivotal.io> wrote:
> Hi Dan,
>
> Thanks for the reply.
> No, we are neither executing the function nor passing keys from a client.
> We are trying to remove a significant portion of the keys from a region
> (most, but not all) at once.
>
> Thanks.
>
> On Mon, May 1, 2017 at 2:40 PM Dan Smith <ds...@pivotal.io> wrote:
>
> > That seems like it should do things fairly quickly. Are you executing the
> > function from a client? Did you find that using a function was actually
> > faster than just calling removeAll from the client? I think removeAll
> from
> > the client should send your keys in a single message, similar to your
> > function approach.
> >
> > How many keys are you trying to remove? If you have a really large number
> > of keys, it might be better to batch up the keys. You could do multiple
> > removeAlls from the client, perhaps even in parallel.
> >
> > -Dan
> >
> > On Mon, May 1, 2017 at 12:19 PM, Goutam Tadi <gt...@pivotal.io> wrote:
> >
> > > Hi Team,
> > >
> > > With +Bradford D Boyle <bb...@pivotal.io>
> > >
> > > We are trying to remove a set of keys from a `PartitionedRegion`.
> > > Currently, we execute a function with `onRegion()`. Inside the
> function,
> > we
> > > call `PartitionRegionHelper.getLocalPrimaryData()` and use the
> returned
> > > region to execute `region.removeAll(keys)`.
> > >
> > > The problem we are facing is that is slow. Is there a faster way to
> > remove
> > > a set of keys from a partitioned region?
> > >
> > > We are considering using `getDataStore().getAllLocalBucketRegions()`
> to
> > > get
> > > the set of `BucketRegion`s and then using a thread pool to remove the
> > keys
> > > in parallel. Are there alternative approaches?
> > >
> > > Thanks,
> > > Goutam Tadi.
> > > --
> > > Regards,
> > > *Goutam Tadi.*
> > >
> >
> --
> Regards,
> *Goutam Tadi.*
>
Re: Remove keys from a PartitionedRegion
Posted by Goutam Tadi <gt...@pivotal.io>.
Hi Dan,
Thanks for the reply.
No, we are neither executing the function nor passing keys from a client.
We are trying to remove a significant portion of the keys from a region
(most, but not all) at once.
Thanks.
On Mon, May 1, 2017 at 2:40 PM Dan Smith <ds...@pivotal.io> wrote:
> That seems like it should do things fairly quickly. Are you executing the
> function from a client? Did you find that using a function was actually
> faster than just calling removeAll from the client? I think removeAll from
> the client should send your keys in a single message, similar to your
> function approach.
>
> How many keys are you trying to remove? If you have a really large number
> of keys, it might be better to batch up the keys. You could do multiple
> removeAlls from the client, perhaps even in parallel.
>
> -Dan
>
> On Mon, May 1, 2017 at 12:19 PM, Goutam Tadi <gt...@pivotal.io> wrote:
>
> > Hi Team,
> >
> > With +Bradford D Boyle <bb...@pivotal.io>
> >
> > We are trying to remove a set of keys from a `PartitionedRegion`.
> > Currently, we execute a function with `onRegion()`. Inside the function,
> we
> > call `PartitionRegionHelper.getLocalPrimaryData()` and use the returned
> > region to execute `region.removeAll(keys)`.
> >
> > The problem we are facing is that is slow. Is there a faster way to
> remove
> > a set of keys from a partitioned region?
> >
> > We are considering using `getDataStore().getAllLocalBucketRegions()` to
> > get
> > the set of `BucketRegion`s and then using a thread pool to remove the
> keys
> > in parallel. Are there alternative approaches?
> >
> > Thanks,
> > Goutam Tadi.
> > --
> > Regards,
> > *Goutam Tadi.*
> >
>
--
Regards,
*Goutam Tadi.*
Re: Remove keys from a PartitionedRegion
Posted by Dan Smith <ds...@pivotal.io>.
That seems like it should do things fairly quickly. Are you executing the
function from a client? Did you find that using a function was actually
faster than just calling removeAll from the client? I think removeAll from
the client should send your keys in a single message, similar to your
function approach.
How many keys are you trying to remove? If you have a really large number
of keys, it might be better to batch up the keys. You could do multiple
removeAlls from the client, perhaps even in parallel.
-Dan
On Mon, May 1, 2017 at 12:19 PM, Goutam Tadi <gt...@pivotal.io> wrote:
> Hi Team,
>
> With +Bradford D Boyle <bb...@pivotal.io>
>
> We are trying to remove a set of keys from a `PartitionedRegion`.
> Currently, we execute a function with `onRegion()`. Inside the function, we
> call `PartitionRegionHelper.getLocalPrimaryData()` and use the returned
> region to execute `region.removeAll(keys)`.
>
> The problem we are facing is that is slow. Is there a faster way to remove
> a set of keys from a partitioned region?
>
> We are considering using `getDataStore().getAllLocalBucketRegions()` to
> get
> the set of `BucketRegion`s and then using a thread pool to remove the keys
> in parallel. Are there alternative approaches?
>
> Thanks,
> Goutam Tadi.
> --
> Regards,
> *Goutam Tadi.*
>