You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by ph...@gmail.com on 2018/11/01 11:26:04 UTC
Removing region data
Hi,
I am investigating removing a large amount of data from a partitioned
region (with 1 redundant copy) and I am wondering what is the recommended
way to run this operation. I understand the operation should be invoked as
a function executing on the servers but as for the remove algorithm there
appear to be multiple options:
1. Retrieve the keys (millions) to be evicted and pass this to
localData.removeAll()
2. Parallel stream over the local data, filtering and calling
localData.remove() on each entry. This is safe as no
ConcurrentModificationException should be thrown from entrySet().
3. Retrieve the keys to evict, then batch calling removeAll() for each
batch on a single thread.
4. Others?
I appreciate this question is fairly open ended and subject to specific
performance tests, but I am interested to hear opinions/recommendations on
how this sort of operations should be performed.
Thanks in advance,
Phil
Re: Removing region data
Posted by Udo Kohlmeyer <ud...@apache.org>.
In addition to this, be aware that when you remove large amounts of data
(in large of small batches) could possibly result in the large scale
garbage generation. Which could lead to the JVM GC kicking in and
wanting to clean up things.. which could cause a "stop the world" gc.
As Mike and Anthony have indicated, when you do the large scale clean
up, doing it in such a manner that it becomes less of an big bang burden
on the system is advisable.
--Udo
On 11/1/18 08:29, Michael Stolz wrote:
>
> 1. Retrieve the keys (millions) to be evicted and pass this to
> localData.removeAll()
> MS: This option is probably not a good idea for a few reasons.
> First, it will take quite a lot of memory to pass the list to
> localData.removeAll, thereby making a significant amount of garbage.
> Second, it will result in one massive collection of GemFire Tombstones
> exactly 10 minutes later, which will likely affect other processing.
>
> 2. Parallel stream over the local data, filtering and calling
> localData.remove() on each entry. This is safe as no
> ConcurrentModificationException should be thrown from entrySet().
> MS: This is likely to be the most efficient in terms of memory usage.
> The individual removes might be significantly slower than removeAll()s
> with batches of 100 or 1000. Also with the removeAll()s there is
> probably no need for multiple threads. It would be good to put some
> time between batches so that the Tombstone collection will be in small
> batches as well.
>
> 3. Retrieve the keys to evict, then batch calling removeAll() for each
> batch on a single thread.
> MS: Actually this if done on the local data is the same as what I
> described in my answer to #2.
>
> Of course these are just my best guess, actual testing and measurement
> is probably a good idea.
>
> One more thing, when it comes to big batch removes I always question
> if my objective is to get them removed as quickly as possible, or get
> them removed with the least impact to system resources. Usually the
> answer ends up being the least impact to system resources. If that is
> the case, doing things that make it go faster are worse than doing
> things that make it go slower.
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Lead
> Mobile: +1-631-835-4771
>
>
>
> On Thu, Nov 1, 2018 at 7:26 AM <phillip.applegate@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi,
>
> I am investigating removing a large amount of data from a
> partitioned region (with 1 redundant copy) and I am wondering what
> is the recommended way to run this operation. I understand the
> operation should be invoked as a function executing on the servers
> but as for the remove algorithm there appear to be multiple options:
>
> 1. Retrieve the keys (millions) to be evicted and pass this to
> localData.removeAll()
>
> 2. Parallel stream over the local data, filtering and calling
> localData.remove() on each entry. This is safe as no
> ConcurrentModificationException should be thrown from entrySet().
>
> 3. Retrieve the keys to evict, then batch calling removeAll() for
> each batch on a single thread.
>
> 4. Others?
>
> I appreciate this question is fairly open ended and subject to
> specific performance tests, but I am interested to hear
> opinions/recommendations on how this sort of operations should be
> performed.
>
> Thanks in advance,
>
> Phil
>
Re: Removing region data
Posted by Michael Stolz <ms...@pivotal.io>.
1. Retrieve the keys (millions) to be evicted and pass this to
localData.removeAll()
MS: This option is probably not a good idea for a few reasons.
First, it will take quite a lot of memory to pass the list to
localData.removeAll, thereby making a significant amount of garbage.
Second, it will result in one massive collection of GemFire Tombstones
exactly 10 minutes later, which will likely affect other processing.
2. Parallel stream over the local data, filtering and calling
localData.remove() on each entry. This is safe as no
ConcurrentModificationException should be thrown from entrySet().
MS: This is likely to be the most efficient in terms of memory usage. The
individual removes might be significantly slower than removeAll()s with
batches of 100 or 1000. Also with the removeAll()s there is probably no
need for multiple threads. It would be good to put some time between
batches so that the Tombstone collection will be in small batches as well.
3. Retrieve the keys to evict, then batch calling removeAll() for each
batch on a single thread.
MS: Actually this if done on the local data is the same as what I described
in my answer to #2.
Of course these are just my best guess, actual testing and measurement is
probably a good idea.
One more thing, when it comes to big batch removes I always question if my
objective is to get them removed as quickly as possible, or get them
removed with the least impact to system resources. Usually the answer ends
up being the least impact to system resources. If that is the case, doing
things that make it go faster are worse than doing things that make it go
slower.
--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
On Thu, Nov 1, 2018 at 7:26 AM <ph...@gmail.com> wrote:
> Hi,
>
> I am investigating removing a large amount of data from a partitioned
> region (with 1 redundant copy) and I am wondering what is the recommended
> way to run this operation. I understand the operation should be invoked as
> a function executing on the servers but as for the remove algorithm there
> appear to be multiple options:
>
> 1. Retrieve the keys (millions) to be evicted and pass this to
> localData.removeAll()
>
> 2. Parallel stream over the local data, filtering and calling
> localData.remove() on each entry. This is safe as no
> ConcurrentModificationException should be thrown from entrySet().
>
> 3. Retrieve the keys to evict, then batch calling removeAll() for each
> batch on a single thread.
>
> 4. Others?
>
> I appreciate this question is fairly open ended and subject to specific
> performance tests, but I am interested to hear opinions/recommendations on
> how this sort of operations should be performed.
>
> Thanks in advance,
>
> Phil
>
Re: Removing region data
Posted by Anthony Baker <ab...@pivotal.io>.
One other consideration is the impact of large batch processing operations on your workload. Do you have sufficient cpu / network capacity to handle the extra work? How will that affect response times? I’ve seen some users break down work into smaller batches and amortize the cost over time.
YMMV,
Anthony
> On Nov 1, 2018, at 4:26 AM, phillip.applegate@gmail.com wrote:
>
> Hi,
>
> I am investigating removing a large amount of data from a partitioned region (with 1 redundant copy) and I am wondering what is the recommended way to run this operation. I understand the operation should be invoked as a function executing on the servers but as for the remove algorithm there appear to be multiple options:
>
> 1. Retrieve the keys (millions) to be evicted and pass this to localData.removeAll()
>
> 2. Parallel stream over the local data, filtering and calling localData.remove() on each entry. This is safe as no ConcurrentModificationException should be thrown from entrySet().
>
> 3. Retrieve the keys to evict, then batch calling removeAll() for each batch on a single thread.
>
> 4. Others?
>
> I appreciate this question is fairly open ended and subject to specific performance tests, but I am interested to hear opinions/recommendations on how this sort of operations should be performed.
>
> Thanks in advance,
>
> Phil
Re: Removing region data
Posted by aashish choudhary <aa...@gmail.com>.
Not sure if you have gone through this but it might help.
https://stackoverflow.com/questions/37392844/delete-query-in-gemfire-cache
Thanks,
Ashish
On Thu, Nov 1, 2018, 4:56 PM <ph...@gmail.com> wrote:
> Hi,
>
> I am investigating removing a large amount of data from a partitioned
> region (with 1 redundant copy) and I am wondering what is the recommended
> way to run this operation. I understand the operation should be invoked as
> a function executing on the servers but as for the remove algorithm there
> appear to be multiple options:
>
> 1. Retrieve the keys (millions) to be evicted and pass this to
> localData.removeAll()
>
> 2. Parallel stream over the local data, filtering and calling
> localData.remove() on each entry. This is safe as no
> ConcurrentModificationException should be thrown from entrySet().
>
> 3. Retrieve the keys to evict, then batch calling removeAll() for each
> batch on a single thread.
>
> 4. Others?
>
> I appreciate this question is fairly open ended and subject to specific
> performance tests, but I am interested to hear opinions/recommendations on
> how this sort of operations should be performed.
>
> Thanks in advance,
>
> Phil
>