Re: [DISCUSS] _db_updates feed in FoundationDB

[Alex Miller <al...@apple.com.INVALID>]

> On 2019/03/20 22:47:42, Adam Kocoloski <ko...@apache.org> wrote: 
> I don’t know in detail what optimizations FoundationDB applies to atomic operations (e.g. coalescing them at a layer above the storage engine). That’s worth checking into, as otherwise I’d be concerned about introducing super-hot keys here.


Inducing hot keys or hot shards would both be of concern.  There is logic that will try to deal with write hotspots, but it splits shard based on write bandwidth to a particular shard, which likely wouldn’t be triggered by a stream of atomic operations.

Clients will merge atomic operations together if you issue multiple in a transaction, e.g. I believe three ATOMIC_ADDs on one key will become one ATOMIC_ADD of the sum.  There isn’t particularly any logic that optimizes the handling of atomic operations once they leave the client.  Storage servers only commit every ~250ms, so this shouldn’t translate into a full page write on each atomic operation to the same key, but the storage server will end up applying each atomic operation individually.

( And I’ve gone and filed https://github.com/apple/foundationdb/issues/1450 to trigger more discussion within FoundationDB on this topic. )


> On Mar 25, 2019, at 11:42 AM, Adam Kocoloski <ko...@apache.org> wrote:
>> On Mar 25, 2019, at 12:48 PM, Mike Rhodes <co...@dx13.co.uk> wrote:
>> 
>> I couldn't immediately see how we cleared out older entries from this potentially very large queue. For example, the worker processing the queue to deduplicate might issue range deletes after processing each "batch". Is this simple enough to do?
> 
> Yes, that’s the (implicit) idea. Simple to implement, not clear to me how well the storage servers can handle the load. I think the “range clears are cheap” statement largely refers to the transaction management system.

At the storage level, range clears are cheap in terms of immediate execution, and slow in terms of total work performed.  You can find the details of this in https://forums.foundationdb.org/t/shards-are-not-splitted-into-smaller-ones/815/4 , and an example of what to be careful about with range clears on https://forums.foundationdb.org/t/used-disk-space-dramatically-increases-while-sum-of-key-value-sizes-is-constant/644


> On Mar 27, 2019, at 11:07 AM, Ilya Khlopotov <ii...@apache.org> wrote:
> What if FDB would support a list type for a value and would have an atomic operation to add the value to the list if it is missing. In this case we could store the data we need as follows (under separate subspace TBD).
> VersionStamp = (DbName, EventType)
> DbName = [versionstamps]
> ...
> The question is how we would implement atomic addition of a value to a list. 

If I’ve understood your proposal correctly, I think it sounds easier to just use APPEND_IF_FITS ?

https://apple.github.io/foundationdb/javadoc/index.html?com/apple/foundationdb/MutationType.html

Re: [DISCUSS] _db_updates feed in FoundationDB

[Adam Kocoloski <ko...@apache.org>]

Thanks Alex - I need to get in the habit of posting more frequently over there :)

Adam

> On Apr 10, 2019, at 3:16 AM, Alex Miller <al...@apple.com.INVALID> wrote:
> 
> 
>> On 2019/03/20 22:47:42, Adam Kocoloski <ko...@apache.org> wrote: 
>> I don’t know in detail what optimizations FoundationDB applies to atomic operations (e.g. coalescing them at a layer above the storage engine). That’s worth checking into, as otherwise I’d be concerned about introducing super-hot keys here.
> 
> 
> Inducing hot keys or hot shards would both be of concern.  There is logic that will try to deal with write hotspots, but it splits shard based on write bandwidth to a particular shard, which likely wouldn’t be triggered by a stream of atomic operations.
> 
> Clients will merge atomic operations together if you issue multiple in a transaction, e.g. I believe three ATOMIC_ADDs on one key will become one ATOMIC_ADD of the sum.  There isn’t particularly any logic that optimizes the handling of atomic operations once they leave the client.  Storage servers only commit every ~250ms, so this shouldn’t translate into a full page write on each atomic operation to the same key, but the storage server will end up applying each atomic operation individually.
> 
> ( And I’ve gone and filed https://github.com/apple/foundationdb/issues/1450 to trigger more discussion within FoundationDB on this topic. )
> 
> 
>> On Mar 25, 2019, at 11:42 AM, Adam Kocoloski <ko...@apache.org> wrote:
>>> On Mar 25, 2019, at 12:48 PM, Mike Rhodes <co...@dx13.co.uk> wrote:
>>> 
>>> I couldn't immediately see how we cleared out older entries from this potentially very large queue. For example, the worker processing the queue to deduplicate might issue range deletes after processing each "batch". Is this simple enough to do?
>> 
>> Yes, that’s the (implicit) idea. Simple to implement, not clear to me how well the storage servers can handle the load. I think the “range clears are cheap” statement largely refers to the transaction management system.
> 
> At the storage level, range clears are cheap in terms of immediate execution, and slow in terms of total work performed.  You can find the details of this in https://forums.foundationdb.org/t/shards-are-not-splitted-into-smaller-ones/815/4 , and an example of what to be careful about with range clears on https://forums.foundationdb.org/t/used-disk-space-dramatically-increases-while-sum-of-key-value-sizes-is-constant/644
> 
> 
>> On Mar 27, 2019, at 11:07 AM, Ilya Khlopotov <ii...@apache.org> wrote:
>> What if FDB would support a list type for a value and would have an atomic operation to add the value to the list if it is missing. In this case we could store the data we need as follows (under separate subspace TBD).
>> VersionStamp = (DbName, EventType)
>> DbName = [versionstamps]
>> ...
>> The question is how we would implement atomic addition of a value to a list. 
> 
> If I’ve understood your proposal correctly, I think it sounds easier to just use APPEND_IF_FITS ?
> 
> https://apple.github.io/foundationdb/javadoc/index.html?com/apple/foundationdb/MutationType.html
>