You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by Hitesh Khamesra <hi...@yahoo.com> on 2017/02/14 22:36:20 UTC
GeodeRedisAdapter improvments/feedback
Current GeodeRedisAdapter implementation is based on https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal.
We are looking for some feedback on Redis commands and their mapping to geode region.
1. Redis Type String
a. Usage Set k1 v1
b. Current implementation creates "STRING_REGION" geode-partition-region upfront
c. This k1/v1 are geode-region key/value
d. Any feedback?
2. List Type
a. usage "rpush mylist A"
b. Current implementation maps each list to geode-partition-region(i.e. mylist is geode-partition-region); with the ability to get item from head/tail
c. Feedback/vote
-- List type operation at region-entry level;
-- region-key = "mylist"
-- region-value = Arraylist (will support all redis list ops)
d. Feedback/vote: both behavior is desirable
3. Hashes
a. this represents field-value or java bean object
b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
c. Current implementation maps each hashes to geode-partition-region(i.e. user1000 is geode-partition-region)
d. Feedback/vote
-- Should we map hashes to region-entry
-- region-key = user1000
-- region-value = map
-- This will provide java bean sort to behaviour with 10s of field-value
-- Personally I would prefer this..
e. Feedback/vote: both behaviour is desirable
4. Sets
a. This represents unique keys in set
b. usage "sadd myset 1 2 3"
c. Current implementation maps each sadd to geode-partition-region(i.e. myset is geode-partition-region)
d. Feedback/vote
-- Should we map set to region-entry
-- region-key = myset
-- region-value = Hashset
e. Feedback/vote: both behaviour is desirable
5. SortedSets
a. This represents unique keys in set with score (usecase Query top-10)
b. usage "zadd hackers 1940 "Alan Kay""
c. Current implementation maps each zadd to geode-partition-region(i.e. hackers is geode-partition-region)
d. Feedback/vote
-- Should we map set to region-entry
-- region-key = hackers
-- region-value = Sorted Hashset
e. Feedback/vote: both behaviour is desirable
6. HyperLogLogs
a. A HyperLogLog is a probabilistic data structure used in order to count unique things (technically this is referred to estimating the cardinality of a set).
b. usage "pfadd hll a b c d"
c. Current implementation creates "HLL_REGION" geode-partition-region upfront
d. hll becomes region-key and value is HLL object
e. any feedback?
7. Default config for geode-region (vote)
a. partition region
b. 1 redundant copy
c. Persistence
d. Eviction
e. Expiration
f. ?
8. It seems; redis knows type(list, hashes, string ,set ..) of each key. Thus for each operation we need to make sure type of key. In current implementation we have different region for each redis type. Thus we have another region(metaTypeRegion) which keeps type for each key. This makes any operation in geode slow as it needs to verify that type. For instance, creating new key need to make sure its already there or not. Whether we should allow type change or not.
a. Feedback/vote
-- type change of key
-- Can we allow two key with same name but two differnt type (as it will endup in two different geode-region)
String type "key1" in string region
HLL type "key1" in HLL region
b. any other feedback
9. Transactions:
a. we will not support transaction in redisAdapter as geode transaction are limited to single node.
b. feedback?
10. Redis COMMAND (https://redis.io/commands/command)
a. should we implement this "COMMAND" ?
11. Any other redis command we should consider?
Thanks.Hitesh
Re: GeodeRedisAdapter improvments/feedback
Posted by Hitesh Khamesra <hi...@yahoo.com.INVALID>.
How's going on. What are you working these days.
Sent from Yahoo Mail on Android
On Tue, Feb 14, 2017 at 4:16 PM, William Markito Oliveira<wi...@gmail.com> wrote: Definitely not by asking in the middle of on-going discussion threads.
Instructions: http://bfy.tw/A5ub :)
Sent from my iPhone
> On Feb 14, 2017, at 6:38 PM, Michael Vos <mv...@pivotal.io> wrote:
>
> How do I unsubscribe from this email list?
>
> Thank you,
>
> Michael Vos
> Strategic Partnerships
> 310-804-7223 | mvos@pivotal.io
>
>
>
>
>
>> On Feb 14, 2017, at 3:37 PM, Dan Smith <ds...@pivotal.io> wrote:
>>
>> I also had a hard time reading this. It sounds like the tl;dr is that your are proposing storing each redis collection in a single region entry, rather than a a partition region? I guess the question is whether users will have a few very large collections that need to be partitioned, or lots and lots of really tiny collections. I'm not quite sure what the more common use case is.
>>
>> -Dan
>>
>>> On Tue, Feb 14, 2017 at 3:22 PM, Swapnil Bawaskar <sb...@pivotal.io> wrote:
>>> The Redis adapter was designed so that we can scale all the Redis data structures horizontally. If you bring the data structures to region entry level, there is no reason for anyone to use our implementation over Redis.
>>>
>>>> On Tue, Feb 14, 2017 at 3:15 PM Jason Huynh <jh...@pivotal.io> wrote:
>>>> Hi Hitesh,
>>>>
>>>> Not sure about everyone else, but I had a hard time reading this, however I think I figured out what you were describing... the only part I still am unsure about is Feedback/vote: both behaviour is desirable. Do you mean you want feedback and voting on whether both behaviors are desired? As in old implementation and new implementation?
>>>>
>>>> 2,3,4) The new implementation would mean all the data for a specific data structure is contained in a single bucket. So the individual data structures are not quite scalable. How would you allow scaling of a single data structure?
>>>>
>>>> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
>>>> In what format do you want the feedback Hitesh? For now I’ll just comment:
>>>>
>>>> 1. Redis Type String
>>>> No comments except that a future Geode value-add would be to extend the Jedis client so that the K/V’s are not compressed. In this way OQL and CQ will work. The tradeoff of this is that the data cannot be read by a native redis client but for Geode users it’s great. Call the new client Geodis.
>>>>
>>>> 2. List/ Hash/ Set/ SortedSet
>>>> Creating a separate region for each creates a constraint that the keys are limited to the characters for region names, which are A-z/0-9/ - and _. Everything else is out. Redis users might start asking questions why their list named ++^^/## throws an error. Your suggestion to make it a key rather than a region solves this. Furthermore, creating a new region every time a new Redis collection is created is going to be slow. I’m not sure why a region was created but I’m sure it made sense to the developer at the time.
>>>>
>>>> 7. Default Config
>>>> Can’t we configure a gfsh option to default to the region types we want? Customer A will want PARTITION but Customer B will want PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT that makes _all_ Redis regions of that type?
>>>>
>>>>
>>>>
>>>> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hi...@yahoo.com>> wrote:
>>>>
>>>> Current GeodeRedisAdapter implementation is based on https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal.
>>>> We are looking for some feedback on Redis commands and their mapping to geode region.
>>>>
>>>> 1. Redis Type String
>>>> a. Usage Set k1 v1
>>>> b. Current implementation creates "STRING_REGION" geode-partition-region upfront
>>>> c. This k1/v1 are geode-region key/value
>>>> d. Any feedback?
>>>>
>>>> 2. List Type
>>>> a. usage "rpush mylist A"
>>>> b. Current implementation maps each list to geode-partition-region(i.e. mylist is geode-partition-region); with the ability to get item from head/tail
>>>> c. Feedback/vote
>>>> -- List type operation at region-entry level;
>>>> -- region-key = "mylist"
>>>> -- region-value = Arraylist (will support all redis list ops)
>>>> d. Feedback/vote: both behavior is desirable
>>>>
>>>>
>>>> 3. Hashes
>>>> a. this represents field-value or java bean object
>>>> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
>>>> c. Current implementation maps each hashes to geode-partition-region(i.e. user1000 is geode-partition-region)
>>>> d. Feedback/vote
>>>> -- Should we map hashes to region-entry
>>>> -- region-key = user1000
>>>> -- region-value = map
>>>> -- This will provide java bean sort to behaviour with 10s of field-value
>>>> -- Personally I would prefer this..
>>>> e. Feedback/vote: both behaviour is desirable
>>>>
>>>> 4. Sets
>>>> a. This represents unique keys in set
>>>> b. usage "sadd myset 1 2 3"
>>>> c. Current implementation maps each sadd to geode-partition-region(i.e. myset is geode-partition-region)
>>>> d. Feedback/vote
>>>> -- Should we map set to region-entry
>>>> -- region-key = myset
>>>> -- region-value = Hashset
>>>> e. Feedback/vote: both behaviour is desirable
>>>>
>>>> 5. SortedSets
>>>> a. This represents unique keys in set with score (usecase Query top-10)
>>>> b. usage "zadd hackers 1940 "Alan Kay""
>>>> c. Current implementation maps each zadd to geode-partition-region(i.e. hackers is geode-partition-region)
>>>> d. Feedback/vote
>>>> -- Should we map set to region-entry
>>>> -- region-key = hackers
>>>> -- region-value = Sorted Hashset
>>>> e. Feedback/vote: both behaviour is desirable
>>>>
>>>> 6. HyperLogLogs
>>>> a. A HyperLogLog is a probabilistic data structure used in order to count unique things (technically this is referred to estimating the cardinality of a set).
>>>> b. usage "pfadd hll a b c d"
>>>> c. Current implementation creates "HLL_REGION" geode-partition-region upfront
>>>> d. hll becomes region-key and value is HLL object
>>>> e. any feedback?
>>>>
>>>> 7. Default config for geode-region (vote)
>>>> a. partition region
>>>> b. 1 redundant copy
>>>> c. Persistence
>>>> d. Eviction
>>>> e. Expiration
>>>> f. ?
>>>>
>>>> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key. Thus for each operation we need to make sure type of key. In current implementation we have different region for each redis type. Thus we have another region(metaTypeRegion) which keeps type for each key. This makes any operation in geode slow as it needs to verify that type. For instance, creating new key need to make sure its already there or not. Whether we should allow type change or not.
>>>> a. Feedback/vote
>>>> -- type change of key
>>>> -- Can we allow two key with same name but two differnt type (as it will endup in two different geode-region)
>>>> String type "key1" in string region
>>>> HLL type "key1" in HLL region
>>>> b. any other feedback
>>>>
>>>> 9. Transactions:
>>>> a. we will not support transaction in redisAdapter as geode transaction are limited to single node.
>>>> b. feedback?
>>>>
>>>> 10. Redis COMMAND (https://redis.io/commands/command)
>>>> a. should we implement this "COMMAND" ?
>>>>
>>>> 11. Any other redis command we should consider?
>>>>
>>>>
>>>> Thanks.
>>>> Hitesh
>>>>
>>>>
>>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by William Markito Oliveira <wi...@gmail.com>.
Definitely not by asking in the middle of on-going discussion threads.
Instructions: http://bfy.tw/A5ub :)
Sent from my iPhone
> On Feb 14, 2017, at 6:38 PM, Michael Vos <mv...@pivotal.io> wrote:
>
> How do I unsubscribe from this email list?
>
> Thank you,
>
> Michael Vos
> Strategic Partnerships
> 310-804-7223 | mvos@pivotal.io
>
>
>
>
>
>> On Feb 14, 2017, at 3:37 PM, Dan Smith <ds...@pivotal.io> wrote:
>>
>> I also had a hard time reading this. It sounds like the tl;dr is that your are proposing storing each redis collection in a single region entry, rather than a a partition region? I guess the question is whether users will have a few very large collections that need to be partitioned, or lots and lots of really tiny collections. I'm not quite sure what the more common use case is.
>>
>> -Dan
>>
>>> On Tue, Feb 14, 2017 at 3:22 PM, Swapnil Bawaskar <sb...@pivotal.io> wrote:
>>> The Redis adapter was designed so that we can scale all the Redis data structures horizontally. If you bring the data structures to region entry level, there is no reason for anyone to use our implementation over Redis.
>>>
>>>> On Tue, Feb 14, 2017 at 3:15 PM Jason Huynh <jh...@pivotal.io> wrote:
>>>> Hi Hitesh,
>>>>
>>>> Not sure about everyone else, but I had a hard time reading this, however I think I figured out what you were describing... the only part I still am unsure about is Feedback/vote: both behaviour is desirable. Do you mean you want feedback and voting on whether both behaviors are desired? As in old implementation and new implementation?
>>>>
>>>> 2,3,4) The new implementation would mean all the data for a specific data structure is contained in a single bucket. So the individual data structures are not quite scalable. How would you allow scaling of a single data structure?
>>>>
>>>> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
>>>> In what format do you want the feedback Hitesh? For now I’ll just comment:
>>>>
>>>> 1. Redis Type String
>>>> No comments except that a future Geode value-add would be to extend the Jedis client so that the K/V’s are not compressed. In this way OQL and CQ will work. The tradeoff of this is that the data cannot be read by a native redis client but for Geode users it’s great. Call the new client Geodis.
>>>>
>>>> 2. List/ Hash/ Set/ SortedSet
>>>> Creating a separate region for each creates a constraint that the keys are limited to the characters for region names, which are A-z/0-9/ - and _. Everything else is out. Redis users might start asking questions why their list named ++^^/## throws an error. Your suggestion to make it a key rather than a region solves this. Furthermore, creating a new region every time a new Redis collection is created is going to be slow. I’m not sure why a region was created but I’m sure it made sense to the developer at the time.
>>>>
>>>> 7. Default Config
>>>> Can’t we configure a gfsh option to default to the region types we want? Customer A will want PARTITION but Customer B will want PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT that makes _all_ Redis regions of that type?
>>>>
>>>>
>>>>
>>>> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hi...@yahoo.com>> wrote:
>>>>
>>>> Current GeodeRedisAdapter implementation is based on https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal.
>>>> We are looking for some feedback on Redis commands and their mapping to geode region.
>>>>
>>>> 1. Redis Type String
>>>> a. Usage Set k1 v1
>>>> b. Current implementation creates "STRING_REGION" geode-partition-region upfront
>>>> c. This k1/v1 are geode-region key/value
>>>> d. Any feedback?
>>>>
>>>> 2. List Type
>>>> a. usage "rpush mylist A"
>>>> b. Current implementation maps each list to geode-partition-region(i.e. mylist is geode-partition-region); with the ability to get item from head/tail
>>>> c. Feedback/vote
>>>> -- List type operation at region-entry level;
>>>> -- region-key = "mylist"
>>>> -- region-value = Arraylist (will support all redis list ops)
>>>> d. Feedback/vote: both behavior is desirable
>>>>
>>>>
>>>> 3. Hashes
>>>> a. this represents field-value or java bean object
>>>> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
>>>> c. Current implementation maps each hashes to geode-partition-region(i.e. user1000 is geode-partition-region)
>>>> d. Feedback/vote
>>>> -- Should we map hashes to region-entry
>>>> -- region-key = user1000
>>>> -- region-value = map
>>>> -- This will provide java bean sort to behaviour with 10s of field-value
>>>> -- Personally I would prefer this..
>>>> e. Feedback/vote: both behaviour is desirable
>>>>
>>>> 4. Sets
>>>> a. This represents unique keys in set
>>>> b. usage "sadd myset 1 2 3"
>>>> c. Current implementation maps each sadd to geode-partition-region(i.e. myset is geode-partition-region)
>>>> d. Feedback/vote
>>>> -- Should we map set to region-entry
>>>> -- region-key = myset
>>>> -- region-value = Hashset
>>>> e. Feedback/vote: both behaviour is desirable
>>>>
>>>> 5. SortedSets
>>>> a. This represents unique keys in set with score (usecase Query top-10)
>>>> b. usage "zadd hackers 1940 "Alan Kay""
>>>> c. Current implementation maps each zadd to geode-partition-region(i.e. hackers is geode-partition-region)
>>>> d. Feedback/vote
>>>> -- Should we map set to region-entry
>>>> -- region-key = hackers
>>>> -- region-value = Sorted Hashset
>>>> e. Feedback/vote: both behaviour is desirable
>>>>
>>>> 6. HyperLogLogs
>>>> a. A HyperLogLog is a probabilistic data structure used in order to count unique things (technically this is referred to estimating the cardinality of a set).
>>>> b. usage "pfadd hll a b c d"
>>>> c. Current implementation creates "HLL_REGION" geode-partition-region upfront
>>>> d. hll becomes region-key and value is HLL object
>>>> e. any feedback?
>>>>
>>>> 7. Default config for geode-region (vote)
>>>> a. partition region
>>>> b. 1 redundant copy
>>>> c. Persistence
>>>> d. Eviction
>>>> e. Expiration
>>>> f. ?
>>>>
>>>> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key. Thus for each operation we need to make sure type of key. In current implementation we have different region for each redis type. Thus we have another region(metaTypeRegion) which keeps type for each key. This makes any operation in geode slow as it needs to verify that type. For instance, creating new key need to make sure its already there or not. Whether we should allow type change or not.
>>>> a. Feedback/vote
>>>> -- type change of key
>>>> -- Can we allow two key with same name but two differnt type (as it will endup in two different geode-region)
>>>> String type "key1" in string region
>>>> HLL type "key1" in HLL region
>>>> b. any other feedback
>>>>
>>>> 9. Transactions:
>>>> a. we will not support transaction in redisAdapter as geode transaction are limited to single node.
>>>> b. feedback?
>>>>
>>>> 10. Redis COMMAND (https://redis.io/commands/command)
>>>> a. should we implement this "COMMAND" ?
>>>>
>>>> 11. Any other redis command we should consider?
>>>>
>>>>
>>>> Thanks.
>>>> Hitesh
>>>>
>>>>
>>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by William Markito Oliveira <wi...@gmail.com>.
Definitely not by asking in the middle of on-going discussion threads.
Instructions: http://bfy.tw/A5ub :)
Sent from my iPhone
> On Feb 14, 2017, at 6:38 PM, Michael Vos <mv...@pivotal.io> wrote:
>
> How do I unsubscribe from this email list?
>
> Thank you,
>
> Michael Vos
> Strategic Partnerships
> 310-804-7223 | mvos@pivotal.io
>
>
>
>
>
>> On Feb 14, 2017, at 3:37 PM, Dan Smith <ds...@pivotal.io> wrote:
>>
>> I also had a hard time reading this. It sounds like the tl;dr is that your are proposing storing each redis collection in a single region entry, rather than a a partition region? I guess the question is whether users will have a few very large collections that need to be partitioned, or lots and lots of really tiny collections. I'm not quite sure what the more common use case is.
>>
>> -Dan
>>
>>> On Tue, Feb 14, 2017 at 3:22 PM, Swapnil Bawaskar <sb...@pivotal.io> wrote:
>>> The Redis adapter was designed so that we can scale all the Redis data structures horizontally. If you bring the data structures to region entry level, there is no reason for anyone to use our implementation over Redis.
>>>
>>>> On Tue, Feb 14, 2017 at 3:15 PM Jason Huynh <jh...@pivotal.io> wrote:
>>>> Hi Hitesh,
>>>>
>>>> Not sure about everyone else, but I had a hard time reading this, however I think I figured out what you were describing... the only part I still am unsure about is Feedback/vote: both behaviour is desirable. Do you mean you want feedback and voting on whether both behaviors are desired? As in old implementation and new implementation?
>>>>
>>>> 2,3,4) The new implementation would mean all the data for a specific data structure is contained in a single bucket. So the individual data structures are not quite scalable. How would you allow scaling of a single data structure?
>>>>
>>>> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
>>>> In what format do you want the feedback Hitesh? For now I’ll just comment:
>>>>
>>>> 1. Redis Type String
>>>> No comments except that a future Geode value-add would be to extend the Jedis client so that the K/V’s are not compressed. In this way OQL and CQ will work. The tradeoff of this is that the data cannot be read by a native redis client but for Geode users it’s great. Call the new client Geodis.
>>>>
>>>> 2. List/ Hash/ Set/ SortedSet
>>>> Creating a separate region for each creates a constraint that the keys are limited to the characters for region names, which are A-z/0-9/ - and _. Everything else is out. Redis users might start asking questions why their list named ++^^/## throws an error. Your suggestion to make it a key rather than a region solves this. Furthermore, creating a new region every time a new Redis collection is created is going to be slow. I’m not sure why a region was created but I’m sure it made sense to the developer at the time.
>>>>
>>>> 7. Default Config
>>>> Can’t we configure a gfsh option to default to the region types we want? Customer A will want PARTITION but Customer B will want PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT that makes _all_ Redis regions of that type?
>>>>
>>>>
>>>>
>>>> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hi...@yahoo.com>> wrote:
>>>>
>>>> Current GeodeRedisAdapter implementation is based on https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal.
>>>> We are looking for some feedback on Redis commands and their mapping to geode region.
>>>>
>>>> 1. Redis Type String
>>>> a. Usage Set k1 v1
>>>> b. Current implementation creates "STRING_REGION" geode-partition-region upfront
>>>> c. This k1/v1 are geode-region key/value
>>>> d. Any feedback?
>>>>
>>>> 2. List Type
>>>> a. usage "rpush mylist A"
>>>> b. Current implementation maps each list to geode-partition-region(i.e. mylist is geode-partition-region); with the ability to get item from head/tail
>>>> c. Feedback/vote
>>>> -- List type operation at region-entry level;
>>>> -- region-key = "mylist"
>>>> -- region-value = Arraylist (will support all redis list ops)
>>>> d. Feedback/vote: both behavior is desirable
>>>>
>>>>
>>>> 3. Hashes
>>>> a. this represents field-value or java bean object
>>>> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
>>>> c. Current implementation maps each hashes to geode-partition-region(i.e. user1000 is geode-partition-region)
>>>> d. Feedback/vote
>>>> -- Should we map hashes to region-entry
>>>> -- region-key = user1000
>>>> -- region-value = map
>>>> -- This will provide java bean sort to behaviour with 10s of field-value
>>>> -- Personally I would prefer this..
>>>> e. Feedback/vote: both behaviour is desirable
>>>>
>>>> 4. Sets
>>>> a. This represents unique keys in set
>>>> b. usage "sadd myset 1 2 3"
>>>> c. Current implementation maps each sadd to geode-partition-region(i.e. myset is geode-partition-region)
>>>> d. Feedback/vote
>>>> -- Should we map set to region-entry
>>>> -- region-key = myset
>>>> -- region-value = Hashset
>>>> e. Feedback/vote: both behaviour is desirable
>>>>
>>>> 5. SortedSets
>>>> a. This represents unique keys in set with score (usecase Query top-10)
>>>> b. usage "zadd hackers 1940 "Alan Kay""
>>>> c. Current implementation maps each zadd to geode-partition-region(i.e. hackers is geode-partition-region)
>>>> d. Feedback/vote
>>>> -- Should we map set to region-entry
>>>> -- region-key = hackers
>>>> -- region-value = Sorted Hashset
>>>> e. Feedback/vote: both behaviour is desirable
>>>>
>>>> 6. HyperLogLogs
>>>> a. A HyperLogLog is a probabilistic data structure used in order to count unique things (technically this is referred to estimating the cardinality of a set).
>>>> b. usage "pfadd hll a b c d"
>>>> c. Current implementation creates "HLL_REGION" geode-partition-region upfront
>>>> d. hll becomes region-key and value is HLL object
>>>> e. any feedback?
>>>>
>>>> 7. Default config for geode-region (vote)
>>>> a. partition region
>>>> b. 1 redundant copy
>>>> c. Persistence
>>>> d. Eviction
>>>> e. Expiration
>>>> f. ?
>>>>
>>>> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key. Thus for each operation we need to make sure type of key. In current implementation we have different region for each redis type. Thus we have another region(metaTypeRegion) which keeps type for each key. This makes any operation in geode slow as it needs to verify that type. For instance, creating new key need to make sure its already there or not. Whether we should allow type change or not.
>>>> a. Feedback/vote
>>>> -- type change of key
>>>> -- Can we allow two key with same name but two differnt type (as it will endup in two different geode-region)
>>>> String type "key1" in string region
>>>> HLL type "key1" in HLL region
>>>> b. any other feedback
>>>>
>>>> 9. Transactions:
>>>> a. we will not support transaction in redisAdapter as geode transaction are limited to single node.
>>>> b. feedback?
>>>>
>>>> 10. Redis COMMAND (https://redis.io/commands/command)
>>>> a. should we implement this "COMMAND" ?
>>>>
>>>> 11. Any other redis command we should consider?
>>>>
>>>>
>>>> Thanks.
>>>> Hitesh
>>>>
>>>>
>>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Michael Vos <mv...@pivotal.io>.
How do I unsubscribe from this email list?
Thank you,
Michael Vos
Strategic Partnerships
310-804-7223 | mvos@pivotal.io
> On Feb 14, 2017, at 3:37 PM, Dan Smith <ds...@pivotal.io> wrote:
>
> I also had a hard time reading this. It sounds like the tl;dr is that your are proposing storing each redis collection in a single region entry, rather than a a partition region? I guess the question is whether users will have a few very large collections that need to be partitioned, or lots and lots of really tiny collections. I'm not quite sure what the more common use case is.
>
> -Dan
>
> On Tue, Feb 14, 2017 at 3:22 PM, Swapnil Bawaskar <sbawaskar@pivotal.io <ma...@pivotal.io>> wrote:
> The Redis adapter was designed so that we can scale all the Redis data structures horizontally. If you bring the data structures to region entry level, there is no reason for anyone to use our implementation over Redis.
>
> On Tue, Feb 14, 2017 at 3:15 PM Jason Huynh <jhuynh@pivotal.io <ma...@pivotal.io>> wrote:
> Hi Hitesh,
>
> Not sure about everyone else, but I had a hard time reading this, however I think I figured out what you were describing... the only part I still am unsure about is Feedback/vote: both behaviour is desirable. Do you mean you want feedback and voting on whether both behaviors are desired? As in old implementation and new implementation?
>
> 2,3,4) The new implementation would mean all the data for a specific data structure is contained in a single bucket. So the individual data structures are not quite scalable. How would you allow scaling of a single data structure?
>
> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <TheRealWes@outlook.com <ma...@outlook.com>> wrote:
> In what format do you want the feedback Hitesh? For now I’ll just comment:
>
> 1. Redis Type String
> No comments except that a future Geode value-add would be to extend the Jedis client so that the K/V’s are not compressed. In this way OQL and CQ will work. The tradeoff of this is that the data cannot be read by a native redis client but for Geode users it’s great. Call the new client Geodis.
>
> 2. List/ Hash/ Set/ SortedSet
> Creating a separate region for each creates a constraint that the keys are limited to the characters for region names, which are A-z/0-9/ - and _. Everything else is out. Redis users might start asking questions why their list named ++^^/## throws an error. Your suggestion to make it a key rather than a region solves this. Furthermore, creating a new region every time a new Redis collection is created is going to be slow. I’m not sure why a region was created but I’m sure it made sense to the developer at the time.
>
> 7. Default Config
> Can’t we configure a gfsh option to default to the region types we want? Customer A will want PARTITION but Customer B will want PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT that makes _all_ Redis regions of that type?
>
>
>
> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com <ma...@yahoo.com><mailto:hiteshk25@yahoo.com <ma...@yahoo.com>>> wrote:
>
> Current GeodeRedisAdapter implementation is based on https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal <https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal>.
> We are looking for some feedback on Redis commands and their mapping to geode region.
>
> 1. Redis Type String
> a. Usage Set k1 v1
> b. Current implementation creates "STRING_REGION" geode-partition-region upfront
> c. This k1/v1 are geode-region key/value
> d. Any feedback?
>
> 2. List Type
> a. usage "rpush mylist A"
> b. Current implementation maps each list to geode-partition-region(i.e. mylist is geode-partition-region); with the ability to get item from head/tail
> c. Feedback/vote
> -- List type operation at region-entry level;
> -- region-key = "mylist"
> -- region-value = Arraylist (will support all redis list ops)
> d. Feedback/vote: both behavior is desirable
>
>
> 3. Hashes
> a. this represents field-value or java bean object
> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> c. Current implementation maps each hashes to geode-partition-region(i.e. user1000 is geode-partition-region)
> d. Feedback/vote
> -- Should we map hashes to region-entry
> -- region-key = user1000
> -- region-value = map
> -- This will provide java bean sort to behaviour with 10s of field-value
> -- Personally I would prefer this..
> e. Feedback/vote: both behaviour is desirable
>
> 4. Sets
> a. This represents unique keys in set
> b. usage "sadd myset 1 2 3"
> c. Current implementation maps each sadd to geode-partition-region(i.e. myset is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = myset
> -- region-value = Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 5. SortedSets
> a. This represents unique keys in set with score (usecase Query top-10)
> b. usage "zadd hackers 1940 "Alan Kay""
> c. Current implementation maps each zadd to geode-partition-region(i.e. hackers is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = hackers
> -- region-value = Sorted Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 6. HyperLogLogs
> a. A HyperLogLog is a probabilistic data structure used in order to count unique things (technically this is referred to estimating the cardinality of a set).
> b. usage "pfadd hll a b c d"
> c. Current implementation creates "HLL_REGION" geode-partition-region upfront
> d. hll becomes region-key and value is HLL object
> e. any feedback?
>
> 7. Default config for geode-region (vote)
> a. partition region
> b. 1 redundant copy
> c. Persistence
> d. Eviction
> e. Expiration
> f. ?
>
> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key. Thus for each operation we need to make sure type of key. In current implementation we have different region for each redis type. Thus we have another region(metaTypeRegion) which keeps type for each key. This makes any operation in geode slow as it needs to verify that type. For instance, creating new key need to make sure its already there or not. Whether we should allow type change or not.
> a. Feedback/vote
> -- type change of key
> -- Can we allow two key with same name but two differnt type (as it will endup in two different geode-region)
> String type "key1" in string region
> HLL type "key1" in HLL region
> b. any other feedback
>
> 9. Transactions:
> a. we will not support transaction in redisAdapter as geode transaction are limited to single node.
> b. feedback?
>
> 10. Redis COMMAND (https://redis.io/commands/command <https://redis.io/commands/command>)
> a. should we implement this "COMMAND" ?
>
> 11. Any other redis command we should consider?
>
>
> Thanks.
> Hitesh
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Michael Vos <mv...@pivotal.io>.
How do I unsubscribe from this email list?
Thank you,
Michael Vos
Strategic Partnerships
310-804-7223 | mvos@pivotal.io
> On Feb 14, 2017, at 3:37 PM, Dan Smith <ds...@pivotal.io> wrote:
>
> I also had a hard time reading this. It sounds like the tl;dr is that your are proposing storing each redis collection in a single region entry, rather than a a partition region? I guess the question is whether users will have a few very large collections that need to be partitioned, or lots and lots of really tiny collections. I'm not quite sure what the more common use case is.
>
> -Dan
>
> On Tue, Feb 14, 2017 at 3:22 PM, Swapnil Bawaskar <sbawaskar@pivotal.io <ma...@pivotal.io>> wrote:
> The Redis adapter was designed so that we can scale all the Redis data structures horizontally. If you bring the data structures to region entry level, there is no reason for anyone to use our implementation over Redis.
>
> On Tue, Feb 14, 2017 at 3:15 PM Jason Huynh <jhuynh@pivotal.io <ma...@pivotal.io>> wrote:
> Hi Hitesh,
>
> Not sure about everyone else, but I had a hard time reading this, however I think I figured out what you were describing... the only part I still am unsure about is Feedback/vote: both behaviour is desirable. Do you mean you want feedback and voting on whether both behaviors are desired? As in old implementation and new implementation?
>
> 2,3,4) The new implementation would mean all the data for a specific data structure is contained in a single bucket. So the individual data structures are not quite scalable. How would you allow scaling of a single data structure?
>
> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <TheRealWes@outlook.com <ma...@outlook.com>> wrote:
> In what format do you want the feedback Hitesh? For now I’ll just comment:
>
> 1. Redis Type String
> No comments except that a future Geode value-add would be to extend the Jedis client so that the K/V’s are not compressed. In this way OQL and CQ will work. The tradeoff of this is that the data cannot be read by a native redis client but for Geode users it’s great. Call the new client Geodis.
>
> 2. List/ Hash/ Set/ SortedSet
> Creating a separate region for each creates a constraint that the keys are limited to the characters for region names, which are A-z/0-9/ - and _. Everything else is out. Redis users might start asking questions why their list named ++^^/## throws an error. Your suggestion to make it a key rather than a region solves this. Furthermore, creating a new region every time a new Redis collection is created is going to be slow. I’m not sure why a region was created but I’m sure it made sense to the developer at the time.
>
> 7. Default Config
> Can’t we configure a gfsh option to default to the region types we want? Customer A will want PARTITION but Customer B will want PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT that makes _all_ Redis regions of that type?
>
>
>
> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com <ma...@yahoo.com><mailto:hiteshk25@yahoo.com <ma...@yahoo.com>>> wrote:
>
> Current GeodeRedisAdapter implementation is based on https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal <https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal>.
> We are looking for some feedback on Redis commands and their mapping to geode region.
>
> 1. Redis Type String
> a. Usage Set k1 v1
> b. Current implementation creates "STRING_REGION" geode-partition-region upfront
> c. This k1/v1 are geode-region key/value
> d. Any feedback?
>
> 2. List Type
> a. usage "rpush mylist A"
> b. Current implementation maps each list to geode-partition-region(i.e. mylist is geode-partition-region); with the ability to get item from head/tail
> c. Feedback/vote
> -- List type operation at region-entry level;
> -- region-key = "mylist"
> -- region-value = Arraylist (will support all redis list ops)
> d. Feedback/vote: both behavior is desirable
>
>
> 3. Hashes
> a. this represents field-value or java bean object
> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> c. Current implementation maps each hashes to geode-partition-region(i.e. user1000 is geode-partition-region)
> d. Feedback/vote
> -- Should we map hashes to region-entry
> -- region-key = user1000
> -- region-value = map
> -- This will provide java bean sort to behaviour with 10s of field-value
> -- Personally I would prefer this..
> e. Feedback/vote: both behaviour is desirable
>
> 4. Sets
> a. This represents unique keys in set
> b. usage "sadd myset 1 2 3"
> c. Current implementation maps each sadd to geode-partition-region(i.e. myset is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = myset
> -- region-value = Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 5. SortedSets
> a. This represents unique keys in set with score (usecase Query top-10)
> b. usage "zadd hackers 1940 "Alan Kay""
> c. Current implementation maps each zadd to geode-partition-region(i.e. hackers is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = hackers
> -- region-value = Sorted Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 6. HyperLogLogs
> a. A HyperLogLog is a probabilistic data structure used in order to count unique things (technically this is referred to estimating the cardinality of a set).
> b. usage "pfadd hll a b c d"
> c. Current implementation creates "HLL_REGION" geode-partition-region upfront
> d. hll becomes region-key and value is HLL object
> e. any feedback?
>
> 7. Default config for geode-region (vote)
> a. partition region
> b. 1 redundant copy
> c. Persistence
> d. Eviction
> e. Expiration
> f. ?
>
> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key. Thus for each operation we need to make sure type of key. In current implementation we have different region for each redis type. Thus we have another region(metaTypeRegion) which keeps type for each key. This makes any operation in geode slow as it needs to verify that type. For instance, creating new key need to make sure its already there or not. Whether we should allow type change or not.
> a. Feedback/vote
> -- type change of key
> -- Can we allow two key with same name but two differnt type (as it will endup in two different geode-region)
> String type "key1" in string region
> HLL type "key1" in HLL region
> b. any other feedback
>
> 9. Transactions:
> a. we will not support transaction in redisAdapter as geode transaction are limited to single node.
> b. feedback?
>
> 10. Redis COMMAND (https://redis.io/commands/command <https://redis.io/commands/command>)
> a. should we implement this "COMMAND" ?
>
> 11. Any other redis command we should consider?
>
>
> Thanks.
> Hitesh
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Dan Smith <ds...@pivotal.io>.
I also had a hard time reading this. It sounds like the tl;dr is that your
are proposing storing each redis collection in a single region entry,
rather than a a partition region? I guess the question is whether users
will have a few very large collections that need to be partitioned, or lots
and lots of really tiny collections. I'm not quite sure what the more
common use case is.
-Dan
On Tue, Feb 14, 2017 at 3:22 PM, Swapnil Bawaskar <sb...@pivotal.io>
wrote:
> The Redis adapter was designed so that we can scale all the Redis data
> structures horizontally. If you bring the data structures to region entry
> level, there is no reason for anyone to use our implementation over Redis.
>
> On Tue, Feb 14, 2017 at 3:15 PM Jason Huynh <jh...@pivotal.io> wrote:
>
>> Hi Hitesh,
>>
>> Not sure about everyone else, but I had a hard time reading this,
>> however I think I figured out what you were describing... the only part I
>> still am unsure about is Feedback/vote: both behaviour is desirable. Do
>> you mean you want feedback and voting on whether both behaviors are
>> desired? As in old implementation and new implementation?
>>
>> 2,3,4) The new implementation would mean all the data for a specific
>> data structure is contained in a single bucket. So the individual data
>> structures are not quite scalable. How would you allow scaling of a single
>> data structure?
>>
>> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
>>
>> In what format do you want the feedback Hitesh? For now I’ll just
>> comment:
>>
>> 1. Redis Type String
>> No comments except that a future Geode value-add would be to extend the
>> Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
>> will work. The tradeoff of this is that the data cannot be read by a
>> native redis client but for Geode users it’s great. Call the new client
>> Geodis.
>>
>> 2. List/ Hash/ Set/ SortedSet
>> Creating a separate region for each creates a constraint that the keys
>> are limited to the characters for region names, which are A-z/0-9/ - and
>> _. Everything else is out. Redis users might start asking questions why
>> their list named ++^^/## throws an error. Your suggestion to make it a key
>> rather than a region solves this. Furthermore, creating a new region every
>> time a new Redis collection is created is going to be slow. I’m not sure
>> why a region was created but I’m sure it made sense to the developer at the
>> time.
>>
>> 7. Default Config
>> Can’t we configure a gfsh option to default to the region types we want?
>> Customer A will want PARTITION but Customer B will want PARTITION_REDUNDANT_EXPIRATION_PERSISTENT.
>> I wonder if we can consider a geode> create region —redisType=PARTITION_
>> REDUNDANT_EXPIRATION_PERSISTENT that makes _all_ Redis regions of that
>> type?
>>
>>
>>
>> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com<mailto:
>> hiteshk25@yahoo.com>> wrote:
>>
>> Current GeodeRedisAdapter implementation is based on
>> https://cwiki.apache.org/confluence/display/GEODE/
>> Geode+Redis+Adapter+Proposal.
>> We are looking for some feedback on Redis commands and their mapping to
>> geode region.
>>
>> 1. Redis Type String
>> a. Usage Set k1 v1
>> b. Current implementation creates "STRING_REGION"
>> geode-partition-region upfront
>> c. This k1/v1 are geode-region key/value
>> d. Any feedback?
>>
>> 2. List Type
>> a. usage "rpush mylist A"
>> b. Current implementation maps each list to geode-partition-region(i.e.
>> mylist is geode-partition-region); with the ability to get item from
>> head/tail
>> c. Feedback/vote
>> -- List type operation at region-entry level;
>> -- region-key = "mylist"
>> -- region-value = Arraylist (will support all redis list ops)
>> d. Feedback/vote: both behavior is desirable
>>
>>
>> 3. Hashes
>> a. this represents field-value or java bean object
>> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
>> c. Current implementation maps each hashes to
>> geode-partition-region(i.e. user1000 is geode-partition-region)
>> d. Feedback/vote
>> -- Should we map hashes to region-entry
>> -- region-key = user1000
>> -- region-value = map
>> -- This will provide java bean sort to behaviour with 10s of
>> field-value
>> -- Personally I would prefer this..
>> e. Feedback/vote: both behaviour is desirable
>>
>> 4. Sets
>> a. This represents unique keys in set
>> b. usage "sadd myset 1 2 3"
>> c. Current implementation maps each sadd to geode-partition-region(i.e.
>> myset is geode-partition-region)
>> d. Feedback/vote
>> -- Should we map set to region-entry
>> -- region-key = myset
>> -- region-value = Hashset
>> e. Feedback/vote: both behaviour is desirable
>>
>> 5. SortedSets
>> a. This represents unique keys in set with score (usecase Query top-10)
>> b. usage "zadd hackers 1940 "Alan Kay""
>> c. Current implementation maps each zadd to geode-partition-region(i.e.
>> hackers is geode-partition-region)
>> d. Feedback/vote
>> -- Should we map set to region-entry
>> -- region-key = hackers
>> -- region-value = Sorted Hashset
>> e. Feedback/vote: both behaviour is desirable
>>
>> 6. HyperLogLogs
>> a. A HyperLogLog is a probabilistic data structure used in order to
>> count unique things (technically this is referred to estimating the
>> cardinality of a set).
>> b. usage "pfadd hll a b c d"
>> c. Current implementation creates "HLL_REGION" geode-partition-region
>> upfront
>> d. hll becomes region-key and value is HLL object
>> e. any feedback?
>>
>> 7. Default config for geode-region (vote)
>> a. partition region
>> b. 1 redundant copy
>> c. Persistence
>> d. Eviction
>> e. Expiration
>> f. ?
>>
>> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
>> Thus for each operation we need to make sure type of key. In current
>> implementation we have different region for each redis type. Thus we have
>> another region(metaTypeRegion) which keeps type for each key. This makes
>> any operation in geode slow as it needs to verify that type. For instance,
>> creating new key need to make sure its already there or not. Whether we
>> should allow type change or not.
>> a. Feedback/vote
>> -- type change of key
>> -- Can we allow two key with same name but two differnt type (as it
>> will endup in two different geode-region)
>> String type "key1" in string region
>> HLL type "key1" in HLL region
>> b. any other feedback
>>
>> 9. Transactions:
>> a. we will not support transaction in redisAdapter as geode transaction
>> are limited to single node.
>> b. feedback?
>>
>> 10. Redis COMMAND (https://redis.io/commands/command)
>> a. should we implement this "COMMAND" ?
>>
>> 11. Any other redis command we should consider?
>>
>>
>> Thanks.
>> Hitesh
>>
>>
>>
Re: GeodeRedisAdapter improvments/feedback
Posted by Real Wes <Th...@outlook.com>.
We should be careful here on a decision. If we start replicating fat lists/sets/hash maps synchronously every update, the Geode user will complain how slow the API is compared with Redis. Note: Redis replicates asynchronously. For fat collections we’re better off creating a region and suffering the penalty of constraining the key name to region name constraints (no colons, only alphanumerics, etc).
On Feb 15, 2017, at 3:24 PM, Hitesh Khamesra <hi...@yahoo.com>> wrote:
>>>The Redis adapter was designed so that we can scale all the Redis data
structures horizontally. If you bring the data structures to region entry
level, there is no reason for anyone to use our implementation over Redis.
hmm, here we need to understand when we need to create partition-region for Redis data types(item 2,3,4,5)
Creating partition-region for each use case may not be feasible. See a couple of use-cases mentioned earlier in the thread.
________________________________
On Tue, Feb 14, 2017 at 3:15 PM Jason Huynh <jh...@pivotal.io>> wrote:
> Hi Hitesh,
>
> Not sure about everyone else, but I had a hard time reading this, however
> I think I figured out what you were describing... the only part I still am
> unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> you want feedback and voting on whether both behaviors are desired? As in
> old implementation and new implementation?
>
> 2,3,4) The new implementation would mean all the data for a specific data
> structure is contained in a single bucket. So the individual data
> structures are not quite scalable. How would you allow scaling of a single
> data structure?
>
> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com>> wrote:
>
> In what format do you want the feedback Hitesh? For now I’ll just comment:
>
> 1. Redis Type String
> No comments except that a future Geode value-add would be to extend the
> Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> will work. The tradeoff of this is that the data cannot be read by a
> native redis client but for Geode users it’s great. Call the new client
> Geodis.
>
> 2. List/ Hash/ Set/ SortedSet
> Creating a separate region for each creates a constraint that the keys are
> limited to the characters for region names, which are A-z/0-9/ - and _.
> Everything else is out. Redis users might start asking questions why their
> list named ++^^/## throws an error. Your suggestion to make it a key rather
> than a region solves this. Furthermore, creating a new region every time a
> new Redis collection is created is going to be slow. I’m not sure why a
> region was created but I’m sure it made sense to the developer at the time.
>
> 7. Default Config
> Can’t we configure a gfsh option to default to the region types we want?
> Customer A will want PARTITION but Customer B will want
> PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a
> geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT
> that makes _all_ Redis regions of that type?
>
>
>
> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hi...@yahoo.com><mailto:
> hiteshk25@yahoo.com<ma...@yahoo.com>>> wrote:
>
> Current GeodeRedisAdapter implementation is based on
> https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal
> .
> We are looking for some feedback on Redis commands and their mapping to
> geode region.
>
> 1. Redis Type String
> a. Usage Set k1 v1
> b. Current implementation creates "STRING_REGION" geode-partition-region
> upfront
> c. This k1/v1 are geode-region key/value
> d. Any feedback?
>
> 2. List Type
> a. usage "rpush mylist A"
> b. Current implementation maps each list to geode-partition-region(i.e.
> mylist is geode-partition-region); with the ability to get item from
> head/tail
> c. Feedback/vote
> -- List type operation at region-entry level;
> -- region-key = "mylist"
> -- region-value = Arraylist (will support all redis list ops)
> d. Feedback/vote: both behavior is desirable
>
>
> 3. Hashes
> a. this represents field-value or java bean object
> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> c. Current implementation maps each hashes to
> geode-partition-region(i.e. user1000 is geode-partition-region)
> d. Feedback/vote
> -- Should we map hashes to region-entry
> -- region-key = user1000
> -- region-value = map
> -- This will provide java bean sort to behaviour with 10s of
> field-value
> -- Personally I would prefer this..
> e. Feedback/vote: both behaviour is desirable
>
> 4. Sets
> a. This represents unique keys in set
> b. usage "sadd myset 1 2 3"
> c. Current implementation maps each sadd to geode-partition-region(i.e.
> myset is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = myset
> -- region-value = Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 5. SortedSets
> a. This represents unique keys in set with score (usecase Query top-10)
> b. usage "zadd hackers 1940 "Alan Kay""
> c. Current implementation maps each zadd to geode-partition-region(i.e.
> hackers is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = hackers
> -- region-value = Sorted Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 6. HyperLogLogs
> a. A HyperLogLog is a probabilistic data structure used in order to
> count unique things (technically this is referred to estimating the
> cardinality of a set).
> b. usage "pfadd hll a b c d"
> c. Current implementation creates "HLL_REGION" geode-partition-region
> upfront
> d. hll becomes region-key and value is HLL object
> e. any feedback?
>
> 7. Default config for geode-region (vote)
> a. partition region
> b. 1 redundant copy
> c. Persistence
> d. Eviction
> e. Expiration
> f. ?
>
> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> Thus for each operation we need to make sure type of key. In current
> implementation we have different region for each redis type. Thus we have
> another region(metaTypeRegion) which keeps type for each key. This makes
> any operation in geode slow as it needs to verify that type. For instance,
> creating new key need to make sure its already there or not. Whether we
> should allow type change or not.
> a. Feedback/vote
> -- type change of key
> -- Can we allow two key with same name but two differnt type (as it
> will endup in two different geode-region)
> String type "key1" in string region
> HLL type "key1" in HLL region
> b. any other feedback
>
> 9. Transactions:
> a. we will not support transaction in redisAdapter as geode transaction
> are limited to single node.
> b. feedback?
>
> 10. Redis COMMAND (https://redis.io/commands/command)
> a. should we implement this "COMMAND" ?
>
> 11. Any other redis command we should consider?
>
>
> Thanks.
> Hitesh
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Real Wes <Th...@outlook.com>.
We should be careful here on a decision. If we start replicating fat lists/sets/hash maps synchronously every update, the Geode user will complain how slow the API is compared with Redis. Note: Redis replicates asynchronously. For fat collections we’re better off creating a region and suffering the penalty of constraining the key name to region name constraints (no colons, only alphanumerics, etc).
On Feb 15, 2017, at 3:24 PM, Hitesh Khamesra <hi...@yahoo.com>> wrote:
>>>The Redis adapter was designed so that we can scale all the Redis data
structures horizontally. If you bring the data structures to region entry
level, there is no reason for anyone to use our implementation over Redis.
hmm, here we need to understand when we need to create partition-region for Redis data types(item 2,3,4,5)
Creating partition-region for each use case may not be feasible. See a couple of use-cases mentioned earlier in the thread.
________________________________
On Tue, Feb 14, 2017 at 3:15 PM Jason Huynh <jh...@pivotal.io>> wrote:
> Hi Hitesh,
>
> Not sure about everyone else, but I had a hard time reading this, however
> I think I figured out what you were describing... the only part I still am
> unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> you want feedback and voting on whether both behaviors are desired? As in
> old implementation and new implementation?
>
> 2,3,4) The new implementation would mean all the data for a specific data
> structure is contained in a single bucket. So the individual data
> structures are not quite scalable. How would you allow scaling of a single
> data structure?
>
> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com>> wrote:
>
> In what format do you want the feedback Hitesh? For now I’ll just comment:
>
> 1. Redis Type String
> No comments except that a future Geode value-add would be to extend the
> Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> will work. The tradeoff of this is that the data cannot be read by a
> native redis client but for Geode users it’s great. Call the new client
> Geodis.
>
> 2. List/ Hash/ Set/ SortedSet
> Creating a separate region for each creates a constraint that the keys are
> limited to the characters for region names, which are A-z/0-9/ - and _.
> Everything else is out. Redis users might start asking questions why their
> list named ++^^/## throws an error. Your suggestion to make it a key rather
> than a region solves this. Furthermore, creating a new region every time a
> new Redis collection is created is going to be slow. I’m not sure why a
> region was created but I’m sure it made sense to the developer at the time.
>
> 7. Default Config
> Can’t we configure a gfsh option to default to the region types we want?
> Customer A will want PARTITION but Customer B will want
> PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a
> geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT
> that makes _all_ Redis regions of that type?
>
>
>
> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hi...@yahoo.com><mailto:
> hiteshk25@yahoo.com<ma...@yahoo.com>>> wrote:
>
> Current GeodeRedisAdapter implementation is based on
> https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal
> .
> We are looking for some feedback on Redis commands and their mapping to
> geode region.
>
> 1. Redis Type String
> a. Usage Set k1 v1
> b. Current implementation creates "STRING_REGION" geode-partition-region
> upfront
> c. This k1/v1 are geode-region key/value
> d. Any feedback?
>
> 2. List Type
> a. usage "rpush mylist A"
> b. Current implementation maps each list to geode-partition-region(i.e.
> mylist is geode-partition-region); with the ability to get item from
> head/tail
> c. Feedback/vote
> -- List type operation at region-entry level;
> -- region-key = "mylist"
> -- region-value = Arraylist (will support all redis list ops)
> d. Feedback/vote: both behavior is desirable
>
>
> 3. Hashes
> a. this represents field-value or java bean object
> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> c. Current implementation maps each hashes to
> geode-partition-region(i.e. user1000 is geode-partition-region)
> d. Feedback/vote
> -- Should we map hashes to region-entry
> -- region-key = user1000
> -- region-value = map
> -- This will provide java bean sort to behaviour with 10s of
> field-value
> -- Personally I would prefer this..
> e. Feedback/vote: both behaviour is desirable
>
> 4. Sets
> a. This represents unique keys in set
> b. usage "sadd myset 1 2 3"
> c. Current implementation maps each sadd to geode-partition-region(i.e.
> myset is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = myset
> -- region-value = Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 5. SortedSets
> a. This represents unique keys in set with score (usecase Query top-10)
> b. usage "zadd hackers 1940 "Alan Kay""
> c. Current implementation maps each zadd to geode-partition-region(i.e.
> hackers is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = hackers
> -- region-value = Sorted Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 6. HyperLogLogs
> a. A HyperLogLog is a probabilistic data structure used in order to
> count unique things (technically this is referred to estimating the
> cardinality of a set).
> b. usage "pfadd hll a b c d"
> c. Current implementation creates "HLL_REGION" geode-partition-region
> upfront
> d. hll becomes region-key and value is HLL object
> e. any feedback?
>
> 7. Default config for geode-region (vote)
> a. partition region
> b. 1 redundant copy
> c. Persistence
> d. Eviction
> e. Expiration
> f. ?
>
> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> Thus for each operation we need to make sure type of key. In current
> implementation we have different region for each redis type. Thus we have
> another region(metaTypeRegion) which keeps type for each key. This makes
> any operation in geode slow as it needs to verify that type. For instance,
> creating new key need to make sure its already there or not. Whether we
> should allow type change or not.
> a. Feedback/vote
> -- type change of key
> -- Can we allow two key with same name but two differnt type (as it
> will endup in two different geode-region)
> String type "key1" in string region
> HLL type "key1" in HLL region
> b. any other feedback
>
> 9. Transactions:
> a. we will not support transaction in redisAdapter as geode transaction
> are limited to single node.
> b. feedback?
>
> 10. Redis COMMAND (https://redis.io/commands/command)
> a. should we implement this "COMMAND" ?
>
> 11. Any other redis command we should consider?
>
>
> Thanks.
> Hitesh
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Hitesh Khamesra <hi...@yahoo.com>.
>>>The Redis adapter was designed so that we can scale all the Redis data
structures horizontally. If you bring the data structures to region entry
level, there is no reason for anyone to use our implementation over Redis.
hmm, here we need to understand when we need to create partition-region for Redis data types(item 2,3,4,5)
Creating partition-region for each use case may not be feasible. See a couple of use-cases mentioned earlier in the thread.
On Tue, Feb 14, 2017 at 3:15 PM Jason Huynh <jh...@pivotal.io> wrote:
> Hi Hitesh,
>
> Not sure about everyone else, but I had a hard time reading this, however
> I think I figured out what you were describing... the only part I still am
> unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> you want feedback and voting on whether both behaviors are desired? As in
> old implementation and new implementation?
>
> 2,3,4) The new implementation would mean all the data for a specific data
> structure is contained in a single bucket. So the individual data
> structures are not quite scalable. How would you allow scaling of a single
> data structure?
>
> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
>
> In what format do you want the feedback Hitesh? For now I’ll just comment:
>
> 1. Redis Type String
> No comments except that a future Geode value-add would be to extend the
> Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> will work. The tradeoff of this is that the data cannot be read by a
> native redis client but for Geode users it’s great. Call the new client
> Geodis.
>
> 2. List/ Hash/ Set/ SortedSet
> Creating a separate region for each creates a constraint that the keys are
> limited to the characters for region names, which are A-z/0-9/ - and _.
> Everything else is out. Redis users might start asking questions why their
> list named ++^^/## throws an error. Your suggestion to make it a key rather
> than a region solves this. Furthermore, creating a new region every time a
> new Redis collection is created is going to be slow. I’m not sure why a
> region was created but I’m sure it made sense to the developer at the time.
>
> 7. Default Config
> Can’t we configure a gfsh option to default to the region types we want?
> Customer A will want PARTITION but Customer B will want
> PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a
> geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT
> that makes _all_ Redis regions of that type?
>
>
>
> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com<mailto:
> hiteshk25@yahoo.com>> wrote:
>
> Current GeodeRedisAdapter implementation is based on
> https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal
> .
> We are looking for some feedback on Redis commands and their mapping to
> geode region.
>
> 1. Redis Type String
> a. Usage Set k1 v1
> b. Current implementation creates "STRING_REGION" geode-partition-region
> upfront
> c. This k1/v1 are geode-region key/value
> d. Any feedback?
>
> 2. List Type
> a. usage "rpush mylist A"
> b. Current implementation maps each list to geode-partition-region(i.e.
> mylist is geode-partition-region); with the ability to get item from
> head/tail
> c. Feedback/vote
> -- List type operation at region-entry level;
> -- region-key = "mylist"
> -- region-value = Arraylist (will support all redis list ops)
> d. Feedback/vote: both behavior is desirable
>
>
> 3. Hashes
> a. this represents field-value or java bean object
> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> c. Current implementation maps each hashes to
> geode-partition-region(i.e. user1000 is geode-partition-region)
> d. Feedback/vote
> -- Should we map hashes to region-entry
> -- region-key = user1000
> -- region-value = map
> -- This will provide java bean sort to behaviour with 10s of
> field-value
> -- Personally I would prefer this..
> e. Feedback/vote: both behaviour is desirable
>
> 4. Sets
> a. This represents unique keys in set
> b. usage "sadd myset 1 2 3"
> c. Current implementation maps each sadd to geode-partition-region(i.e.
> myset is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = myset
> -- region-value = Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 5. SortedSets
> a. This represents unique keys in set with score (usecase Query top-10)
> b. usage "zadd hackers 1940 "Alan Kay""
> c. Current implementation maps each zadd to geode-partition-region(i.e.
> hackers is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = hackers
> -- region-value = Sorted Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 6. HyperLogLogs
> a. A HyperLogLog is a probabilistic data structure used in order to
> count unique things (technically this is referred to estimating the
> cardinality of a set).
> b. usage "pfadd hll a b c d"
> c. Current implementation creates "HLL_REGION" geode-partition-region
> upfront
> d. hll becomes region-key and value is HLL object
> e. any feedback?
>
> 7. Default config for geode-region (vote)
> a. partition region
> b. 1 redundant copy
> c. Persistence
> d. Eviction
> e. Expiration
> f. ?
>
> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> Thus for each operation we need to make sure type of key. In current
> implementation we have different region for each redis type. Thus we have
> another region(metaTypeRegion) which keeps type for each key. This makes
> any operation in geode slow as it needs to verify that type. For instance,
> creating new key need to make sure its already there or not. Whether we
> should allow type change or not.
> a. Feedback/vote
> -- type change of key
> -- Can we allow two key with same name but two differnt type (as it
> will endup in two different geode-region)
> String type "key1" in string region
> HLL type "key1" in HLL region
> b. any other feedback
>
> 9. Transactions:
> a. we will not support transaction in redisAdapter as geode transaction
> are limited to single node.
> b. feedback?
>
> 10. Redis COMMAND (https://redis.io/commands/command)
> a. should we implement this "COMMAND" ?
>
> 11. Any other redis command we should consider?
>
>
> Thanks.
> Hitesh
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Hitesh Khamesra <hi...@yahoo.com.INVALID>.
>>>The Redis adapter was designed so that we can scale all the Redis data
structures horizontally. If you bring the data structures to region entry
level, there is no reason for anyone to use our implementation over Redis.
hmm, here we need to understand when we need to create partition-region for Redis data types(item 2,3,4,5)
Creating partition-region for each use case may not be feasible. See a couple of use-cases mentioned earlier in the thread.
On Tue, Feb 14, 2017 at 3:15 PM Jason Huynh <jh...@pivotal.io> wrote:
> Hi Hitesh,
>
> Not sure about everyone else, but I had a hard time reading this, however
> I think I figured out what you were describing... the only part I still am
> unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> you want feedback and voting on whether both behaviors are desired? As in
> old implementation and new implementation?
>
> 2,3,4) The new implementation would mean all the data for a specific data
> structure is contained in a single bucket. So the individual data
> structures are not quite scalable. How would you allow scaling of a single
> data structure?
>
> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
>
> In what format do you want the feedback Hitesh? For now I’ll just comment:
>
> 1. Redis Type String
> No comments except that a future Geode value-add would be to extend the
> Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> will work. The tradeoff of this is that the data cannot be read by a
> native redis client but for Geode users it’s great. Call the new client
> Geodis.
>
> 2. List/ Hash/ Set/ SortedSet
> Creating a separate region for each creates a constraint that the keys are
> limited to the characters for region names, which are A-z/0-9/ - and _.
> Everything else is out. Redis users might start asking questions why their
> list named ++^^/## throws an error. Your suggestion to make it a key rather
> than a region solves this. Furthermore, creating a new region every time a
> new Redis collection is created is going to be slow. I’m not sure why a
> region was created but I’m sure it made sense to the developer at the time.
>
> 7. Default Config
> Can’t we configure a gfsh option to default to the region types we want?
> Customer A will want PARTITION but Customer B will want
> PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a
> geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT
> that makes _all_ Redis regions of that type?
>
>
>
> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com<mailto:
> hiteshk25@yahoo.com>> wrote:
>
> Current GeodeRedisAdapter implementation is based on
> https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal
> .
> We are looking for some feedback on Redis commands and their mapping to
> geode region.
>
> 1. Redis Type String
> a. Usage Set k1 v1
> b. Current implementation creates "STRING_REGION" geode-partition-region
> upfront
> c. This k1/v1 are geode-region key/value
> d. Any feedback?
>
> 2. List Type
> a. usage "rpush mylist A"
> b. Current implementation maps each list to geode-partition-region(i.e.
> mylist is geode-partition-region); with the ability to get item from
> head/tail
> c. Feedback/vote
> -- List type operation at region-entry level;
> -- region-key = "mylist"
> -- region-value = Arraylist (will support all redis list ops)
> d. Feedback/vote: both behavior is desirable
>
>
> 3. Hashes
> a. this represents field-value or java bean object
> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> c. Current implementation maps each hashes to
> geode-partition-region(i.e. user1000 is geode-partition-region)
> d. Feedback/vote
> -- Should we map hashes to region-entry
> -- region-key = user1000
> -- region-value = map
> -- This will provide java bean sort to behaviour with 10s of
> field-value
> -- Personally I would prefer this..
> e. Feedback/vote: both behaviour is desirable
>
> 4. Sets
> a. This represents unique keys in set
> b. usage "sadd myset 1 2 3"
> c. Current implementation maps each sadd to geode-partition-region(i.e.
> myset is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = myset
> -- region-value = Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 5. SortedSets
> a. This represents unique keys in set with score (usecase Query top-10)
> b. usage "zadd hackers 1940 "Alan Kay""
> c. Current implementation maps each zadd to geode-partition-region(i.e.
> hackers is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = hackers
> -- region-value = Sorted Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 6. HyperLogLogs
> a. A HyperLogLog is a probabilistic data structure used in order to
> count unique things (technically this is referred to estimating the
> cardinality of a set).
> b. usage "pfadd hll a b c d"
> c. Current implementation creates "HLL_REGION" geode-partition-region
> upfront
> d. hll becomes region-key and value is HLL object
> e. any feedback?
>
> 7. Default config for geode-region (vote)
> a. partition region
> b. 1 redundant copy
> c. Persistence
> d. Eviction
> e. Expiration
> f. ?
>
> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> Thus for each operation we need to make sure type of key. In current
> implementation we have different region for each redis type. Thus we have
> another region(metaTypeRegion) which keeps type for each key. This makes
> any operation in geode slow as it needs to verify that type. For instance,
> creating new key need to make sure its already there or not. Whether we
> should allow type change or not.
> a. Feedback/vote
> -- type change of key
> -- Can we allow two key with same name but two differnt type (as it
> will endup in two different geode-region)
> String type "key1" in string region
> HLL type "key1" in HLL region
> b. any other feedback
>
> 9. Transactions:
> a. we will not support transaction in redisAdapter as geode transaction
> are limited to single node.
> b. feedback?
>
> 10. Redis COMMAND (https://redis.io/commands/command)
> a. should we implement this "COMMAND" ?
>
> 11. Any other redis command we should consider?
>
>
> Thanks.
> Hitesh
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Dan Smith <ds...@pivotal.io>.
I also had a hard time reading this. It sounds like the tl;dr is that your
are proposing storing each redis collection in a single region entry,
rather than a a partition region? I guess the question is whether users
will have a few very large collections that need to be partitioned, or lots
and lots of really tiny collections. I'm not quite sure what the more
common use case is.
-Dan
On Tue, Feb 14, 2017 at 3:22 PM, Swapnil Bawaskar <sb...@pivotal.io>
wrote:
> The Redis adapter was designed so that we can scale all the Redis data
> structures horizontally. If you bring the data structures to region entry
> level, there is no reason for anyone to use our implementation over Redis.
>
> On Tue, Feb 14, 2017 at 3:15 PM Jason Huynh <jh...@pivotal.io> wrote:
>
>> Hi Hitesh,
>>
>> Not sure about everyone else, but I had a hard time reading this,
>> however I think I figured out what you were describing... the only part I
>> still am unsure about is Feedback/vote: both behaviour is desirable. Do
>> you mean you want feedback and voting on whether both behaviors are
>> desired? As in old implementation and new implementation?
>>
>> 2,3,4) The new implementation would mean all the data for a specific
>> data structure is contained in a single bucket. So the individual data
>> structures are not quite scalable. How would you allow scaling of a single
>> data structure?
>>
>> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
>>
>> In what format do you want the feedback Hitesh? For now I’ll just
>> comment:
>>
>> 1. Redis Type String
>> No comments except that a future Geode value-add would be to extend the
>> Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
>> will work. The tradeoff of this is that the data cannot be read by a
>> native redis client but for Geode users it’s great. Call the new client
>> Geodis.
>>
>> 2. List/ Hash/ Set/ SortedSet
>> Creating a separate region for each creates a constraint that the keys
>> are limited to the characters for region names, which are A-z/0-9/ - and
>> _. Everything else is out. Redis users might start asking questions why
>> their list named ++^^/## throws an error. Your suggestion to make it a key
>> rather than a region solves this. Furthermore, creating a new region every
>> time a new Redis collection is created is going to be slow. I’m not sure
>> why a region was created but I’m sure it made sense to the developer at the
>> time.
>>
>> 7. Default Config
>> Can’t we configure a gfsh option to default to the region types we want?
>> Customer A will want PARTITION but Customer B will want PARTITION_REDUNDANT_EXPIRATION_PERSISTENT.
>> I wonder if we can consider a geode> create region —redisType=PARTITION_
>> REDUNDANT_EXPIRATION_PERSISTENT that makes _all_ Redis regions of that
>> type?
>>
>>
>>
>> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com<mailto:
>> hiteshk25@yahoo.com>> wrote:
>>
>> Current GeodeRedisAdapter implementation is based on
>> https://cwiki.apache.org/confluence/display/GEODE/
>> Geode+Redis+Adapter+Proposal.
>> We are looking for some feedback on Redis commands and their mapping to
>> geode region.
>>
>> 1. Redis Type String
>> a. Usage Set k1 v1
>> b. Current implementation creates "STRING_REGION"
>> geode-partition-region upfront
>> c. This k1/v1 are geode-region key/value
>> d. Any feedback?
>>
>> 2. List Type
>> a. usage "rpush mylist A"
>> b. Current implementation maps each list to geode-partition-region(i.e.
>> mylist is geode-partition-region); with the ability to get item from
>> head/tail
>> c. Feedback/vote
>> -- List type operation at region-entry level;
>> -- region-key = "mylist"
>> -- region-value = Arraylist (will support all redis list ops)
>> d. Feedback/vote: both behavior is desirable
>>
>>
>> 3. Hashes
>> a. this represents field-value or java bean object
>> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
>> c. Current implementation maps each hashes to
>> geode-partition-region(i.e. user1000 is geode-partition-region)
>> d. Feedback/vote
>> -- Should we map hashes to region-entry
>> -- region-key = user1000
>> -- region-value = map
>> -- This will provide java bean sort to behaviour with 10s of
>> field-value
>> -- Personally I would prefer this..
>> e. Feedback/vote: both behaviour is desirable
>>
>> 4. Sets
>> a. This represents unique keys in set
>> b. usage "sadd myset 1 2 3"
>> c. Current implementation maps each sadd to geode-partition-region(i.e.
>> myset is geode-partition-region)
>> d. Feedback/vote
>> -- Should we map set to region-entry
>> -- region-key = myset
>> -- region-value = Hashset
>> e. Feedback/vote: both behaviour is desirable
>>
>> 5. SortedSets
>> a. This represents unique keys in set with score (usecase Query top-10)
>> b. usage "zadd hackers 1940 "Alan Kay""
>> c. Current implementation maps each zadd to geode-partition-region(i.e.
>> hackers is geode-partition-region)
>> d. Feedback/vote
>> -- Should we map set to region-entry
>> -- region-key = hackers
>> -- region-value = Sorted Hashset
>> e. Feedback/vote: both behaviour is desirable
>>
>> 6. HyperLogLogs
>> a. A HyperLogLog is a probabilistic data structure used in order to
>> count unique things (technically this is referred to estimating the
>> cardinality of a set).
>> b. usage "pfadd hll a b c d"
>> c. Current implementation creates "HLL_REGION" geode-partition-region
>> upfront
>> d. hll becomes region-key and value is HLL object
>> e. any feedback?
>>
>> 7. Default config for geode-region (vote)
>> a. partition region
>> b. 1 redundant copy
>> c. Persistence
>> d. Eviction
>> e. Expiration
>> f. ?
>>
>> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
>> Thus for each operation we need to make sure type of key. In current
>> implementation we have different region for each redis type. Thus we have
>> another region(metaTypeRegion) which keeps type for each key. This makes
>> any operation in geode slow as it needs to verify that type. For instance,
>> creating new key need to make sure its already there or not. Whether we
>> should allow type change or not.
>> a. Feedback/vote
>> -- type change of key
>> -- Can we allow two key with same name but two differnt type (as it
>> will endup in two different geode-region)
>> String type "key1" in string region
>> HLL type "key1" in HLL region
>> b. any other feedback
>>
>> 9. Transactions:
>> a. we will not support transaction in redisAdapter as geode transaction
>> are limited to single node.
>> b. feedback?
>>
>> 10. Redis COMMAND (https://redis.io/commands/command)
>> a. should we implement this "COMMAND" ?
>>
>> 11. Any other redis command we should consider?
>>
>>
>> Thanks.
>> Hitesh
>>
>>
>>
Re: GeodeRedisAdapter improvments/feedback
Posted by Swapnil Bawaskar <sb...@pivotal.io>.
The Redis adapter was designed so that we can scale all the Redis data
structures horizontally. If you bring the data structures to region entry
level, there is no reason for anyone to use our implementation over Redis.
On Tue, Feb 14, 2017 at 3:15 PM Jason Huynh <jh...@pivotal.io> wrote:
> Hi Hitesh,
>
> Not sure about everyone else, but I had a hard time reading this, however
> I think I figured out what you were describing... the only part I still am
> unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> you want feedback and voting on whether both behaviors are desired? As in
> old implementation and new implementation?
>
> 2,3,4) The new implementation would mean all the data for a specific data
> structure is contained in a single bucket. So the individual data
> structures are not quite scalable. How would you allow scaling of a single
> data structure?
>
> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
>
> In what format do you want the feedback Hitesh? For now I’ll just comment:
>
> 1. Redis Type String
> No comments except that a future Geode value-add would be to extend the
> Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> will work. The tradeoff of this is that the data cannot be read by a
> native redis client but for Geode users it’s great. Call the new client
> Geodis.
>
> 2. List/ Hash/ Set/ SortedSet
> Creating a separate region for each creates a constraint that the keys are
> limited to the characters for region names, which are A-z/0-9/ - and _.
> Everything else is out. Redis users might start asking questions why their
> list named ++^^/## throws an error. Your suggestion to make it a key rather
> than a region solves this. Furthermore, creating a new region every time a
> new Redis collection is created is going to be slow. I’m not sure why a
> region was created but I’m sure it made sense to the developer at the time.
>
> 7. Default Config
> Can’t we configure a gfsh option to default to the region types we want?
> Customer A will want PARTITION but Customer B will want
> PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a
> geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT
> that makes _all_ Redis regions of that type?
>
>
>
> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com<mailto:
> hiteshk25@yahoo.com>> wrote:
>
> Current GeodeRedisAdapter implementation is based on
> https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal
> .
> We are looking for some feedback on Redis commands and their mapping to
> geode region.
>
> 1. Redis Type String
> a. Usage Set k1 v1
> b. Current implementation creates "STRING_REGION" geode-partition-region
> upfront
> c. This k1/v1 are geode-region key/value
> d. Any feedback?
>
> 2. List Type
> a. usage "rpush mylist A"
> b. Current implementation maps each list to geode-partition-region(i.e.
> mylist is geode-partition-region); with the ability to get item from
> head/tail
> c. Feedback/vote
> -- List type operation at region-entry level;
> -- region-key = "mylist"
> -- region-value = Arraylist (will support all redis list ops)
> d. Feedback/vote: both behavior is desirable
>
>
> 3. Hashes
> a. this represents field-value or java bean object
> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> c. Current implementation maps each hashes to
> geode-partition-region(i.e. user1000 is geode-partition-region)
> d. Feedback/vote
> -- Should we map hashes to region-entry
> -- region-key = user1000
> -- region-value = map
> -- This will provide java bean sort to behaviour with 10s of
> field-value
> -- Personally I would prefer this..
> e. Feedback/vote: both behaviour is desirable
>
> 4. Sets
> a. This represents unique keys in set
> b. usage "sadd myset 1 2 3"
> c. Current implementation maps each sadd to geode-partition-region(i.e.
> myset is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = myset
> -- region-value = Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 5. SortedSets
> a. This represents unique keys in set with score (usecase Query top-10)
> b. usage "zadd hackers 1940 "Alan Kay""
> c. Current implementation maps each zadd to geode-partition-region(i.e.
> hackers is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = hackers
> -- region-value = Sorted Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 6. HyperLogLogs
> a. A HyperLogLog is a probabilistic data structure used in order to
> count unique things (technically this is referred to estimating the
> cardinality of a set).
> b. usage "pfadd hll a b c d"
> c. Current implementation creates "HLL_REGION" geode-partition-region
> upfront
> d. hll becomes region-key and value is HLL object
> e. any feedback?
>
> 7. Default config for geode-region (vote)
> a. partition region
> b. 1 redundant copy
> c. Persistence
> d. Eviction
> e. Expiration
> f. ?
>
> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> Thus for each operation we need to make sure type of key. In current
> implementation we have different region for each redis type. Thus we have
> another region(metaTypeRegion) which keeps type for each key. This makes
> any operation in geode slow as it needs to verify that type. For instance,
> creating new key need to make sure its already there or not. Whether we
> should allow type change or not.
> a. Feedback/vote
> -- type change of key
> -- Can we allow two key with same name but two differnt type (as it
> will endup in two different geode-region)
> String type "key1" in string region
> HLL type "key1" in HLL region
> b. any other feedback
>
> 9. Transactions:
> a. we will not support transaction in redisAdapter as geode transaction
> are limited to single node.
> b. feedback?
>
> 10. Redis COMMAND (https://redis.io/commands/command)
> a. should we implement this "COMMAND" ?
>
> 11. Any other redis command we should consider?
>
>
> Thanks.
> Hitesh
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Hitesh Khamesra <hi...@yahoo.com.INVALID>.
Right..
Sent from Yahoo Mail on Android
On Fri, Feb 17, 2017 at 8:05 AM, Wes Williams<ww...@pivotal.io> wrote: I'm not clear on the reference to "I like the idea of first class data
structures like Lists and Sorted Sets."
Is the suggestion here to extend Geode to not only support a distributed
ConcurrentHashMap but also distributed ConcurrentList's and
ConcurrentSortedSet's?
*Wes Williams | Pivotal Advisory **Data Engineer*
781.606.0325
http://pivotal.io/big-data/pivotal-gemfire
On Thu, Feb 16, 2017 at 10:34 AM, Michael Stolz <ms...@pivotal.io> wrote:
> I like the idea of first class data structures like Lists and Sorted Sets.
>
> I'm not sure which side of the fence I'm on in terms of very large objects
> and using Regions to represent them. Feels very heavy because of all the
> overhead of a Region entry in Geode (over 300 bytes per entry).
>
> I think the main reason people will want to use Geode in place of Redis
> will be horizontal scale in terms of the number of structures first, size
> of structures second, ability to get additional enterprise features like
> synchronous instead of asynchronous replication from masters to slaves
> (zero-data-loss) multi-site and even multi-cloud use cases (WAN Gateway).
>
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Manager
> Mobile: +1-631-835-4771
>
> On Wed, Feb 15, 2017 at 8:09 PM, Swapnil Bawaskar <sb...@pivotal.io>
> wrote:
>
> > I think we as a community need to determine what value do we want to add
> > with the Redis adapter. Redis already does a great job storing small data
> > structures in memory and sharding them. We do a great job of making sure
> > that these data structures are horizontally scalable; why would we want
> to
> > replicate what another open source project is already implementing?
> >
> > Having said that, I would like to see a configuration property that lets
> > the user chose between a single server vs a distributed collection.
> >
> >
> > > I think we could have the following options:
> > >
> > > 1. have a property that could be set to use either single server
> > > collections over use the current distributed collection
> > > 2. have first class collection implementations that are distributed by
> > > nature, as using key:value as the hammer for all does not make
> sense
> > >
> >
> > I don't think these options are mutually exclusive. We should make lists
> > and SortedSets first class data structures in Geode alongside regions.
> >
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Wes Williams <ww...@pivotal.io>.
I'm not clear on the reference to "I like the idea of first class data
structures like Lists and Sorted Sets."
Is the suggestion here to extend Geode to not only support a distributed
ConcurrentHashMap but also distributed ConcurrentList's and
ConcurrentSortedSet's?
*Wes Williams | Pivotal Advisory **Data Engineer*
781.606.0325
http://pivotal.io/big-data/pivotal-gemfire
On Thu, Feb 16, 2017 at 10:34 AM, Michael Stolz <ms...@pivotal.io> wrote:
> I like the idea of first class data structures like Lists and Sorted Sets.
>
> I'm not sure which side of the fence I'm on in terms of very large objects
> and using Regions to represent them. Feels very heavy because of all the
> overhead of a Region entry in Geode (over 300 bytes per entry).
>
> I think the main reason people will want to use Geode in place of Redis
> will be horizontal scale in terms of the number of structures first, size
> of structures second, ability to get additional enterprise features like
> synchronous instead of asynchronous replication from masters to slaves
> (zero-data-loss) multi-site and even multi-cloud use cases (WAN Gateway).
>
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Manager
> Mobile: +1-631-835-4771
>
> On Wed, Feb 15, 2017 at 8:09 PM, Swapnil Bawaskar <sb...@pivotal.io>
> wrote:
>
> > I think we as a community need to determine what value do we want to add
> > with the Redis adapter. Redis already does a great job storing small data
> > structures in memory and sharding them. We do a great job of making sure
> > that these data structures are horizontally scalable; why would we want
> to
> > replicate what another open source project is already implementing?
> >
> > Having said that, I would like to see a configuration property that lets
> > the user chose between a single server vs a distributed collection.
> >
> >
> > > I think we could have the following options:
> > >
> > > 1. have a property that could be set to use either single server
> > > collections over use the current distributed collection
> > > 2. have first class collection implementations that are distributed by
> > > nature, as using key:value as the hammer for all does not make
> sense
> > >
> >
> > I don't think these options are mutually exclusive. We should make lists
> > and SortedSets first class data structures in Geode alongside regions.
> >
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Anthony Baker <ab...@pivotal.io>.
Me too! Bi-modal behaviors can result in systems with a large step function in performance when a threshold is crossed (I’m looking at you Compressed OOPS!). It would be nice to keep operational complexity low and give users a consistent behavior expectation.
Anthony
> On Feb 16, 2017, at 8:51 AM, John Blum <jb...@pivotal.io> wrote:
>
> I agree with Swapnil here on all points. Why duplicate what Redis already
> does? We need to focus on our strengths and why we are targeting Redis
> users... i.e. our "value add". Geode is not going to win any popularity
> contest against Redis for the exact reasons/UCs why Redis exists in the
> first place.
>
> Having said that, I was thinking to both minimize the overhead and as well
> as to preserve our horizontal scale out nature (unlike Redis), then just
> "spill one way" (perhaps). This is certainly far less complex to implement.
>
> Like our serialization mechanism, once a value is deserialized, it is
> always kept in the Region that way. Once a Redis data structure becomes
> too large and spills over (so to say), it stays that way. And the spill
> over should be based on the capacity of the Region/node; not (yet another)
> property.
>
> Just a thought.
>
>
> On Thu, Feb 16, 2017 at 7:34 AM, Michael Stolz <ms...@pivotal.io> wrote:
>
>> I like the idea of first class data structures like Lists and Sorted Sets.
>>
>> I'm not sure which side of the fence I'm on in terms of very large objects
>> and using Regions to represent them. Feels very heavy because of all the
>> overhead of a Region entry in Geode (over 300 bytes per entry).
>>
>> I think the main reason people will want to use Geode in place of Redis
>> will be horizontal scale in terms of the number of structures first, size
>> of structures second, ability to get additional enterprise features like
>> synchronous instead of asynchronous replication from masters to slaves
>> (zero-data-loss) multi-site and even multi-cloud use cases (WAN Gateway).
>>
>>
>>
>> --
>> Mike Stolz
>> Principal Engineer, GemFire Product Manager
>> Mobile: +1-631-835-4771
>>
>> On Wed, Feb 15, 2017 at 8:09 PM, Swapnil Bawaskar <sb...@pivotal.io>
>> wrote:
>>
>>> I think we as a community need to determine what value do we want to add
>>> with the Redis adapter. Redis already does a great job storing small data
>>> structures in memory and sharding them. We do a great job of making sure
>>> that these data structures are horizontally scalable; why would we want
>> to
>>> replicate what another open source project is already implementing?
>>>
>>> Having said that, I would like to see a configuration property that lets
>>> the user chose between a single server vs a distributed collection.
>>>
>>>
>>>> I think we could have the following options:
>>>>
>>>> 1. have a property that could be set to use either single server
>>>> collections over use the current distributed collection
>>>> 2. have first class collection implementations that are distributed by
>>>> nature, as using key:value as the hammer for all does not make
>> sense
>>>>
>>>
>>> I don't think these options are mutually exclusive. We should make lists
>>> and SortedSets first class data structures in Geode alongside regions.
>>>
>>
>
>
>
> --
> -John
> john.blum10101 (skype)
Re: GeodeRedisAdapter improvments/feedback
Posted by John Blum <jb...@pivotal.io>.
I agree with Swapnil here on all points. Why duplicate what Redis already
does? We need to focus on our strengths and why we are targeting Redis
users... i.e. our "value add". Geode is not going to win any popularity
contest against Redis for the exact reasons/UCs why Redis exists in the
first place.
Having said that, I was thinking to both minimize the overhead and as well
as to preserve our horizontal scale out nature (unlike Redis), then just
"spill one way" (perhaps). This is certainly far less complex to implement.
Like our serialization mechanism, once a value is deserialized, it is
always kept in the Region that way. Once a Redis data structure becomes
too large and spills over (so to say), it stays that way. And the spill
over should be based on the capacity of the Region/node; not (yet another)
property.
Just a thought.
On Thu, Feb 16, 2017 at 7:34 AM, Michael Stolz <ms...@pivotal.io> wrote:
> I like the idea of first class data structures like Lists and Sorted Sets.
>
> I'm not sure which side of the fence I'm on in terms of very large objects
> and using Regions to represent them. Feels very heavy because of all the
> overhead of a Region entry in Geode (over 300 bytes per entry).
>
> I think the main reason people will want to use Geode in place of Redis
> will be horizontal scale in terms of the number of structures first, size
> of structures second, ability to get additional enterprise features like
> synchronous instead of asynchronous replication from masters to slaves
> (zero-data-loss) multi-site and even multi-cloud use cases (WAN Gateway).
>
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Manager
> Mobile: +1-631-835-4771
>
> On Wed, Feb 15, 2017 at 8:09 PM, Swapnil Bawaskar <sb...@pivotal.io>
> wrote:
>
> > I think we as a community need to determine what value do we want to add
> > with the Redis adapter. Redis already does a great job storing small data
> > structures in memory and sharding them. We do a great job of making sure
> > that these data structures are horizontally scalable; why would we want
> to
> > replicate what another open source project is already implementing?
> >
> > Having said that, I would like to see a configuration property that lets
> > the user chose between a single server vs a distributed collection.
> >
> >
> > > I think we could have the following options:
> > >
> > > 1. have a property that could be set to use either single server
> > > collections over use the current distributed collection
> > > 2. have first class collection implementations that are distributed by
> > > nature, as using key:value as the hammer for all does not make
> sense
> > >
> >
> > I don't think these options are mutually exclusive. We should make lists
> > and SortedSets first class data structures in Geode alongside regions.
> >
>
--
-John
john.blum10101 (skype)
Re: GeodeRedisAdapter improvments/feedback
Posted by Michael Stolz <ms...@pivotal.io>.
I like the idea of first class data structures like Lists and Sorted Sets.
I'm not sure which side of the fence I'm on in terms of very large objects
and using Regions to represent them. Feels very heavy because of all the
overhead of a Region entry in Geode (over 300 bytes per entry).
I think the main reason people will want to use Geode in place of Redis
will be horizontal scale in terms of the number of structures first, size
of structures second, ability to get additional enterprise features like
synchronous instead of asynchronous replication from masters to slaves
(zero-data-loss) multi-site and even multi-cloud use cases (WAN Gateway).
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771
On Wed, Feb 15, 2017 at 8:09 PM, Swapnil Bawaskar <sb...@pivotal.io>
wrote:
> I think we as a community need to determine what value do we want to add
> with the Redis adapter. Redis already does a great job storing small data
> structures in memory and sharding them. We do a great job of making sure
> that these data structures are horizontally scalable; why would we want to
> replicate what another open source project is already implementing?
>
> Having said that, I would like to see a configuration property that lets
> the user chose between a single server vs a distributed collection.
>
>
> > I think we could have the following options:
> >
> > 1. have a property that could be set to use either single server
> > collections over use the current distributed collection
> > 2. have first class collection implementations that are distributed by
> > nature, as using key:value as the hammer for all does not make sense
> >
>
> I don't think these options are mutually exclusive. We should make lists
> and SortedSets first class data structures in Geode alongside regions.
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Swapnil Bawaskar <sb...@pivotal.io>.
I think we as a community need to determine what value do we want to add
with the Redis adapter. Redis already does a great job storing small data
structures in memory and sharding them. We do a great job of making sure
that these data structures are horizontally scalable; why would we want to
replicate what another open source project is already implementing?
Having said that, I would like to see a configuration property that lets
the user chose between a single server vs a distributed collection.
> I think we could have the following options:
>
> 1. have a property that could be set to use either single server
> collections over use the current distributed collection
> 2. have first class collection implementations that are distributed by
> nature, as using key:value as the hammer for all does not make sense
>
I don't think these options are mutually exclusive. We should make lists
and SortedSets first class data structures in Geode alongside regions.
Re: GeodeRedisAdapter improvments/feedback
Posted by Udo Kohlmeyer <uk...@pivotal.io>.
I believe that with delta propagation we could have single collections
on a single server, then have a redundant copy on another. I mean the
collections would have to be really large to really start warranting
distribution across multiple servers. As the perf impact we incur due to
network hops is too great to say that the Geode-Redis is "better" than
normal Redis.
I think we could have the following options:
1. have a property that could be set to use either single server
collections over use the current distributed collection
2. have first class collection implementations that are distributed by
nature, as using key:value as the hammer for all does not make sense
The only value add that the current region implementation has, is that
collections can scale. BUT how large are really large collections?
Thousands? Hundred of thousands? Millions?
My concern is that any benefit we gain by scaling we'd lose because of
per entry overhead. In addition to that, because of the single server
connection that the redis client has, we really don't benefit from the
"single" hop optimization that normal Geode clients gets.
On 2/15/17 13:34, Real Wes wrote:
> Does delta propagation make worrying about frequently updated fat collections moot?
>
> On Feb 15, 2017, at 4:29 PM, Dan Smith <ds...@pivotal.io>> wrote:
>
> Doing the spill/unspill option could be pretty tricky to implement, so you have to do a lot of fancy logic in the transition period. I think Jason's suggestion of configuring things might make more sense.
>
> -Dan
>
> On Wed, Feb 15, 2017 at 1:12 PM, Jason Huynh <jh...@pivotal.io>> wrote:
> With the suggestion from Wes, the constraint on the names would have to
> apply for both small and large. We wouldn't want the thing to explode when
> it gets converted...
>
> Is there a way to just make it configurable? If they know they want a
> "large" set, somehow let them specify it. Otherwise go with the "small"
> set?
>
> On Wed, Feb 15, 2017 at 1:01 PM Real Wes <Th...@outlook.com>> wrote:
>
>> Thinking about this, I think that the \u201cspill\u201d/ \u201cunspill\u201d option may
>> actually be the best solution. If the criteria waffles back and forth
>> along the threshold, well, that\u2019s the acceptable worst case.
>>
>> How\u2019s this?:
>>
>> 1) Create a separate region for the collection key
>> - for fat collections that are updated frequently
>> ADVANTAGE: speed of replication
>> DISADVANTAGE: constraint on key name
>>
>> 2) Put the collection as an entry value:
>> - for small collections and read-only fat collections
>> ADVANTAGE: no need to create a separate region
>>
>> We would track the metrics and automatically convert based on a
>> combination of frequency of updates and size.
>>
>> We next define what a fat collection is, such as over nnMB.
>>
>>
>> On Feb 14, 2017, at 8:12 PM, Jason Huynh <jh...@pivotal.io><mailto:
>> jhuynh@pivotal.io<ma...@pivotal.io>>> wrote:
>>
>> The concern about the threshold to spill over would be do you "unspill"
>> over? Like what if the collection contracts under the threshold and
>> teeters around the threshold. If the user can configure this size, then
>> wouldn't they just know they want a "large" vs a "small?"
>>
>> I think Swapnil makes a good point that our value add would be that we can
>> scale those structures, whereas redis can already do what the "new"
>> implementation is doing.
>>
>>
>>
>> On Tue, Feb 14, 2017 at 4:59 PM Galen M O'Sullivan <go...@pivotal.io>
>> <ma...@pivotal.io>>> wrote:
>> If we put them in separate regions, we'll have the overhead of looking up
>> in two regions added to each and every operation, and the overhead of
>> creating all these regions.
>>
>> If we really wanted to we could have some threshold at which we spill
>> collections over into their own regions, and have something like the best
>> of both worlds. It's more complex, though, and I don't know how many people
>> actually use truly huge collections.
>>
>> On Tue, Feb 14, 2017 at 4:21 PM, Hitesh Khamesra <
>> hiteshk25@yahoo.com.invalid<ma...@yahoo.com.invalid>>> wrote:
>>
>>> Jason/Dan: Sorry to hear about that. But both of you have asked the right
>>> question.
>>> it depends on your use-case(item 2,3,4,5) . For example "hashes" can be
>>> use to define key-value pair or java bean. In this case probably it is
>>> better to keep that hash at region-entry level. But if you want to know
>>> top 10 tweets which are trending then probably you want use
>>> partition-region for "sorted-set".
>>>
>>>
>>> From: Jason Huynh <jh...@pivotal.io>>>
>>> To: dev@geode.apache.org<ma...@geode.apache.org>>; "
>> user@geode.apache.org<ma...@geode.apache.org>>" <
>> user@geode.apache.org<ma...@geode.apache.org>>>;
>>> Hitesh Khamesra <hi...@yahoo.com>>>
>>> Sent: Tuesday, February 14, 2017 3:15 PM
>>> Subject: Re: GeodeRedisAdapter improvments/feedback
>>>
>>> Hi Hitesh,
>>>
>>> Not sure about everyone else, but I had a hard time reading this,
>> however
>>> I think I figured out what you were describing... the only part I still
>> am
>>> unsure about is Feedback/vote: both behaviour is desirable. Do you mean
>>> you want feedback and voting on whether both behaviors are desired? As
>> in
>>> old implementation and new implementation?
>>>
>>> 2,3,4) The new implementation would mean all the data for a specific
>> data
>>> structure is contained in a single bucket. So the individual data
>>> structures are not quite scalable. How would you allow scaling of a
>> single
>>> data structure?
>>>
>>> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com><mailto:
>> TheRealWes@outlook.com<ma...@outlook.com>>> wrote:
>>>> In what format do you want the feedback Hitesh? For now I\u2019ll just
>>> comment:
>>>> 1. Redis Type String
>>>> No comments except that a future Geode value-add would be to extend the
>>>> Jedis client so that the K/V\u2019s are not compressed. In this way OQL and
>> CQ
>>>> will work. The tradeoff of this is that the data cannot be read by a
>>>> native redis client but for Geode users it\u2019s great. Call the new client
>>>> Geodis.
>>>>
>>>> 2. List/ Hash/ Set/ SortedSet
>>>> Creating a separate region for each creates a constraint that the keys
>>> are
>>>> limited to the characters for region names, which are A-z/0-9/ - and _.
>>>> Everything else is out. Redis users might start asking questions why
>>> their
>>>> list named ++^^/## throws an error. Your suggestion to make it a key
>>> rather
>>>> than a region solves this. Furthermore, creating a new region every
>> time
>>> a
>>>> new Redis collection is created is going to be slow. I\u2019m not sure why a
>>>> region was created but I\u2019m sure it made sense to the developer at the
>>> time.
>>>> 7. Default Config
>>>> Can\u2019t we configure a gfsh option to default to the region types we
>> want?
>>>> Customer A will want PARTITION but Customer B will want
>>>> PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider
>>> a
>>>> geode> create region \u2014redisType=PARTITION_REDUNDANT_EXPIRATION_
>>> PERSISTENT
>>>> that makes _all_ Redis regions of that type?
>>>>
>>>>
>>>>
>>>> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hi...@yahoo.com>
>> <ma...@yahoo.com>>
>>> <mailto:
>>>> hiteshk25@yahoo.com<ma...@yahoo.com>>>> wrote:
>>>>
>>>> Current GeodeRedisAdapter implementation is based on
>>>> https://cwiki.apache.org/confluence/display/GEODE/
>>> Geode+Redis+Adapter+Proposal
>>>> .
>>>> We are looking for some feedback on Redis commands and their mapping to
>>>> geode region.
>>>>
>>>> 1. Redis Type String
>>>> a. Usage Set k1 v1
>>>> b. Current implementation creates "STRING_REGION"
>> geode-partition-region
>>>> upfront
>>>> c. This k1/v1 are geode-region key/value
>>>> d. Any feedback?
>>>>
>>>> 2. List Type
>>>> a. usage "rpush mylist A"
>>>> b. Current implementation maps each list to
>> geode-partition-region(i.e.
>>>> mylist is geode-partition-region); with the ability to get item from
>>>> head/tail
>>>> c. Feedback/vote
>>>> -- List type operation at region-entry level;
>>>> -- region-key = "mylist"
>>>> -- region-value = Arraylist (will support all redis list ops)
>>>> d. Feedback/vote: both behavior is desirable
>>>>
>>>>
>>>> 3. Hashes
>>>> a. this represents field-value or java bean object
>>>> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
>>>> c. Current implementation maps each hashes to
>>>> geode-partition-region(i.e. user1000 is geode-partition-region)
>>>> d. Feedback/vote
>>>> -- Should we map hashes to region-entry
>>>> -- region-key = user1000
>>>> -- region-value = map
>>>> -- This will provide java bean sort to behaviour with 10s of
>>>> field-value
>>>> -- Personally I would prefer this..
>>>> e. Feedback/vote: both behaviour is desirable
>>>>
>>>> 4. Sets
>>>> a. This represents unique keys in set
>>>> b. usage "sadd myset 1 2 3"
>>>> c. Current implementation maps each sadd to
>> geode-partition-region(i.e.
>>>> myset is geode-partition-region)
>>>> d. Feedback/vote
>>>> -- Should we map set to region-entry
>>>> -- region-key = myset
>>>> -- region-value = Hashset
>>>> e. Feedback/vote: both behaviour is desirable
>>>>
>>>> 5. SortedSets
>>>> a. This represents unique keys in set with score (usecase Query
>> top-10)
>>>> b. usage "zadd hackers 1940 "Alan Kay""
>>>> c. Current implementation maps each zadd to
>> geode-partition-region(i.e.
>>>> hackers is geode-partition-region)
>>>> d. Feedback/vote
>>>> -- Should we map set to region-entry
>>>> -- region-key = hackers
>>>> -- region-value = Sorted Hashset
>>>> e. Feedback/vote: both behaviour is desirable
>>>>
>>>> 6. HyperLogLogs
>>>> a. A HyperLogLog is a probabilistic data structure used in order to
>>>> count unique things (technically this is referred to estimating the
>>>> cardinality of a set).
>>>> b. usage "pfadd hll a b c d"
>>>> c. Current implementation creates "HLL_REGION" geode-partition-region
>>>> upfront
>>>> d. hll becomes region-key and value is HLL object
>>>> e. any feedback?
>>>>
>>>> 7. Default config for geode-region (vote)
>>>> a. partition region
>>>> b. 1 redundant copy
>>>> c. Persistence
>>>> d. Eviction
>>>> e. Expiration
>>>> f. ?
>>>>
>>>> 8. It seems; redis knows type(list, hashes, string ,set ..) of each
>> key.
>>>> Thus for each operation we need to make sure type of key. In current
>>>> implementation we have different region for each redis type. Thus we
>> have
>>>> another region(metaTypeRegion) which keeps type for each key. This
>> makes
>>>> any operation in geode slow as it needs to verify that type. For
>>> instance,
>>>> creating new key need to make sure its already there or not. Whether we
>>>> should allow type change or not.
>>>> a. Feedback/vote
>>>> -- type change of key
>>>> -- Can we allow two key with same name but two differnt type (as
>> it
>>>> will endup in two different geode-region)
>>>> String type "key1" in string region
>>>> HLL type "key1" in HLL region
>>>> b. any other feedback
>>>>
>>>> 9. Transactions:
>>>> a. we will not support transaction in redisAdapter as geode
>> transaction
>>>> are limited to single node.
>>>> b. feedback?
>>>>
>>>> 10. Redis COMMAND (https://redis.io/commands/command)
>>>> a. should we implement this "COMMAND" ?
>>>>
>>>> 11. Any other redis command we should consider?
>>>>
>>>>
>>>> Thanks.
>>>> Hitesh
>>>>
>>>>
>>>>
>>>
>>>
>>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Udo Kohlmeyer <uk...@pivotal.io>.
I think we are now straddling into the territory where any distributed
storage gains would be negated by the memory we require to make the
queries/lookup/operation fast.
In addition to the added memory pressure, we still face network latency
for every operation.
On 2/15/17 15:12, Dan Smith wrote:
> Deltas would help. If the regions are persistent, the whole new value does
> get written to disk though.
>
> I suppose a third option would be to store all collections in the same
> region, but store each element of the collection as a separate entry. For
> example for HSET rkey rfield rvalue would create an entry with a geode key
> of (rkey, rfield) in a REDIS_MAPS region. We could index the based on rkey.
> That might make operations on the entire collection more expensive though;
> those operations would all require using OQL queries.
>
> -Dan
>
>
>
> On Wed, Feb 15, 2017 at 2:35 PM, Galen M O'Sullivan <go...@pivotal.io>
> wrote:
>
>> On Wed, Feb 15, 2017 at 1:34 PM, Real Wes <Th...@outlook.com> wrote:
>>> Does delta propagation make worrying about frequently updated fat
>> collections moot?
>>
>> As long as we don't have collections on the order of the available RAM on a
>> single server (in which case we'd want to distribute it across multiple
>> servers), and pending testing, I think it does. Good idea, I didn't even
>> know we could do this.
>>
Re: GeodeRedisAdapter improvments/feedback
Posted by Hitesh Khamesra <hi...@yahoo.com.INVALID>.
Thanks for all your feedback.
Dan, your third option looks good ( cost of memory/oql need to consider). But this should solves popular redis use cases "Top k/ multiple leaderboards use cases, where subset of users/entity involves".
In general, Redis brings beautiful user apis which is very handful to end user. They don't need to think about distributed nature of stores.
Geode brings strong consistency, high availability(in few seconds secondary copy is available), wan-replication, Async queue and many more enterprise features.
Thus providing those apis on top of Geode should generate good value.
Thanks Again.Hitesh.
From: Dan Smith <ds...@pivotal.io>
To: dev@geode.apache.org
Sent: Wednesday, February 15, 2017 3:12 PM
Subject: Re: GeodeRedisAdapter improvments/feedback
Deltas would help. If the regions are persistent, the whole new value does
get written to disk though.
I suppose a third option would be to store all collections in the same
region, but store each element of the collection as a separate entry. For
example for HSET rkey rfield rvalue would create an entry with a geode key
of (rkey, rfield) in a REDIS_MAPS region. We could index the based on rkey.
That might make operations on the entire collection more expensive though;
those operations would all require using OQL queries.
-Dan
On Wed, Feb 15, 2017 at 2:35 PM, Galen M O'Sullivan <go...@pivotal.io>
wrote:
> On Wed, Feb 15, 2017 at 1:34 PM, Real Wes <Th...@outlook.com> wrote:
> > Does delta propagation make worrying about frequently updated fat
> collections moot?
>
> As long as we don't have collections on the order of the available RAM on a
> single server (in which case we'd want to distribute it across multiple
> servers), and pending testing, I think it does. Good idea, I didn't even
> know we could do this.
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Dan Smith <ds...@pivotal.io>.
Deltas would help. If the regions are persistent, the whole new value does
get written to disk though.
I suppose a third option would be to store all collections in the same
region, but store each element of the collection as a separate entry. For
example for HSET rkey rfield rvalue would create an entry with a geode key
of (rkey, rfield) in a REDIS_MAPS region. We could index the based on rkey.
That might make operations on the entire collection more expensive though;
those operations would all require using OQL queries.
-Dan
On Wed, Feb 15, 2017 at 2:35 PM, Galen M O'Sullivan <go...@pivotal.io>
wrote:
> On Wed, Feb 15, 2017 at 1:34 PM, Real Wes <Th...@outlook.com> wrote:
> > Does delta propagation make worrying about frequently updated fat
> collections moot?
>
> As long as we don't have collections on the order of the available RAM on a
> single server (in which case we'd want to distribute it across multiple
> servers), and pending testing, I think it does. Good idea, I didn't even
> know we could do this.
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Galen M O'Sullivan <go...@pivotal.io>.
On Wed, Feb 15, 2017 at 1:34 PM, Real Wes <Th...@outlook.com> wrote:
> Does delta propagation make worrying about frequently updated fat
collections moot?
As long as we don't have collections on the order of the available RAM on a
single server (in which case we'd want to distribute it across multiple
servers), and pending testing, I think it does. Good idea, I didn't even
know we could do this.
Re: GeodeRedisAdapter improvments/feedback
Posted by Real Wes <Th...@outlook.com>.
Does delta propagation make worrying about frequently updated fat collections moot?
On Feb 15, 2017, at 4:29 PM, Dan Smith <ds...@pivotal.io>> wrote:
Doing the spill/unspill option could be pretty tricky to implement, so you have to do a lot of fancy logic in the transition period. I think Jason's suggestion of configuring things might make more sense.
-Dan
On Wed, Feb 15, 2017 at 1:12 PM, Jason Huynh <jh...@pivotal.io>> wrote:
With the suggestion from Wes, the constraint on the names would have to
apply for both small and large. We wouldn't want the thing to explode when
it gets converted...
Is there a way to just make it configurable? If they know they want a
"large" set, somehow let them specify it. Otherwise go with the "small"
set?
On Wed, Feb 15, 2017 at 1:01 PM Real Wes <Th...@outlook.com>> wrote:
> Thinking about this, I think that the “spill”/ “unspill” option may
> actually be the best solution. If the criteria waffles back and forth
> along the threshold, well, that’s the acceptable worst case.
>
> How’s this?:
>
> 1) Create a separate region for the collection key
> - for fat collections that are updated frequently
> ADVANTAGE: speed of replication
> DISADVANTAGE: constraint on key name
>
> 2) Put the collection as an entry value:
> - for small collections and read-only fat collections
> ADVANTAGE: no need to create a separate region
>
> We would track the metrics and automatically convert based on a
> combination of frequency of updates and size.
>
> We next define what a fat collection is, such as over nnMB.
>
>
> On Feb 14, 2017, at 8:12 PM, Jason Huynh <jh...@pivotal.io><mailto:
> jhuynh@pivotal.io<ma...@pivotal.io>>> wrote:
>
> The concern about the threshold to spill over would be do you "unspill"
> over? Like what if the collection contracts under the threshold and
> teeters around the threshold. If the user can configure this size, then
> wouldn't they just know they want a "large" vs a "small?"
>
> I think Swapnil makes a good point that our value add would be that we can
> scale those structures, whereas redis can already do what the "new"
> implementation is doing.
>
>
>
> On Tue, Feb 14, 2017 at 4:59 PM Galen M O'Sullivan <go...@pivotal.io>
> <ma...@pivotal.io>>> wrote:
> If we put them in separate regions, we'll have the overhead of looking up
> in two regions added to each and every operation, and the overhead of
> creating all these regions.
>
> If we really wanted to we could have some threshold at which we spill
> collections over into their own regions, and have something like the best
> of both worlds. It's more complex, though, and I don't know how many people
> actually use truly huge collections.
>
> On Tue, Feb 14, 2017 at 4:21 PM, Hitesh Khamesra <
> hiteshk25@yahoo.com.invalid<ma...@yahoo.com.invalid>>> wrote:
>
> > Jason/Dan: Sorry to hear about that. But both of you have asked the right
> > question.
> > it depends on your use-case(item 2,3,4,5) . For example "hashes" can be
> > use to define key-value pair or java bean. In this case probably it is
> > better to keep that hash at region-entry level. But if you want to know
> > top 10 tweets which are trending then probably you want use
> > partition-region for "sorted-set".
> >
> >
> > From: Jason Huynh <jh...@pivotal.io>>>
> > To: dev@geode.apache.org<ma...@geode.apache.org>>; "
> user@geode.apache.org<ma...@geode.apache.org>>" <
> user@geode.apache.org<ma...@geode.apache.org>>>;
> > Hitesh Khamesra <hi...@yahoo.com>>>
> > Sent: Tuesday, February 14, 2017 3:15 PM
> > Subject: Re: GeodeRedisAdapter improvments/feedback
> >
> > Hi Hitesh,
> >
> > Not sure about everyone else, but I had a hard time reading this,
> however
> > I think I figured out what you were describing... the only part I still
> am
> > unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> > you want feedback and voting on whether both behaviors are desired? As
> in
> > old implementation and new implementation?
> >
> > 2,3,4) The new implementation would mean all the data for a specific
> data
> > structure is contained in a single bucket. So the individual data
> > structures are not quite scalable. How would you allow scaling of a
> single
> > data structure?
> >
> > On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com><mailto:
> TheRealWes@outlook.com<ma...@outlook.com>>> wrote:
> >
> > > In what format do you want the feedback Hitesh? For now I’ll just
> > comment:
> > >
> > > 1. Redis Type String
> > > No comments except that a future Geode value-add would be to extend the
> > > Jedis client so that the K/V’s are not compressed. In this way OQL and
> CQ
> > > will work. The tradeoff of this is that the data cannot be read by a
> > > native redis client but for Geode users it’s great. Call the new client
> > > Geodis.
> > >
> > > 2. List/ Hash/ Set/ SortedSet
> > > Creating a separate region for each creates a constraint that the keys
> > are
> > > limited to the characters for region names, which are A-z/0-9/ - and _.
> > > Everything else is out. Redis users might start asking questions why
> > their
> > > list named ++^^/## throws an error. Your suggestion to make it a key
> > rather
> > > than a region solves this. Furthermore, creating a new region every
> time
> > a
> > > new Redis collection is created is going to be slow. I’m not sure why a
> > > region was created but I’m sure it made sense to the developer at the
> > time.
> > >
> > > 7. Default Config
> > > Can’t we configure a gfsh option to default to the region types we
> want?
> > > Customer A will want PARTITION but Customer B will want
> > > PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider
> > a
> > > geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_
> > PERSISTENT
> > > that makes _all_ Redis regions of that type?
> > >
> > >
> > >
> > > On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hi...@yahoo.com>
> <ma...@yahoo.com>>
> > <mailto:
> > > hiteshk25@yahoo.com<ma...@yahoo.com>>>> wrote:
> > >
> > > Current GeodeRedisAdapter implementation is based on
> > > https://cwiki.apache.org/confluence/display/GEODE/
> > Geode+Redis+Adapter+Proposal
> > > .
> > > We are looking for some feedback on Redis commands and their mapping to
> > > geode region.
> > >
> > > 1. Redis Type String
> > > a. Usage Set k1 v1
> > > b. Current implementation creates "STRING_REGION"
> geode-partition-region
> > > upfront
> > > c. This k1/v1 are geode-region key/value
> > > d. Any feedback?
> > >
> > > 2. List Type
> > > a. usage "rpush mylist A"
> > > b. Current implementation maps each list to
> geode-partition-region(i.e.
> > > mylist is geode-partition-region); with the ability to get item from
> > > head/tail
> > > c. Feedback/vote
> > > -- List type operation at region-entry level;
> > > -- region-key = "mylist"
> > > -- region-value = Arraylist (will support all redis list ops)
> > > d. Feedback/vote: both behavior is desirable
> > >
> > >
> > > 3. Hashes
> > > a. this represents field-value or java bean object
> > > b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> > > c. Current implementation maps each hashes to
> > > geode-partition-region(i.e. user1000 is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map hashes to region-entry
> > > -- region-key = user1000
> > > -- region-value = map
> > > -- This will provide java bean sort to behaviour with 10s of
> > > field-value
> > > -- Personally I would prefer this..
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 4. Sets
> > > a. This represents unique keys in set
> > > b. usage "sadd myset 1 2 3"
> > > c. Current implementation maps each sadd to
> geode-partition-region(i.e.
> > > myset is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map set to region-entry
> > > -- region-key = myset
> > > -- region-value = Hashset
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 5. SortedSets
> > > a. This represents unique keys in set with score (usecase Query
> top-10)
> > > b. usage "zadd hackers 1940 "Alan Kay""
> > > c. Current implementation maps each zadd to
> geode-partition-region(i.e.
> > > hackers is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map set to region-entry
> > > -- region-key = hackers
> > > -- region-value = Sorted Hashset
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 6. HyperLogLogs
> > > a. A HyperLogLog is a probabilistic data structure used in order to
> > > count unique things (technically this is referred to estimating the
> > > cardinality of a set).
> > > b. usage "pfadd hll a b c d"
> > > c. Current implementation creates "HLL_REGION" geode-partition-region
> > > upfront
> > > d. hll becomes region-key and value is HLL object
> > > e. any feedback?
> > >
> > > 7. Default config for geode-region (vote)
> > > a. partition region
> > > b. 1 redundant copy
> > > c. Persistence
> > > d. Eviction
> > > e. Expiration
> > > f. ?
> > >
> > > 8. It seems; redis knows type(list, hashes, string ,set ..) of each
> key.
> > > Thus for each operation we need to make sure type of key. In current
> > > implementation we have different region for each redis type. Thus we
> have
> > > another region(metaTypeRegion) which keeps type for each key. This
> makes
> > > any operation in geode slow as it needs to verify that type. For
> > instance,
> > > creating new key need to make sure its already there or not. Whether we
> > > should allow type change or not.
> > > a. Feedback/vote
> > > -- type change of key
> > > -- Can we allow two key with same name but two differnt type (as
> it
> > > will endup in two different geode-region)
> > > String type "key1" in string region
> > > HLL type "key1" in HLL region
> > > b. any other feedback
> > >
> > > 9. Transactions:
> > > a. we will not support transaction in redisAdapter as geode
> transaction
> > > are limited to single node.
> > > b. feedback?
> > >
> > > 10. Redis COMMAND (https://redis.io/commands/command)
> > > a. should we implement this "COMMAND" ?
> > >
> > > 11. Any other redis command we should consider?
> > >
> > >
> > > Thanks.
> > > Hitesh
> > >
> > >
> > >
> >
> >
> >
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Real Wes <Th...@outlook.com>.
Does delta propagation make worrying about frequently updated fat collections moot?
On Feb 15, 2017, at 4:29 PM, Dan Smith <ds...@pivotal.io>> wrote:
Doing the spill/unspill option could be pretty tricky to implement, so you have to do a lot of fancy logic in the transition period. I think Jason's suggestion of configuring things might make more sense.
-Dan
On Wed, Feb 15, 2017 at 1:12 PM, Jason Huynh <jh...@pivotal.io>> wrote:
With the suggestion from Wes, the constraint on the names would have to
apply for both small and large. We wouldn't want the thing to explode when
it gets converted...
Is there a way to just make it configurable? If they know they want a
"large" set, somehow let them specify it. Otherwise go with the "small"
set?
On Wed, Feb 15, 2017 at 1:01 PM Real Wes <Th...@outlook.com>> wrote:
> Thinking about this, I think that the “spill”/ “unspill” option may
> actually be the best solution. If the criteria waffles back and forth
> along the threshold, well, that’s the acceptable worst case.
>
> How’s this?:
>
> 1) Create a separate region for the collection key
> - for fat collections that are updated frequently
> ADVANTAGE: speed of replication
> DISADVANTAGE: constraint on key name
>
> 2) Put the collection as an entry value:
> - for small collections and read-only fat collections
> ADVANTAGE: no need to create a separate region
>
> We would track the metrics and automatically convert based on a
> combination of frequency of updates and size.
>
> We next define what a fat collection is, such as over nnMB.
>
>
> On Feb 14, 2017, at 8:12 PM, Jason Huynh <jh...@pivotal.io><mailto:
> jhuynh@pivotal.io<ma...@pivotal.io>>> wrote:
>
> The concern about the threshold to spill over would be do you "unspill"
> over? Like what if the collection contracts under the threshold and
> teeters around the threshold. If the user can configure this size, then
> wouldn't they just know they want a "large" vs a "small?"
>
> I think Swapnil makes a good point that our value add would be that we can
> scale those structures, whereas redis can already do what the "new"
> implementation is doing.
>
>
>
> On Tue, Feb 14, 2017 at 4:59 PM Galen M O'Sullivan <go...@pivotal.io>
> <ma...@pivotal.io>>> wrote:
> If we put them in separate regions, we'll have the overhead of looking up
> in two regions added to each and every operation, and the overhead of
> creating all these regions.
>
> If we really wanted to we could have some threshold at which we spill
> collections over into their own regions, and have something like the best
> of both worlds. It's more complex, though, and I don't know how many people
> actually use truly huge collections.
>
> On Tue, Feb 14, 2017 at 4:21 PM, Hitesh Khamesra <
> hiteshk25@yahoo.com.invalid<ma...@yahoo.com.invalid>>> wrote:
>
> > Jason/Dan: Sorry to hear about that. But both of you have asked the right
> > question.
> > it depends on your use-case(item 2,3,4,5) . For example "hashes" can be
> > use to define key-value pair or java bean. In this case probably it is
> > better to keep that hash at region-entry level. But if you want to know
> > top 10 tweets which are trending then probably you want use
> > partition-region for "sorted-set".
> >
> >
> > From: Jason Huynh <jh...@pivotal.io>>>
> > To: dev@geode.apache.org<ma...@geode.apache.org>>; "
> user@geode.apache.org<ma...@geode.apache.org>>" <
> user@geode.apache.org<ma...@geode.apache.org>>>;
> > Hitesh Khamesra <hi...@yahoo.com>>>
> > Sent: Tuesday, February 14, 2017 3:15 PM
> > Subject: Re: GeodeRedisAdapter improvments/feedback
> >
> > Hi Hitesh,
> >
> > Not sure about everyone else, but I had a hard time reading this,
> however
> > I think I figured out what you were describing... the only part I still
> am
> > unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> > you want feedback and voting on whether both behaviors are desired? As
> in
> > old implementation and new implementation?
> >
> > 2,3,4) The new implementation would mean all the data for a specific
> data
> > structure is contained in a single bucket. So the individual data
> > structures are not quite scalable. How would you allow scaling of a
> single
> > data structure?
> >
> > On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com><mailto:
> TheRealWes@outlook.com<ma...@outlook.com>>> wrote:
> >
> > > In what format do you want the feedback Hitesh? For now I’ll just
> > comment:
> > >
> > > 1. Redis Type String
> > > No comments except that a future Geode value-add would be to extend the
> > > Jedis client so that the K/V’s are not compressed. In this way OQL and
> CQ
> > > will work. The tradeoff of this is that the data cannot be read by a
> > > native redis client but for Geode users it’s great. Call the new client
> > > Geodis.
> > >
> > > 2. List/ Hash/ Set/ SortedSet
> > > Creating a separate region for each creates a constraint that the keys
> > are
> > > limited to the characters for region names, which are A-z/0-9/ - and _.
> > > Everything else is out. Redis users might start asking questions why
> > their
> > > list named ++^^/## throws an error. Your suggestion to make it a key
> > rather
> > > than a region solves this. Furthermore, creating a new region every
> time
> > a
> > > new Redis collection is created is going to be slow. I’m not sure why a
> > > region was created but I’m sure it made sense to the developer at the
> > time.
> > >
> > > 7. Default Config
> > > Can’t we configure a gfsh option to default to the region types we
> want?
> > > Customer A will want PARTITION but Customer B will want
> > > PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider
> > a
> > > geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_
> > PERSISTENT
> > > that makes _all_ Redis regions of that type?
> > >
> > >
> > >
> > > On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hi...@yahoo.com>
> <ma...@yahoo.com>>
> > <mailto:
> > > hiteshk25@yahoo.com<ma...@yahoo.com>>>> wrote:
> > >
> > > Current GeodeRedisAdapter implementation is based on
> > > https://cwiki.apache.org/confluence/display/GEODE/
> > Geode+Redis+Adapter+Proposal
> > > .
> > > We are looking for some feedback on Redis commands and their mapping to
> > > geode region.
> > >
> > > 1. Redis Type String
> > > a. Usage Set k1 v1
> > > b. Current implementation creates "STRING_REGION"
> geode-partition-region
> > > upfront
> > > c. This k1/v1 are geode-region key/value
> > > d. Any feedback?
> > >
> > > 2. List Type
> > > a. usage "rpush mylist A"
> > > b. Current implementation maps each list to
> geode-partition-region(i.e.
> > > mylist is geode-partition-region); with the ability to get item from
> > > head/tail
> > > c. Feedback/vote
> > > -- List type operation at region-entry level;
> > > -- region-key = "mylist"
> > > -- region-value = Arraylist (will support all redis list ops)
> > > d. Feedback/vote: both behavior is desirable
> > >
> > >
> > > 3. Hashes
> > > a. this represents field-value or java bean object
> > > b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> > > c. Current implementation maps each hashes to
> > > geode-partition-region(i.e. user1000 is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map hashes to region-entry
> > > -- region-key = user1000
> > > -- region-value = map
> > > -- This will provide java bean sort to behaviour with 10s of
> > > field-value
> > > -- Personally I would prefer this..
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 4. Sets
> > > a. This represents unique keys in set
> > > b. usage "sadd myset 1 2 3"
> > > c. Current implementation maps each sadd to
> geode-partition-region(i.e.
> > > myset is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map set to region-entry
> > > -- region-key = myset
> > > -- region-value = Hashset
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 5. SortedSets
> > > a. This represents unique keys in set with score (usecase Query
> top-10)
> > > b. usage "zadd hackers 1940 "Alan Kay""
> > > c. Current implementation maps each zadd to
> geode-partition-region(i.e.
> > > hackers is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map set to region-entry
> > > -- region-key = hackers
> > > -- region-value = Sorted Hashset
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 6. HyperLogLogs
> > > a. A HyperLogLog is a probabilistic data structure used in order to
> > > count unique things (technically this is referred to estimating the
> > > cardinality of a set).
> > > b. usage "pfadd hll a b c d"
> > > c. Current implementation creates "HLL_REGION" geode-partition-region
> > > upfront
> > > d. hll becomes region-key and value is HLL object
> > > e. any feedback?
> > >
> > > 7. Default config for geode-region (vote)
> > > a. partition region
> > > b. 1 redundant copy
> > > c. Persistence
> > > d. Eviction
> > > e. Expiration
> > > f. ?
> > >
> > > 8. It seems; redis knows type(list, hashes, string ,set ..) of each
> key.
> > > Thus for each operation we need to make sure type of key. In current
> > > implementation we have different region for each redis type. Thus we
> have
> > > another region(metaTypeRegion) which keeps type for each key. This
> makes
> > > any operation in geode slow as it needs to verify that type. For
> > instance,
> > > creating new key need to make sure its already there or not. Whether we
> > > should allow type change or not.
> > > a. Feedback/vote
> > > -- type change of key
> > > -- Can we allow two key with same name but two differnt type (as
> it
> > > will endup in two different geode-region)
> > > String type "key1" in string region
> > > HLL type "key1" in HLL region
> > > b. any other feedback
> > >
> > > 9. Transactions:
> > > a. we will not support transaction in redisAdapter as geode
> transaction
> > > are limited to single node.
> > > b. feedback?
> > >
> > > 10. Redis COMMAND (https://redis.io/commands/command)
> > > a. should we implement this "COMMAND" ?
> > >
> > > 11. Any other redis command we should consider?
> > >
> > >
> > > Thanks.
> > > Hitesh
> > >
> > >
> > >
> >
> >
> >
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Dan Smith <ds...@pivotal.io>.
Doing the spill/unspill option could be pretty tricky to implement, so you
have to do a lot of fancy logic in the transition period. I think Jason's
suggestion of configuring things might make more sense.
-Dan
On Wed, Feb 15, 2017 at 1:12 PM, Jason Huynh <jh...@pivotal.io> wrote:
> With the suggestion from Wes, the constraint on the names would have to
> apply for both small and large. We wouldn't want the thing to explode when
> it gets converted...
>
> Is there a way to just make it configurable? If they know they want a
> "large" set, somehow let them specify it. Otherwise go with the "small"
> set?
>
> On Wed, Feb 15, 2017 at 1:01 PM Real Wes <Th...@outlook.com> wrote:
>
> > Thinking about this, I think that the “spill”/ “unspill” option may
> > actually be the best solution. If the criteria waffles back and forth
> > along the threshold, well, that’s the acceptable worst case.
> >
> > How’s this?:
> >
> > 1) Create a separate region for the collection key
> > - for fat collections that are updated frequently
> > ADVANTAGE: speed of replication
> > DISADVANTAGE: constraint on key name
> >
> > 2) Put the collection as an entry value:
> > - for small collections and read-only fat collections
> > ADVANTAGE: no need to create a separate region
> >
> > We would track the metrics and automatically convert based on a
> > combination of frequency of updates and size.
> >
> > We next define what a fat collection is, such as over nnMB.
> >
> >
> > On Feb 14, 2017, at 8:12 PM, Jason Huynh <jhuynh@pivotal.io<mailto:
> > jhuynh@pivotal.io>> wrote:
> >
> > The concern about the threshold to spill over would be do you "unspill"
> > over? Like what if the collection contracts under the threshold and
> > teeters around the threshold. If the user can configure this size, then
> > wouldn't they just know they want a "large" vs a "small?"
> >
> > I think Swapnil makes a good point that our value add would be that we
> can
> > scale those structures, whereas redis can already do what the "new"
> > implementation is doing.
> >
> >
> >
> > On Tue, Feb 14, 2017 at 4:59 PM Galen M O'Sullivan <
> gosullivan@pivotal.io
> > <ma...@pivotal.io>> wrote:
> > If we put them in separate regions, we'll have the overhead of looking up
> > in two regions added to each and every operation, and the overhead of
> > creating all these regions.
> >
> > If we really wanted to we could have some threshold at which we spill
> > collections over into their own regions, and have something like the best
> > of both worlds. It's more complex, though, and I don't know how many
> people
> > actually use truly huge collections.
> >
> > On Tue, Feb 14, 2017 at 4:21 PM, Hitesh Khamesra <
> > hiteshk25@yahoo.com.invalid<ma...@yahoo.com.invalid>> wrote:
> >
> > > Jason/Dan: Sorry to hear about that. But both of you have asked the
> right
> > > question.
> > > it depends on your use-case(item 2,3,4,5) . For example "hashes" can be
> > > use to define key-value pair or java bean. In this case probably it is
> > > better to keep that hash at region-entry level. But if you want to
> know
> > > top 10 tweets which are trending then probably you want use
> > > partition-region for "sorted-set".
> > >
> > >
> > > From: Jason Huynh <jh...@pivotal.io>>
> > > To: dev@geode.apache.org<ma...@geode.apache.org>; "
> > user@geode.apache.org<ma...@geode.apache.org>" <
> > user@geode.apache.org<ma...@geode.apache.org>>;
> > > Hitesh Khamesra <hi...@yahoo.com>>
> > > Sent: Tuesday, February 14, 2017 3:15 PM
> > > Subject: Re: GeodeRedisAdapter improvments/feedback
> > >
> > > Hi Hitesh,
> > >
> > > Not sure about everyone else, but I had a hard time reading this,
> > however
> > > I think I figured out what you were describing... the only part I still
> > am
> > > unsure about is Feedback/vote: both behaviour is desirable. Do you
> mean
> > > you want feedback and voting on whether both behaviors are desired? As
> > in
> > > old implementation and new implementation?
> > >
> > > 2,3,4) The new implementation would mean all the data for a specific
> > data
> > > structure is contained in a single bucket. So the individual data
> > > structures are not quite scalable. How would you allow scaling of a
> > single
> > > data structure?
> > >
> > > On Tue, Feb 14, 2017 at 3:05 PM Real Wes <TheRealWes@outlook.com<
> mailto:
> > TheRealWes@outlook.com>> wrote:
> > >
> > > > In what format do you want the feedback Hitesh? For now I’ll just
> > > comment:
> > > >
> > > > 1. Redis Type String
> > > > No comments except that a future Geode value-add would be to extend
> the
> > > > Jedis client so that the K/V’s are not compressed. In this way OQL
> and
> > CQ
> > > > will work. The tradeoff of this is that the data cannot be read by a
> > > > native redis client but for Geode users it’s great. Call the new
> client
> > > > Geodis.
> > > >
> > > > 2. List/ Hash/ Set/ SortedSet
> > > > Creating a separate region for each creates a constraint that the
> keys
> > > are
> > > > limited to the characters for region names, which are A-z/0-9/ - and
> _.
> > > > Everything else is out. Redis users might start asking questions why
> > > their
> > > > list named ++^^/## throws an error. Your suggestion to make it a key
> > > rather
> > > > than a region solves this. Furthermore, creating a new region every
> > time
> > > a
> > > > new Redis collection is created is going to be slow. I’m not sure
> why a
> > > > region was created but I’m sure it made sense to the developer at the
> > > time.
> > > >
> > > > 7. Default Config
> > > > Can’t we configure a gfsh option to default to the region types we
> > want?
> > > > Customer A will want PARTITION but Customer B will want
> > > > PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can
> consider
> > > a
> > > > geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_
> > > PERSISTENT
> > > > that makes _all_ Redis regions of that type?
> > > >
> > > >
> > > >
> > > > On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com
> > <ma...@yahoo.com>
> > > <mailto:
> > > > hiteshk25@yahoo.com<ma...@yahoo.com>>> wrote:
> > > >
> > > > Current GeodeRedisAdapter implementation is based on
> > > > https://cwiki.apache.org/confluence/display/GEODE/
> > > Geode+Redis+Adapter+Proposal
> > > > .
> > > > We are looking for some feedback on Redis commands and their mapping
> to
> > > > geode region.
> > > >
> > > > 1. Redis Type String
> > > > a. Usage Set k1 v1
> > > > b. Current implementation creates "STRING_REGION"
> > geode-partition-region
> > > > upfront
> > > > c. This k1/v1 are geode-region key/value
> > > > d. Any feedback?
> > > >
> > > > 2. List Type
> > > > a. usage "rpush mylist A"
> > > > b. Current implementation maps each list to
> > geode-partition-region(i.e.
> > > > mylist is geode-partition-region); with the ability to get item from
> > > > head/tail
> > > > c. Feedback/vote
> > > > -- List type operation at region-entry level;
> > > > -- region-key = "mylist"
> > > > -- region-value = Arraylist (will support all redis list ops)
> > > > d. Feedback/vote: both behavior is desirable
> > > >
> > > >
> > > > 3. Hashes
> > > > a. this represents field-value or java bean object
> > > > b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> > > > c. Current implementation maps each hashes to
> > > > geode-partition-region(i.e. user1000 is geode-partition-region)
> > > > d. Feedback/vote
> > > > -- Should we map hashes to region-entry
> > > > -- region-key = user1000
> > > > -- region-value = map
> > > > -- This will provide java bean sort to behaviour with 10s of
> > > > field-value
> > > > -- Personally I would prefer this..
> > > > e. Feedback/vote: both behaviour is desirable
> > > >
> > > > 4. Sets
> > > > a. This represents unique keys in set
> > > > b. usage "sadd myset 1 2 3"
> > > > c. Current implementation maps each sadd to
> > geode-partition-region(i.e.
> > > > myset is geode-partition-region)
> > > > d. Feedback/vote
> > > > -- Should we map set to region-entry
> > > > -- region-key = myset
> > > > -- region-value = Hashset
> > > > e. Feedback/vote: both behaviour is desirable
> > > >
> > > > 5. SortedSets
> > > > a. This represents unique keys in set with score (usecase Query
> > top-10)
> > > > b. usage "zadd hackers 1940 "Alan Kay""
> > > > c. Current implementation maps each zadd to
> > geode-partition-region(i.e.
> > > > hackers is geode-partition-region)
> > > > d. Feedback/vote
> > > > -- Should we map set to region-entry
> > > > -- region-key = hackers
> > > > -- region-value = Sorted Hashset
> > > > e. Feedback/vote: both behaviour is desirable
> > > >
> > > > 6. HyperLogLogs
> > > > a. A HyperLogLog is a probabilistic data structure used in order to
> > > > count unique things (technically this is referred to estimating the
> > > > cardinality of a set).
> > > > b. usage "pfadd hll a b c d"
> > > > c. Current implementation creates "HLL_REGION"
> geode-partition-region
> > > > upfront
> > > > d. hll becomes region-key and value is HLL object
> > > > e. any feedback?
> > > >
> > > > 7. Default config for geode-region (vote)
> > > > a. partition region
> > > > b. 1 redundant copy
> > > > c. Persistence
> > > > d. Eviction
> > > > e. Expiration
> > > > f. ?
> > > >
> > > > 8. It seems; redis knows type(list, hashes, string ,set ..) of each
> > key.
> > > > Thus for each operation we need to make sure type of key. In current
> > > > implementation we have different region for each redis type. Thus we
> > have
> > > > another region(metaTypeRegion) which keeps type for each key. This
> > makes
> > > > any operation in geode slow as it needs to verify that type. For
> > > instance,
> > > > creating new key need to make sure its already there or not. Whether
> we
> > > > should allow type change or not.
> > > > a. Feedback/vote
> > > > -- type change of key
> > > > -- Can we allow two key with same name but two differnt type (as
> > it
> > > > will endup in two different geode-region)
> > > > String type "key1" in string region
> > > > HLL type "key1" in HLL region
> > > > b. any other feedback
> > > >
> > > > 9. Transactions:
> > > > a. we will not support transaction in redisAdapter as geode
> > transaction
> > > > are limited to single node.
> > > > b. feedback?
> > > >
> > > > 10. Redis COMMAND (https://redis.io/commands/command)
> > > > a. should we implement this "COMMAND" ?
> > > >
> > > > 11. Any other redis command we should consider?
> > > >
> > > >
> > > > Thanks.
> > > > Hitesh
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Dan Smith <ds...@pivotal.io>.
Doing the spill/unspill option could be pretty tricky to implement, so you
have to do a lot of fancy logic in the transition period. I think Jason's
suggestion of configuring things might make more sense.
-Dan
On Wed, Feb 15, 2017 at 1:12 PM, Jason Huynh <jh...@pivotal.io> wrote:
> With the suggestion from Wes, the constraint on the names would have to
> apply for both small and large. We wouldn't want the thing to explode when
> it gets converted...
>
> Is there a way to just make it configurable? If they know they want a
> "large" set, somehow let them specify it. Otherwise go with the "small"
> set?
>
> On Wed, Feb 15, 2017 at 1:01 PM Real Wes <Th...@outlook.com> wrote:
>
> > Thinking about this, I think that the “spill”/ “unspill” option may
> > actually be the best solution. If the criteria waffles back and forth
> > along the threshold, well, that’s the acceptable worst case.
> >
> > How’s this?:
> >
> > 1) Create a separate region for the collection key
> > - for fat collections that are updated frequently
> > ADVANTAGE: speed of replication
> > DISADVANTAGE: constraint on key name
> >
> > 2) Put the collection as an entry value:
> > - for small collections and read-only fat collections
> > ADVANTAGE: no need to create a separate region
> >
> > We would track the metrics and automatically convert based on a
> > combination of frequency of updates and size.
> >
> > We next define what a fat collection is, such as over nnMB.
> >
> >
> > On Feb 14, 2017, at 8:12 PM, Jason Huynh <jhuynh@pivotal.io<mailto:
> > jhuynh@pivotal.io>> wrote:
> >
> > The concern about the threshold to spill over would be do you "unspill"
> > over? Like what if the collection contracts under the threshold and
> > teeters around the threshold. If the user can configure this size, then
> > wouldn't they just know they want a "large" vs a "small?"
> >
> > I think Swapnil makes a good point that our value add would be that we
> can
> > scale those structures, whereas redis can already do what the "new"
> > implementation is doing.
> >
> >
> >
> > On Tue, Feb 14, 2017 at 4:59 PM Galen M O'Sullivan <
> gosullivan@pivotal.io
> > <ma...@pivotal.io>> wrote:
> > If we put them in separate regions, we'll have the overhead of looking up
> > in two regions added to each and every operation, and the overhead of
> > creating all these regions.
> >
> > If we really wanted to we could have some threshold at which we spill
> > collections over into their own regions, and have something like the best
> > of both worlds. It's more complex, though, and I don't know how many
> people
> > actually use truly huge collections.
> >
> > On Tue, Feb 14, 2017 at 4:21 PM, Hitesh Khamesra <
> > hiteshk25@yahoo.com.invalid<ma...@yahoo.com.invalid>> wrote:
> >
> > > Jason/Dan: Sorry to hear about that. But both of you have asked the
> right
> > > question.
> > > it depends on your use-case(item 2,3,4,5) . For example "hashes" can be
> > > use to define key-value pair or java bean. In this case probably it is
> > > better to keep that hash at region-entry level. But if you want to
> know
> > > top 10 tweets which are trending then probably you want use
> > > partition-region for "sorted-set".
> > >
> > >
> > > From: Jason Huynh <jh...@pivotal.io>>
> > > To: dev@geode.apache.org<ma...@geode.apache.org>; "
> > user@geode.apache.org<ma...@geode.apache.org>" <
> > user@geode.apache.org<ma...@geode.apache.org>>;
> > > Hitesh Khamesra <hi...@yahoo.com>>
> > > Sent: Tuesday, February 14, 2017 3:15 PM
> > > Subject: Re: GeodeRedisAdapter improvments/feedback
> > >
> > > Hi Hitesh,
> > >
> > > Not sure about everyone else, but I had a hard time reading this,
> > however
> > > I think I figured out what you were describing... the only part I still
> > am
> > > unsure about is Feedback/vote: both behaviour is desirable. Do you
> mean
> > > you want feedback and voting on whether both behaviors are desired? As
> > in
> > > old implementation and new implementation?
> > >
> > > 2,3,4) The new implementation would mean all the data for a specific
> > data
> > > structure is contained in a single bucket. So the individual data
> > > structures are not quite scalable. How would you allow scaling of a
> > single
> > > data structure?
> > >
> > > On Tue, Feb 14, 2017 at 3:05 PM Real Wes <TheRealWes@outlook.com<
> mailto:
> > TheRealWes@outlook.com>> wrote:
> > >
> > > > In what format do you want the feedback Hitesh? For now I’ll just
> > > comment:
> > > >
> > > > 1. Redis Type String
> > > > No comments except that a future Geode value-add would be to extend
> the
> > > > Jedis client so that the K/V’s are not compressed. In this way OQL
> and
> > CQ
> > > > will work. The tradeoff of this is that the data cannot be read by a
> > > > native redis client but for Geode users it’s great. Call the new
> client
> > > > Geodis.
> > > >
> > > > 2. List/ Hash/ Set/ SortedSet
> > > > Creating a separate region for each creates a constraint that the
> keys
> > > are
> > > > limited to the characters for region names, which are A-z/0-9/ - and
> _.
> > > > Everything else is out. Redis users might start asking questions why
> > > their
> > > > list named ++^^/## throws an error. Your suggestion to make it a key
> > > rather
> > > > than a region solves this. Furthermore, creating a new region every
> > time
> > > a
> > > > new Redis collection is created is going to be slow. I’m not sure
> why a
> > > > region was created but I’m sure it made sense to the developer at the
> > > time.
> > > >
> > > > 7. Default Config
> > > > Can’t we configure a gfsh option to default to the region types we
> > want?
> > > > Customer A will want PARTITION but Customer B will want
> > > > PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can
> consider
> > > a
> > > > geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_
> > > PERSISTENT
> > > > that makes _all_ Redis regions of that type?
> > > >
> > > >
> > > >
> > > > On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com
> > <ma...@yahoo.com>
> > > <mailto:
> > > > hiteshk25@yahoo.com<ma...@yahoo.com>>> wrote:
> > > >
> > > > Current GeodeRedisAdapter implementation is based on
> > > > https://cwiki.apache.org/confluence/display/GEODE/
> > > Geode+Redis+Adapter+Proposal
> > > > .
> > > > We are looking for some feedback on Redis commands and their mapping
> to
> > > > geode region.
> > > >
> > > > 1. Redis Type String
> > > > a. Usage Set k1 v1
> > > > b. Current implementation creates "STRING_REGION"
> > geode-partition-region
> > > > upfront
> > > > c. This k1/v1 are geode-region key/value
> > > > d. Any feedback?
> > > >
> > > > 2. List Type
> > > > a. usage "rpush mylist A"
> > > > b. Current implementation maps each list to
> > geode-partition-region(i.e.
> > > > mylist is geode-partition-region); with the ability to get item from
> > > > head/tail
> > > > c. Feedback/vote
> > > > -- List type operation at region-entry level;
> > > > -- region-key = "mylist"
> > > > -- region-value = Arraylist (will support all redis list ops)
> > > > d. Feedback/vote: both behavior is desirable
> > > >
> > > >
> > > > 3. Hashes
> > > > a. this represents field-value or java bean object
> > > > b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> > > > c. Current implementation maps each hashes to
> > > > geode-partition-region(i.e. user1000 is geode-partition-region)
> > > > d. Feedback/vote
> > > > -- Should we map hashes to region-entry
> > > > -- region-key = user1000
> > > > -- region-value = map
> > > > -- This will provide java bean sort to behaviour with 10s of
> > > > field-value
> > > > -- Personally I would prefer this..
> > > > e. Feedback/vote: both behaviour is desirable
> > > >
> > > > 4. Sets
> > > > a. This represents unique keys in set
> > > > b. usage "sadd myset 1 2 3"
> > > > c. Current implementation maps each sadd to
> > geode-partition-region(i.e.
> > > > myset is geode-partition-region)
> > > > d. Feedback/vote
> > > > -- Should we map set to region-entry
> > > > -- region-key = myset
> > > > -- region-value = Hashset
> > > > e. Feedback/vote: both behaviour is desirable
> > > >
> > > > 5. SortedSets
> > > > a. This represents unique keys in set with score (usecase Query
> > top-10)
> > > > b. usage "zadd hackers 1940 "Alan Kay""
> > > > c. Current implementation maps each zadd to
> > geode-partition-region(i.e.
> > > > hackers is geode-partition-region)
> > > > d. Feedback/vote
> > > > -- Should we map set to region-entry
> > > > -- region-key = hackers
> > > > -- region-value = Sorted Hashset
> > > > e. Feedback/vote: both behaviour is desirable
> > > >
> > > > 6. HyperLogLogs
> > > > a. A HyperLogLog is a probabilistic data structure used in order to
> > > > count unique things (technically this is referred to estimating the
> > > > cardinality of a set).
> > > > b. usage "pfadd hll a b c d"
> > > > c. Current implementation creates "HLL_REGION"
> geode-partition-region
> > > > upfront
> > > > d. hll becomes region-key and value is HLL object
> > > > e. any feedback?
> > > >
> > > > 7. Default config for geode-region (vote)
> > > > a. partition region
> > > > b. 1 redundant copy
> > > > c. Persistence
> > > > d. Eviction
> > > > e. Expiration
> > > > f. ?
> > > >
> > > > 8. It seems; redis knows type(list, hashes, string ,set ..) of each
> > key.
> > > > Thus for each operation we need to make sure type of key. In current
> > > > implementation we have different region for each redis type. Thus we
> > have
> > > > another region(metaTypeRegion) which keeps type for each key. This
> > makes
> > > > any operation in geode slow as it needs to verify that type. For
> > > instance,
> > > > creating new key need to make sure its already there or not. Whether
> we
> > > > should allow type change or not.
> > > > a. Feedback/vote
> > > > -- type change of key
> > > > -- Can we allow two key with same name but two differnt type (as
> > it
> > > > will endup in two different geode-region)
> > > > String type "key1" in string region
> > > > HLL type "key1" in HLL region
> > > > b. any other feedback
> > > >
> > > > 9. Transactions:
> > > > a. we will not support transaction in redisAdapter as geode
> > transaction
> > > > are limited to single node.
> > > > b. feedback?
> > > >
> > > > 10. Redis COMMAND (https://redis.io/commands/command)
> > > > a. should we implement this "COMMAND" ?
> > > >
> > > > 11. Any other redis command we should consider?
> > > >
> > > >
> > > > Thanks.
> > > > Hitesh
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Jason Huynh <jh...@pivotal.io>.
With the suggestion from Wes, the constraint on the names would have to
apply for both small and large. We wouldn't want the thing to explode when
it gets converted...
Is there a way to just make it configurable? If they know they want a
"large" set, somehow let them specify it. Otherwise go with the "small"
set?
On Wed, Feb 15, 2017 at 1:01 PM Real Wes <Th...@outlook.com> wrote:
> Thinking about this, I think that the “spill”/ “unspill” option may
> actually be the best solution. If the criteria waffles back and forth
> along the threshold, well, that’s the acceptable worst case.
>
> How’s this?:
>
> 1) Create a separate region for the collection key
> - for fat collections that are updated frequently
> ADVANTAGE: speed of replication
> DISADVANTAGE: constraint on key name
>
> 2) Put the collection as an entry value:
> - for small collections and read-only fat collections
> ADVANTAGE: no need to create a separate region
>
> We would track the metrics and automatically convert based on a
> combination of frequency of updates and size.
>
> We next define what a fat collection is, such as over nnMB.
>
>
> On Feb 14, 2017, at 8:12 PM, Jason Huynh <jhuynh@pivotal.io<mailto:
> jhuynh@pivotal.io>> wrote:
>
> The concern about the threshold to spill over would be do you "unspill"
> over? Like what if the collection contracts under the threshold and
> teeters around the threshold. If the user can configure this size, then
> wouldn't they just know they want a "large" vs a "small?"
>
> I think Swapnil makes a good point that our value add would be that we can
> scale those structures, whereas redis can already do what the "new"
> implementation is doing.
>
>
>
> On Tue, Feb 14, 2017 at 4:59 PM Galen M O'Sullivan <gosullivan@pivotal.io
> <ma...@pivotal.io>> wrote:
> If we put them in separate regions, we'll have the overhead of looking up
> in two regions added to each and every operation, and the overhead of
> creating all these regions.
>
> If we really wanted to we could have some threshold at which we spill
> collections over into their own regions, and have something like the best
> of both worlds. It's more complex, though, and I don't know how many people
> actually use truly huge collections.
>
> On Tue, Feb 14, 2017 at 4:21 PM, Hitesh Khamesra <
> hiteshk25@yahoo.com.invalid<ma...@yahoo.com.invalid>> wrote:
>
> > Jason/Dan: Sorry to hear about that. But both of you have asked the right
> > question.
> > it depends on your use-case(item 2,3,4,5) . For example "hashes" can be
> > use to define key-value pair or java bean. In this case probably it is
> > better to keep that hash at region-entry level. But if you want to know
> > top 10 tweets which are trending then probably you want use
> > partition-region for "sorted-set".
> >
> >
> > From: Jason Huynh <jh...@pivotal.io>>
> > To: dev@geode.apache.org<ma...@geode.apache.org>; "
> user@geode.apache.org<ma...@geode.apache.org>" <
> user@geode.apache.org<ma...@geode.apache.org>>;
> > Hitesh Khamesra <hi...@yahoo.com>>
> > Sent: Tuesday, February 14, 2017 3:15 PM
> > Subject: Re: GeodeRedisAdapter improvments/feedback
> >
> > Hi Hitesh,
> >
> > Not sure about everyone else, but I had a hard time reading this,
> however
> > I think I figured out what you were describing... the only part I still
> am
> > unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> > you want feedback and voting on whether both behaviors are desired? As
> in
> > old implementation and new implementation?
> >
> > 2,3,4) The new implementation would mean all the data for a specific
> data
> > structure is contained in a single bucket. So the individual data
> > structures are not quite scalable. How would you allow scaling of a
> single
> > data structure?
> >
> > On Tue, Feb 14, 2017 at 3:05 PM Real Wes <TheRealWes@outlook.com<mailto:
> TheRealWes@outlook.com>> wrote:
> >
> > > In what format do you want the feedback Hitesh? For now I’ll just
> > comment:
> > >
> > > 1. Redis Type String
> > > No comments except that a future Geode value-add would be to extend the
> > > Jedis client so that the K/V’s are not compressed. In this way OQL and
> CQ
> > > will work. The tradeoff of this is that the data cannot be read by a
> > > native redis client but for Geode users it’s great. Call the new client
> > > Geodis.
> > >
> > > 2. List/ Hash/ Set/ SortedSet
> > > Creating a separate region for each creates a constraint that the keys
> > are
> > > limited to the characters for region names, which are A-z/0-9/ - and _.
> > > Everything else is out. Redis users might start asking questions why
> > their
> > > list named ++^^/## throws an error. Your suggestion to make it a key
> > rather
> > > than a region solves this. Furthermore, creating a new region every
> time
> > a
> > > new Redis collection is created is going to be slow. I’m not sure why a
> > > region was created but I’m sure it made sense to the developer at the
> > time.
> > >
> > > 7. Default Config
> > > Can’t we configure a gfsh option to default to the region types we
> want?
> > > Customer A will want PARTITION but Customer B will want
> > > PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider
> > a
> > > geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_
> > PERSISTENT
> > > that makes _all_ Redis regions of that type?
> > >
> > >
> > >
> > > On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com
> <ma...@yahoo.com>
> > <mailto:
> > > hiteshk25@yahoo.com<ma...@yahoo.com>>> wrote:
> > >
> > > Current GeodeRedisAdapter implementation is based on
> > > https://cwiki.apache.org/confluence/display/GEODE/
> > Geode+Redis+Adapter+Proposal
> > > .
> > > We are looking for some feedback on Redis commands and their mapping to
> > > geode region.
> > >
> > > 1. Redis Type String
> > > a. Usage Set k1 v1
> > > b. Current implementation creates "STRING_REGION"
> geode-partition-region
> > > upfront
> > > c. This k1/v1 are geode-region key/value
> > > d. Any feedback?
> > >
> > > 2. List Type
> > > a. usage "rpush mylist A"
> > > b. Current implementation maps each list to
> geode-partition-region(i.e.
> > > mylist is geode-partition-region); with the ability to get item from
> > > head/tail
> > > c. Feedback/vote
> > > -- List type operation at region-entry level;
> > > -- region-key = "mylist"
> > > -- region-value = Arraylist (will support all redis list ops)
> > > d. Feedback/vote: both behavior is desirable
> > >
> > >
> > > 3. Hashes
> > > a. this represents field-value or java bean object
> > > b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> > > c. Current implementation maps each hashes to
> > > geode-partition-region(i.e. user1000 is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map hashes to region-entry
> > > -- region-key = user1000
> > > -- region-value = map
> > > -- This will provide java bean sort to behaviour with 10s of
> > > field-value
> > > -- Personally I would prefer this..
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 4. Sets
> > > a. This represents unique keys in set
> > > b. usage "sadd myset 1 2 3"
> > > c. Current implementation maps each sadd to
> geode-partition-region(i.e.
> > > myset is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map set to region-entry
> > > -- region-key = myset
> > > -- region-value = Hashset
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 5. SortedSets
> > > a. This represents unique keys in set with score (usecase Query
> top-10)
> > > b. usage "zadd hackers 1940 "Alan Kay""
> > > c. Current implementation maps each zadd to
> geode-partition-region(i.e.
> > > hackers is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map set to region-entry
> > > -- region-key = hackers
> > > -- region-value = Sorted Hashset
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 6. HyperLogLogs
> > > a. A HyperLogLog is a probabilistic data structure used in order to
> > > count unique things (technically this is referred to estimating the
> > > cardinality of a set).
> > > b. usage "pfadd hll a b c d"
> > > c. Current implementation creates "HLL_REGION" geode-partition-region
> > > upfront
> > > d. hll becomes region-key and value is HLL object
> > > e. any feedback?
> > >
> > > 7. Default config for geode-region (vote)
> > > a. partition region
> > > b. 1 redundant copy
> > > c. Persistence
> > > d. Eviction
> > > e. Expiration
> > > f. ?
> > >
> > > 8. It seems; redis knows type(list, hashes, string ,set ..) of each
> key.
> > > Thus for each operation we need to make sure type of key. In current
> > > implementation we have different region for each redis type. Thus we
> have
> > > another region(metaTypeRegion) which keeps type for each key. This
> makes
> > > any operation in geode slow as it needs to verify that type. For
> > instance,
> > > creating new key need to make sure its already there or not. Whether we
> > > should allow type change or not.
> > > a. Feedback/vote
> > > -- type change of key
> > > -- Can we allow two key with same name but two differnt type (as
> it
> > > will endup in two different geode-region)
> > > String type "key1" in string region
> > > HLL type "key1" in HLL region
> > > b. any other feedback
> > >
> > > 9. Transactions:
> > > a. we will not support transaction in redisAdapter as geode
> transaction
> > > are limited to single node.
> > > b. feedback?
> > >
> > > 10. Redis COMMAND (https://redis.io/commands/command)
> > > a. should we implement this "COMMAND" ?
> > >
> > > 11. Any other redis command we should consider?
> > >
> > >
> > > Thanks.
> > > Hitesh
> > >
> > >
> > >
> >
> >
> >
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Jason Huynh <jh...@pivotal.io>.
With the suggestion from Wes, the constraint on the names would have to
apply for both small and large. We wouldn't want the thing to explode when
it gets converted...
Is there a way to just make it configurable? If they know they want a
"large" set, somehow let them specify it. Otherwise go with the "small"
set?
On Wed, Feb 15, 2017 at 1:01 PM Real Wes <Th...@outlook.com> wrote:
> Thinking about this, I think that the “spill”/ “unspill” option may
> actually be the best solution. If the criteria waffles back and forth
> along the threshold, well, that’s the acceptable worst case.
>
> How’s this?:
>
> 1) Create a separate region for the collection key
> - for fat collections that are updated frequently
> ADVANTAGE: speed of replication
> DISADVANTAGE: constraint on key name
>
> 2) Put the collection as an entry value:
> - for small collections and read-only fat collections
> ADVANTAGE: no need to create a separate region
>
> We would track the metrics and automatically convert based on a
> combination of frequency of updates and size.
>
> We next define what a fat collection is, such as over nnMB.
>
>
> On Feb 14, 2017, at 8:12 PM, Jason Huynh <jhuynh@pivotal.io<mailto:
> jhuynh@pivotal.io>> wrote:
>
> The concern about the threshold to spill over would be do you "unspill"
> over? Like what if the collection contracts under the threshold and
> teeters around the threshold. If the user can configure this size, then
> wouldn't they just know they want a "large" vs a "small?"
>
> I think Swapnil makes a good point that our value add would be that we can
> scale those structures, whereas redis can already do what the "new"
> implementation is doing.
>
>
>
> On Tue, Feb 14, 2017 at 4:59 PM Galen M O'Sullivan <gosullivan@pivotal.io
> <ma...@pivotal.io>> wrote:
> If we put them in separate regions, we'll have the overhead of looking up
> in two regions added to each and every operation, and the overhead of
> creating all these regions.
>
> If we really wanted to we could have some threshold at which we spill
> collections over into their own regions, and have something like the best
> of both worlds. It's more complex, though, and I don't know how many people
> actually use truly huge collections.
>
> On Tue, Feb 14, 2017 at 4:21 PM, Hitesh Khamesra <
> hiteshk25@yahoo.com.invalid<ma...@yahoo.com.invalid>> wrote:
>
> > Jason/Dan: Sorry to hear about that. But both of you have asked the right
> > question.
> > it depends on your use-case(item 2,3,4,5) . For example "hashes" can be
> > use to define key-value pair or java bean. In this case probably it is
> > better to keep that hash at region-entry level. But if you want to know
> > top 10 tweets which are trending then probably you want use
> > partition-region for "sorted-set".
> >
> >
> > From: Jason Huynh <jh...@pivotal.io>>
> > To: dev@geode.apache.org<ma...@geode.apache.org>; "
> user@geode.apache.org<ma...@geode.apache.org>" <
> user@geode.apache.org<ma...@geode.apache.org>>;
> > Hitesh Khamesra <hi...@yahoo.com>>
> > Sent: Tuesday, February 14, 2017 3:15 PM
> > Subject: Re: GeodeRedisAdapter improvments/feedback
> >
> > Hi Hitesh,
> >
> > Not sure about everyone else, but I had a hard time reading this,
> however
> > I think I figured out what you were describing... the only part I still
> am
> > unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> > you want feedback and voting on whether both behaviors are desired? As
> in
> > old implementation and new implementation?
> >
> > 2,3,4) The new implementation would mean all the data for a specific
> data
> > structure is contained in a single bucket. So the individual data
> > structures are not quite scalable. How would you allow scaling of a
> single
> > data structure?
> >
> > On Tue, Feb 14, 2017 at 3:05 PM Real Wes <TheRealWes@outlook.com<mailto:
> TheRealWes@outlook.com>> wrote:
> >
> > > In what format do you want the feedback Hitesh? For now I’ll just
> > comment:
> > >
> > > 1. Redis Type String
> > > No comments except that a future Geode value-add would be to extend the
> > > Jedis client so that the K/V’s are not compressed. In this way OQL and
> CQ
> > > will work. The tradeoff of this is that the data cannot be read by a
> > > native redis client but for Geode users it’s great. Call the new client
> > > Geodis.
> > >
> > > 2. List/ Hash/ Set/ SortedSet
> > > Creating a separate region for each creates a constraint that the keys
> > are
> > > limited to the characters for region names, which are A-z/0-9/ - and _.
> > > Everything else is out. Redis users might start asking questions why
> > their
> > > list named ++^^/## throws an error. Your suggestion to make it a key
> > rather
> > > than a region solves this. Furthermore, creating a new region every
> time
> > a
> > > new Redis collection is created is going to be slow. I’m not sure why a
> > > region was created but I’m sure it made sense to the developer at the
> > time.
> > >
> > > 7. Default Config
> > > Can’t we configure a gfsh option to default to the region types we
> want?
> > > Customer A will want PARTITION but Customer B will want
> > > PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider
> > a
> > > geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_
> > PERSISTENT
> > > that makes _all_ Redis regions of that type?
> > >
> > >
> > >
> > > On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com
> <ma...@yahoo.com>
> > <mailto:
> > > hiteshk25@yahoo.com<ma...@yahoo.com>>> wrote:
> > >
> > > Current GeodeRedisAdapter implementation is based on
> > > https://cwiki.apache.org/confluence/display/GEODE/
> > Geode+Redis+Adapter+Proposal
> > > .
> > > We are looking for some feedback on Redis commands and their mapping to
> > > geode region.
> > >
> > > 1. Redis Type String
> > > a. Usage Set k1 v1
> > > b. Current implementation creates "STRING_REGION"
> geode-partition-region
> > > upfront
> > > c. This k1/v1 are geode-region key/value
> > > d. Any feedback?
> > >
> > > 2. List Type
> > > a. usage "rpush mylist A"
> > > b. Current implementation maps each list to
> geode-partition-region(i.e.
> > > mylist is geode-partition-region); with the ability to get item from
> > > head/tail
> > > c. Feedback/vote
> > > -- List type operation at region-entry level;
> > > -- region-key = "mylist"
> > > -- region-value = Arraylist (will support all redis list ops)
> > > d. Feedback/vote: both behavior is desirable
> > >
> > >
> > > 3. Hashes
> > > a. this represents field-value or java bean object
> > > b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> > > c. Current implementation maps each hashes to
> > > geode-partition-region(i.e. user1000 is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map hashes to region-entry
> > > -- region-key = user1000
> > > -- region-value = map
> > > -- This will provide java bean sort to behaviour with 10s of
> > > field-value
> > > -- Personally I would prefer this..
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 4. Sets
> > > a. This represents unique keys in set
> > > b. usage "sadd myset 1 2 3"
> > > c. Current implementation maps each sadd to
> geode-partition-region(i.e.
> > > myset is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map set to region-entry
> > > -- region-key = myset
> > > -- region-value = Hashset
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 5. SortedSets
> > > a. This represents unique keys in set with score (usecase Query
> top-10)
> > > b. usage "zadd hackers 1940 "Alan Kay""
> > > c. Current implementation maps each zadd to
> geode-partition-region(i.e.
> > > hackers is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map set to region-entry
> > > -- region-key = hackers
> > > -- region-value = Sorted Hashset
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 6. HyperLogLogs
> > > a. A HyperLogLog is a probabilistic data structure used in order to
> > > count unique things (technically this is referred to estimating the
> > > cardinality of a set).
> > > b. usage "pfadd hll a b c d"
> > > c. Current implementation creates "HLL_REGION" geode-partition-region
> > > upfront
> > > d. hll becomes region-key and value is HLL object
> > > e. any feedback?
> > >
> > > 7. Default config for geode-region (vote)
> > > a. partition region
> > > b. 1 redundant copy
> > > c. Persistence
> > > d. Eviction
> > > e. Expiration
> > > f. ?
> > >
> > > 8. It seems; redis knows type(list, hashes, string ,set ..) of each
> key.
> > > Thus for each operation we need to make sure type of key. In current
> > > implementation we have different region for each redis type. Thus we
> have
> > > another region(metaTypeRegion) which keeps type for each key. This
> makes
> > > any operation in geode slow as it needs to verify that type. For
> > instance,
> > > creating new key need to make sure its already there or not. Whether we
> > > should allow type change or not.
> > > a. Feedback/vote
> > > -- type change of key
> > > -- Can we allow two key with same name but two differnt type (as
> it
> > > will endup in two different geode-region)
> > > String type "key1" in string region
> > > HLL type "key1" in HLL region
> > > b. any other feedback
> > >
> > > 9. Transactions:
> > > a. we will not support transaction in redisAdapter as geode
> transaction
> > > are limited to single node.
> > > b. feedback?
> > >
> > > 10. Redis COMMAND (https://redis.io/commands/command)
> > > a. should we implement this "COMMAND" ?
> > >
> > > 11. Any other redis command we should consider?
> > >
> > >
> > > Thanks.
> > > Hitesh
> > >
> > >
> > >
> >
> >
> >
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Real Wes <Th...@outlook.com>.
Thinking about this, I think that the “spill”/ “unspill” option may actually be the best solution. If the criteria waffles back and forth along the threshold, well, that’s the acceptable worst case.
How’s this?:
1) Create a separate region for the collection key
- for fat collections that are updated frequently
ADVANTAGE: speed of replication
DISADVANTAGE: constraint on key name
2) Put the collection as an entry value:
- for small collections and read-only fat collections
ADVANTAGE: no need to create a separate region
We would track the metrics and automatically convert based on a combination of frequency of updates and size.
We next define what a fat collection is, such as over nnMB.
On Feb 14, 2017, at 8:12 PM, Jason Huynh <jh...@pivotal.io>> wrote:
The concern about the threshold to spill over would be do you "unspill" over? Like what if the collection contracts under the threshold and teeters around the threshold. If the user can configure this size, then wouldn't they just know they want a "large" vs a "small?"
I think Swapnil makes a good point that our value add would be that we can scale those structures, whereas redis can already do what the "new" implementation is doing.
On Tue, Feb 14, 2017 at 4:59 PM Galen M O'Sullivan <go...@pivotal.io>> wrote:
If we put them in separate regions, we'll have the overhead of looking up
in two regions added to each and every operation, and the overhead of
creating all these regions.
If we really wanted to we could have some threshold at which we spill
collections over into their own regions, and have something like the best
of both worlds. It's more complex, though, and I don't know how many people
actually use truly huge collections.
On Tue, Feb 14, 2017 at 4:21 PM, Hitesh Khamesra <
hiteshk25@yahoo.com.invalid<ma...@yahoo.com.invalid>> wrote:
> Jason/Dan: Sorry to hear about that. But both of you have asked the right
> question.
> it depends on your use-case(item 2,3,4,5) . For example "hashes" can be
> use to define key-value pair or java bean. In this case probably it is
> better to keep that hash at region-entry level. But if you want to know
> top 10 tweets which are trending then probably you want use
> partition-region for "sorted-set".
>
>
> From: Jason Huynh <jh...@pivotal.io>>
> To: dev@geode.apache.org<ma...@geode.apache.org>; "user@geode.apache.org<ma...@geode.apache.org>" <us...@geode.apache.org>>;
> Hitesh Khamesra <hi...@yahoo.com>>
> Sent: Tuesday, February 14, 2017 3:15 PM
> Subject: Re: GeodeRedisAdapter improvments/feedback
>
> Hi Hitesh,
>
> Not sure about everyone else, but I had a hard time reading this, however
> I think I figured out what you were describing... the only part I still am
> unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> you want feedback and voting on whether both behaviors are desired? As in
> old implementation and new implementation?
>
> 2,3,4) The new implementation would mean all the data for a specific data
> structure is contained in a single bucket. So the individual data
> structures are not quite scalable. How would you allow scaling of a single
> data structure?
>
> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com>> wrote:
>
> > In what format do you want the feedback Hitesh? For now I’ll just
> comment:
> >
> > 1. Redis Type String
> > No comments except that a future Geode value-add would be to extend the
> > Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> > will work. The tradeoff of this is that the data cannot be read by a
> > native redis client but for Geode users it’s great. Call the new client
> > Geodis.
> >
> > 2. List/ Hash/ Set/ SortedSet
> > Creating a separate region for each creates a constraint that the keys
> are
> > limited to the characters for region names, which are A-z/0-9/ - and _.
> > Everything else is out. Redis users might start asking questions why
> their
> > list named ++^^/## throws an error. Your suggestion to make it a key
> rather
> > than a region solves this. Furthermore, creating a new region every time
> a
> > new Redis collection is created is going to be slow. I’m not sure why a
> > region was created but I’m sure it made sense to the developer at the
> time.
> >
> > 7. Default Config
> > Can’t we configure a gfsh option to default to the region types we want?
> > Customer A will want PARTITION but Customer B will want
> > PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider
> a
> > geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_
> PERSISTENT
> > that makes _all_ Redis regions of that type?
> >
> >
> >
> > On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hi...@yahoo.com>
> <mailto:
> > hiteshk25@yahoo.com<ma...@yahoo.com>>> wrote:
> >
> > Current GeodeRedisAdapter implementation is based on
> > https://cwiki.apache.org/confluence/display/GEODE/
> Geode+Redis+Adapter+Proposal
> > .
> > We are looking for some feedback on Redis commands and their mapping to
> > geode region.
> >
> > 1. Redis Type String
> > a. Usage Set k1 v1
> > b. Current implementation creates "STRING_REGION" geode-partition-region
> > upfront
> > c. This k1/v1 are geode-region key/value
> > d. Any feedback?
> >
> > 2. List Type
> > a. usage "rpush mylist A"
> > b. Current implementation maps each list to geode-partition-region(i.e.
> > mylist is geode-partition-region); with the ability to get item from
> > head/tail
> > c. Feedback/vote
> > -- List type operation at region-entry level;
> > -- region-key = "mylist"
> > -- region-value = Arraylist (will support all redis list ops)
> > d. Feedback/vote: both behavior is desirable
> >
> >
> > 3. Hashes
> > a. this represents field-value or java bean object
> > b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> > c. Current implementation maps each hashes to
> > geode-partition-region(i.e. user1000 is geode-partition-region)
> > d. Feedback/vote
> > -- Should we map hashes to region-entry
> > -- region-key = user1000
> > -- region-value = map
> > -- This will provide java bean sort to behaviour with 10s of
> > field-value
> > -- Personally I would prefer this..
> > e. Feedback/vote: both behaviour is desirable
> >
> > 4. Sets
> > a. This represents unique keys in set
> > b. usage "sadd myset 1 2 3"
> > c. Current implementation maps each sadd to geode-partition-region(i.e.
> > myset is geode-partition-region)
> > d. Feedback/vote
> > -- Should we map set to region-entry
> > -- region-key = myset
> > -- region-value = Hashset
> > e. Feedback/vote: both behaviour is desirable
> >
> > 5. SortedSets
> > a. This represents unique keys in set with score (usecase Query top-10)
> > b. usage "zadd hackers 1940 "Alan Kay""
> > c. Current implementation maps each zadd to geode-partition-region(i.e.
> > hackers is geode-partition-region)
> > d. Feedback/vote
> > -- Should we map set to region-entry
> > -- region-key = hackers
> > -- region-value = Sorted Hashset
> > e. Feedback/vote: both behaviour is desirable
> >
> > 6. HyperLogLogs
> > a. A HyperLogLog is a probabilistic data structure used in order to
> > count unique things (technically this is referred to estimating the
> > cardinality of a set).
> > b. usage "pfadd hll a b c d"
> > c. Current implementation creates "HLL_REGION" geode-partition-region
> > upfront
> > d. hll becomes region-key and value is HLL object
> > e. any feedback?
> >
> > 7. Default config for geode-region (vote)
> > a. partition region
> > b. 1 redundant copy
> > c. Persistence
> > d. Eviction
> > e. Expiration
> > f. ?
> >
> > 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> > Thus for each operation we need to make sure type of key. In current
> > implementation we have different region for each redis type. Thus we have
> > another region(metaTypeRegion) which keeps type for each key. This makes
> > any operation in geode slow as it needs to verify that type. For
> instance,
> > creating new key need to make sure its already there or not. Whether we
> > should allow type change or not.
> > a. Feedback/vote
> > -- type change of key
> > -- Can we allow two key with same name but two differnt type (as it
> > will endup in two different geode-region)
> > String type "key1" in string region
> > HLL type "key1" in HLL region
> > b. any other feedback
> >
> > 9. Transactions:
> > a. we will not support transaction in redisAdapter as geode transaction
> > are limited to single node.
> > b. feedback?
> >
> > 10. Redis COMMAND (https://redis.io/commands/command)
> > a. should we implement this "COMMAND" ?
> >
> > 11. Any other redis command we should consider?
> >
> >
> > Thanks.
> > Hitesh
> >
> >
> >
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Real Wes <Th...@outlook.com>.
Thinking about this, I think that the “spill”/ “unspill” option may actually be the best solution. If the criteria waffles back and forth along the threshold, well, that’s the acceptable worst case.
How’s this?:
1) Create a separate region for the collection key
- for fat collections that are updated frequently
ADVANTAGE: speed of replication
DISADVANTAGE: constraint on key name
2) Put the collection as an entry value:
- for small collections and read-only fat collections
ADVANTAGE: no need to create a separate region
We would track the metrics and automatically convert based on a combination of frequency of updates and size.
We next define what a fat collection is, such as over nnMB.
On Feb 14, 2017, at 8:12 PM, Jason Huynh <jh...@pivotal.io>> wrote:
The concern about the threshold to spill over would be do you "unspill" over? Like what if the collection contracts under the threshold and teeters around the threshold. If the user can configure this size, then wouldn't they just know they want a "large" vs a "small?"
I think Swapnil makes a good point that our value add would be that we can scale those structures, whereas redis can already do what the "new" implementation is doing.
On Tue, Feb 14, 2017 at 4:59 PM Galen M O'Sullivan <go...@pivotal.io>> wrote:
If we put them in separate regions, we'll have the overhead of looking up
in two regions added to each and every operation, and the overhead of
creating all these regions.
If we really wanted to we could have some threshold at which we spill
collections over into their own regions, and have something like the best
of both worlds. It's more complex, though, and I don't know how many people
actually use truly huge collections.
On Tue, Feb 14, 2017 at 4:21 PM, Hitesh Khamesra <
hiteshk25@yahoo.com.invalid<ma...@yahoo.com.invalid>> wrote:
> Jason/Dan: Sorry to hear about that. But both of you have asked the right
> question.
> it depends on your use-case(item 2,3,4,5) . For example "hashes" can be
> use to define key-value pair or java bean. In this case probably it is
> better to keep that hash at region-entry level. But if you want to know
> top 10 tweets which are trending then probably you want use
> partition-region for "sorted-set".
>
>
> From: Jason Huynh <jh...@pivotal.io>>
> To: dev@geode.apache.org<ma...@geode.apache.org>; "user@geode.apache.org<ma...@geode.apache.org>" <us...@geode.apache.org>>;
> Hitesh Khamesra <hi...@yahoo.com>>
> Sent: Tuesday, February 14, 2017 3:15 PM
> Subject: Re: GeodeRedisAdapter improvments/feedback
>
> Hi Hitesh,
>
> Not sure about everyone else, but I had a hard time reading this, however
> I think I figured out what you were describing... the only part I still am
> unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> you want feedback and voting on whether both behaviors are desired? As in
> old implementation and new implementation?
>
> 2,3,4) The new implementation would mean all the data for a specific data
> structure is contained in a single bucket. So the individual data
> structures are not quite scalable. How would you allow scaling of a single
> data structure?
>
> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com>> wrote:
>
> > In what format do you want the feedback Hitesh? For now I’ll just
> comment:
> >
> > 1. Redis Type String
> > No comments except that a future Geode value-add would be to extend the
> > Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> > will work. The tradeoff of this is that the data cannot be read by a
> > native redis client but for Geode users it’s great. Call the new client
> > Geodis.
> >
> > 2. List/ Hash/ Set/ SortedSet
> > Creating a separate region for each creates a constraint that the keys
> are
> > limited to the characters for region names, which are A-z/0-9/ - and _.
> > Everything else is out. Redis users might start asking questions why
> their
> > list named ++^^/## throws an error. Your suggestion to make it a key
> rather
> > than a region solves this. Furthermore, creating a new region every time
> a
> > new Redis collection is created is going to be slow. I’m not sure why a
> > region was created but I’m sure it made sense to the developer at the
> time.
> >
> > 7. Default Config
> > Can’t we configure a gfsh option to default to the region types we want?
> > Customer A will want PARTITION but Customer B will want
> > PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider
> a
> > geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_
> PERSISTENT
> > that makes _all_ Redis regions of that type?
> >
> >
> >
> > On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hi...@yahoo.com>
> <mailto:
> > hiteshk25@yahoo.com<ma...@yahoo.com>>> wrote:
> >
> > Current GeodeRedisAdapter implementation is based on
> > https://cwiki.apache.org/confluence/display/GEODE/
> Geode+Redis+Adapter+Proposal
> > .
> > We are looking for some feedback on Redis commands and their mapping to
> > geode region.
> >
> > 1. Redis Type String
> > a. Usage Set k1 v1
> > b. Current implementation creates "STRING_REGION" geode-partition-region
> > upfront
> > c. This k1/v1 are geode-region key/value
> > d. Any feedback?
> >
> > 2. List Type
> > a. usage "rpush mylist A"
> > b. Current implementation maps each list to geode-partition-region(i.e.
> > mylist is geode-partition-region); with the ability to get item from
> > head/tail
> > c. Feedback/vote
> > -- List type operation at region-entry level;
> > -- region-key = "mylist"
> > -- region-value = Arraylist (will support all redis list ops)
> > d. Feedback/vote: both behavior is desirable
> >
> >
> > 3. Hashes
> > a. this represents field-value or java bean object
> > b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> > c. Current implementation maps each hashes to
> > geode-partition-region(i.e. user1000 is geode-partition-region)
> > d. Feedback/vote
> > -- Should we map hashes to region-entry
> > -- region-key = user1000
> > -- region-value = map
> > -- This will provide java bean sort to behaviour with 10s of
> > field-value
> > -- Personally I would prefer this..
> > e. Feedback/vote: both behaviour is desirable
> >
> > 4. Sets
> > a. This represents unique keys in set
> > b. usage "sadd myset 1 2 3"
> > c. Current implementation maps each sadd to geode-partition-region(i.e.
> > myset is geode-partition-region)
> > d. Feedback/vote
> > -- Should we map set to region-entry
> > -- region-key = myset
> > -- region-value = Hashset
> > e. Feedback/vote: both behaviour is desirable
> >
> > 5. SortedSets
> > a. This represents unique keys in set with score (usecase Query top-10)
> > b. usage "zadd hackers 1940 "Alan Kay""
> > c. Current implementation maps each zadd to geode-partition-region(i.e.
> > hackers is geode-partition-region)
> > d. Feedback/vote
> > -- Should we map set to region-entry
> > -- region-key = hackers
> > -- region-value = Sorted Hashset
> > e. Feedback/vote: both behaviour is desirable
> >
> > 6. HyperLogLogs
> > a. A HyperLogLog is a probabilistic data structure used in order to
> > count unique things (technically this is referred to estimating the
> > cardinality of a set).
> > b. usage "pfadd hll a b c d"
> > c. Current implementation creates "HLL_REGION" geode-partition-region
> > upfront
> > d. hll becomes region-key and value is HLL object
> > e. any feedback?
> >
> > 7. Default config for geode-region (vote)
> > a. partition region
> > b. 1 redundant copy
> > c. Persistence
> > d. Eviction
> > e. Expiration
> > f. ?
> >
> > 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> > Thus for each operation we need to make sure type of key. In current
> > implementation we have different region for each redis type. Thus we have
> > another region(metaTypeRegion) which keeps type for each key. This makes
> > any operation in geode slow as it needs to verify that type. For
> instance,
> > creating new key need to make sure its already there or not. Whether we
> > should allow type change or not.
> > a. Feedback/vote
> > -- type change of key
> > -- Can we allow two key with same name but two differnt type (as it
> > will endup in two different geode-region)
> > String type "key1" in string region
> > HLL type "key1" in HLL region
> > b. any other feedback
> >
> > 9. Transactions:
> > a. we will not support transaction in redisAdapter as geode transaction
> > are limited to single node.
> > b. feedback?
> >
> > 10. Redis COMMAND (https://redis.io/commands/command)
> > a. should we implement this "COMMAND" ?
> >
> > 11. Any other redis command we should consider?
> >
> >
> > Thanks.
> > Hitesh
> >
> >
> >
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Jason Huynh <jh...@pivotal.io>.
The concern about the threshold to spill over would be do you "unspill"
over? Like what if the collection contracts under the threshold and
teeters around the threshold. If the user can configure this size, then
wouldn't they just know they want a "large" vs a "small?"
I think Swapnil makes a good point that our value add would be that we can
scale those structures, whereas redis can already do what the "new"
implementation is doing.
On Tue, Feb 14, 2017 at 4:59 PM Galen M O'Sullivan <go...@pivotal.io>
wrote:
> If we put them in separate regions, we'll have the overhead of looking up
> in two regions added to each and every operation, and the overhead of
> creating all these regions.
>
> If we really wanted to we could have some threshold at which we spill
> collections over into their own regions, and have something like the best
> of both worlds. It's more complex, though, and I don't know how many people
> actually use truly huge collections.
>
> On Tue, Feb 14, 2017 at 4:21 PM, Hitesh Khamesra <
> hiteshk25@yahoo.com.invalid> wrote:
>
> > Jason/Dan: Sorry to hear about that. But both of you have asked the right
> > question.
> > it depends on your use-case(item 2,3,4,5) . For example "hashes" can be
> > use to define key-value pair or java bean. In this case probably it is
> > better to keep that hash at region-entry level. But if you want to know
> > top 10 tweets which are trending then probably you want use
> > partition-region for "sorted-set".
> >
> >
> > From: Jason Huynh <jh...@pivotal.io>
> > To: dev@geode.apache.org; "user@geode.apache.org" <
> user@geode.apache.org>;
> > Hitesh Khamesra <hi...@yahoo.com>
> > Sent: Tuesday, February 14, 2017 3:15 PM
> > Subject: Re: GeodeRedisAdapter improvments/feedback
> >
> > Hi Hitesh,
> >
> > Not sure about everyone else, but I had a hard time reading this,
> however
> > I think I figured out what you were describing... the only part I still
> am
> > unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> > you want feedback and voting on whether both behaviors are desired? As
> in
> > old implementation and new implementation?
> >
> > 2,3,4) The new implementation would mean all the data for a specific
> data
> > structure is contained in a single bucket. So the individual data
> > structures are not quite scalable. How would you allow scaling of a
> single
> > data structure?
> >
> > On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
> >
> > > In what format do you want the feedback Hitesh? For now I’ll just
> > comment:
> > >
> > > 1. Redis Type String
> > > No comments except that a future Geode value-add would be to extend the
> > > Jedis client so that the K/V’s are not compressed. In this way OQL and
> CQ
> > > will work. The tradeoff of this is that the data cannot be read by a
> > > native redis client but for Geode users it’s great. Call the new client
> > > Geodis.
> > >
> > > 2. List/ Hash/ Set/ SortedSet
> > > Creating a separate region for each creates a constraint that the keys
> > are
> > > limited to the characters for region names, which are A-z/0-9/ - and _.
> > > Everything else is out. Redis users might start asking questions why
> > their
> > > list named ++^^/## throws an error. Your suggestion to make it a key
> > rather
> > > than a region solves this. Furthermore, creating a new region every
> time
> > a
> > > new Redis collection is created is going to be slow. I’m not sure why a
> > > region was created but I’m sure it made sense to the developer at the
> > time.
> > >
> > > 7. Default Config
> > > Can’t we configure a gfsh option to default to the region types we
> want?
> > > Customer A will want PARTITION but Customer B will want
> > > PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider
> > a
> > > geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_
> > PERSISTENT
> > > that makes _all_ Redis regions of that type?
> > >
> > >
> > >
> > > On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com
> > <mailto:
> > > hiteshk25@yahoo.com>> wrote:
> > >
> > > Current GeodeRedisAdapter implementation is based on
> > > https://cwiki.apache.org/confluence/display/GEODE/
> > Geode+Redis+Adapter+Proposal
> > > .
> > > We are looking for some feedback on Redis commands and their mapping to
> > > geode region.
> > >
> > > 1. Redis Type String
> > > a. Usage Set k1 v1
> > > b. Current implementation creates "STRING_REGION"
> geode-partition-region
> > > upfront
> > > c. This k1/v1 are geode-region key/value
> > > d. Any feedback?
> > >
> > > 2. List Type
> > > a. usage "rpush mylist A"
> > > b. Current implementation maps each list to
> geode-partition-region(i.e.
> > > mylist is geode-partition-region); with the ability to get item from
> > > head/tail
> > > c. Feedback/vote
> > > -- List type operation at region-entry level;
> > > -- region-key = "mylist"
> > > -- region-value = Arraylist (will support all redis list ops)
> > > d. Feedback/vote: both behavior is desirable
> > >
> > >
> > > 3. Hashes
> > > a. this represents field-value or java bean object
> > > b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> > > c. Current implementation maps each hashes to
> > > geode-partition-region(i.e. user1000 is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map hashes to region-entry
> > > -- region-key = user1000
> > > -- region-value = map
> > > -- This will provide java bean sort to behaviour with 10s of
> > > field-value
> > > -- Personally I would prefer this..
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 4. Sets
> > > a. This represents unique keys in set
> > > b. usage "sadd myset 1 2 3"
> > > c. Current implementation maps each sadd to
> geode-partition-region(i.e.
> > > myset is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map set to region-entry
> > > -- region-key = myset
> > > -- region-value = Hashset
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 5. SortedSets
> > > a. This represents unique keys in set with score (usecase Query
> top-10)
> > > b. usage "zadd hackers 1940 "Alan Kay""
> > > c. Current implementation maps each zadd to
> geode-partition-region(i.e.
> > > hackers is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map set to region-entry
> > > -- region-key = hackers
> > > -- region-value = Sorted Hashset
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 6. HyperLogLogs
> > > a. A HyperLogLog is a probabilistic data structure used in order to
> > > count unique things (technically this is referred to estimating the
> > > cardinality of a set).
> > > b. usage "pfadd hll a b c d"
> > > c. Current implementation creates "HLL_REGION" geode-partition-region
> > > upfront
> > > d. hll becomes region-key and value is HLL object
> > > e. any feedback?
> > >
> > > 7. Default config for geode-region (vote)
> > > a. partition region
> > > b. 1 redundant copy
> > > c. Persistence
> > > d. Eviction
> > > e. Expiration
> > > f. ?
> > >
> > > 8. It seems; redis knows type(list, hashes, string ,set ..) of each
> key.
> > > Thus for each operation we need to make sure type of key. In current
> > > implementation we have different region for each redis type. Thus we
> have
> > > another region(metaTypeRegion) which keeps type for each key. This
> makes
> > > any operation in geode slow as it needs to verify that type. For
> > instance,
> > > creating new key need to make sure its already there or not. Whether we
> > > should allow type change or not.
> > > a. Feedback/vote
> > > -- type change of key
> > > -- Can we allow two key with same name but two differnt type (as
> it
> > > will endup in two different geode-region)
> > > String type "key1" in string region
> > > HLL type "key1" in HLL region
> > > b. any other feedback
> > >
> > > 9. Transactions:
> > > a. we will not support transaction in redisAdapter as geode
> transaction
> > > are limited to single node.
> > > b. feedback?
> > >
> > > 10. Redis COMMAND (https://redis.io/commands/command)
> > > a. should we implement this "COMMAND" ?
> > >
> > > 11. Any other redis command we should consider?
> > >
> > >
> > > Thanks.
> > > Hitesh
> > >
> > >
> > >
> >
> >
> >
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Jason Huynh <jh...@pivotal.io>.
The concern about the threshold to spill over would be do you "unspill"
over? Like what if the collection contracts under the threshold and
teeters around the threshold. If the user can configure this size, then
wouldn't they just know they want a "large" vs a "small?"
I think Swapnil makes a good point that our value add would be that we can
scale those structures, whereas redis can already do what the "new"
implementation is doing.
On Tue, Feb 14, 2017 at 4:59 PM Galen M O'Sullivan <go...@pivotal.io>
wrote:
> If we put them in separate regions, we'll have the overhead of looking up
> in two regions added to each and every operation, and the overhead of
> creating all these regions.
>
> If we really wanted to we could have some threshold at which we spill
> collections over into their own regions, and have something like the best
> of both worlds. It's more complex, though, and I don't know how many people
> actually use truly huge collections.
>
> On Tue, Feb 14, 2017 at 4:21 PM, Hitesh Khamesra <
> hiteshk25@yahoo.com.invalid> wrote:
>
> > Jason/Dan: Sorry to hear about that. But both of you have asked the right
> > question.
> > it depends on your use-case(item 2,3,4,5) . For example "hashes" can be
> > use to define key-value pair or java bean. In this case probably it is
> > better to keep that hash at region-entry level. But if you want to know
> > top 10 tweets which are trending then probably you want use
> > partition-region for "sorted-set".
> >
> >
> > From: Jason Huynh <jh...@pivotal.io>
> > To: dev@geode.apache.org; "user@geode.apache.org" <
> user@geode.apache.org>;
> > Hitesh Khamesra <hi...@yahoo.com>
> > Sent: Tuesday, February 14, 2017 3:15 PM
> > Subject: Re: GeodeRedisAdapter improvments/feedback
> >
> > Hi Hitesh,
> >
> > Not sure about everyone else, but I had a hard time reading this,
> however
> > I think I figured out what you were describing... the only part I still
> am
> > unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> > you want feedback and voting on whether both behaviors are desired? As
> in
> > old implementation and new implementation?
> >
> > 2,3,4) The new implementation would mean all the data for a specific
> data
> > structure is contained in a single bucket. So the individual data
> > structures are not quite scalable. How would you allow scaling of a
> single
> > data structure?
> >
> > On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
> >
> > > In what format do you want the feedback Hitesh? For now I’ll just
> > comment:
> > >
> > > 1. Redis Type String
> > > No comments except that a future Geode value-add would be to extend the
> > > Jedis client so that the K/V’s are not compressed. In this way OQL and
> CQ
> > > will work. The tradeoff of this is that the data cannot be read by a
> > > native redis client but for Geode users it’s great. Call the new client
> > > Geodis.
> > >
> > > 2. List/ Hash/ Set/ SortedSet
> > > Creating a separate region for each creates a constraint that the keys
> > are
> > > limited to the characters for region names, which are A-z/0-9/ - and _.
> > > Everything else is out. Redis users might start asking questions why
> > their
> > > list named ++^^/## throws an error. Your suggestion to make it a key
> > rather
> > > than a region solves this. Furthermore, creating a new region every
> time
> > a
> > > new Redis collection is created is going to be slow. I’m not sure why a
> > > region was created but I’m sure it made sense to the developer at the
> > time.
> > >
> > > 7. Default Config
> > > Can’t we configure a gfsh option to default to the region types we
> want?
> > > Customer A will want PARTITION but Customer B will want
> > > PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider
> > a
> > > geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_
> > PERSISTENT
> > > that makes _all_ Redis regions of that type?
> > >
> > >
> > >
> > > On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com
> > <mailto:
> > > hiteshk25@yahoo.com>> wrote:
> > >
> > > Current GeodeRedisAdapter implementation is based on
> > > https://cwiki.apache.org/confluence/display/GEODE/
> > Geode+Redis+Adapter+Proposal
> > > .
> > > We are looking for some feedback on Redis commands and their mapping to
> > > geode region.
> > >
> > > 1. Redis Type String
> > > a. Usage Set k1 v1
> > > b. Current implementation creates "STRING_REGION"
> geode-partition-region
> > > upfront
> > > c. This k1/v1 are geode-region key/value
> > > d. Any feedback?
> > >
> > > 2. List Type
> > > a. usage "rpush mylist A"
> > > b. Current implementation maps each list to
> geode-partition-region(i.e.
> > > mylist is geode-partition-region); with the ability to get item from
> > > head/tail
> > > c. Feedback/vote
> > > -- List type operation at region-entry level;
> > > -- region-key = "mylist"
> > > -- region-value = Arraylist (will support all redis list ops)
> > > d. Feedback/vote: both behavior is desirable
> > >
> > >
> > > 3. Hashes
> > > a. this represents field-value or java bean object
> > > b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> > > c. Current implementation maps each hashes to
> > > geode-partition-region(i.e. user1000 is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map hashes to region-entry
> > > -- region-key = user1000
> > > -- region-value = map
> > > -- This will provide java bean sort to behaviour with 10s of
> > > field-value
> > > -- Personally I would prefer this..
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 4. Sets
> > > a. This represents unique keys in set
> > > b. usage "sadd myset 1 2 3"
> > > c. Current implementation maps each sadd to
> geode-partition-region(i.e.
> > > myset is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map set to region-entry
> > > -- region-key = myset
> > > -- region-value = Hashset
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 5. SortedSets
> > > a. This represents unique keys in set with score (usecase Query
> top-10)
> > > b. usage "zadd hackers 1940 "Alan Kay""
> > > c. Current implementation maps each zadd to
> geode-partition-region(i.e.
> > > hackers is geode-partition-region)
> > > d. Feedback/vote
> > > -- Should we map set to region-entry
> > > -- region-key = hackers
> > > -- region-value = Sorted Hashset
> > > e. Feedback/vote: both behaviour is desirable
> > >
> > > 6. HyperLogLogs
> > > a. A HyperLogLog is a probabilistic data structure used in order to
> > > count unique things (technically this is referred to estimating the
> > > cardinality of a set).
> > > b. usage "pfadd hll a b c d"
> > > c. Current implementation creates "HLL_REGION" geode-partition-region
> > > upfront
> > > d. hll becomes region-key and value is HLL object
> > > e. any feedback?
> > >
> > > 7. Default config for geode-region (vote)
> > > a. partition region
> > > b. 1 redundant copy
> > > c. Persistence
> > > d. Eviction
> > > e. Expiration
> > > f. ?
> > >
> > > 8. It seems; redis knows type(list, hashes, string ,set ..) of each
> key.
> > > Thus for each operation we need to make sure type of key. In current
> > > implementation we have different region for each redis type. Thus we
> have
> > > another region(metaTypeRegion) which keeps type for each key. This
> makes
> > > any operation in geode slow as it needs to verify that type. For
> > instance,
> > > creating new key need to make sure its already there or not. Whether we
> > > should allow type change or not.
> > > a. Feedback/vote
> > > -- type change of key
> > > -- Can we allow two key with same name but two differnt type (as
> it
> > > will endup in two different geode-region)
> > > String type "key1" in string region
> > > HLL type "key1" in HLL region
> > > b. any other feedback
> > >
> > > 9. Transactions:
> > > a. we will not support transaction in redisAdapter as geode
> transaction
> > > are limited to single node.
> > > b. feedback?
> > >
> > > 10. Redis COMMAND (https://redis.io/commands/command)
> > > a. should we implement this "COMMAND" ?
> > >
> > > 11. Any other redis command we should consider?
> > >
> > >
> > > Thanks.
> > > Hitesh
> > >
> > >
> > >
> >
> >
> >
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Galen M O'Sullivan <go...@pivotal.io>.
If we put them in separate regions, we'll have the overhead of looking up
in two regions added to each and every operation, and the overhead of
creating all these regions.
If we really wanted to we could have some threshold at which we spill
collections over into their own regions, and have something like the best
of both worlds. It's more complex, though, and I don't know how many people
actually use truly huge collections.
On Tue, Feb 14, 2017 at 4:21 PM, Hitesh Khamesra <
hiteshk25@yahoo.com.invalid> wrote:
> Jason/Dan: Sorry to hear about that. But both of you have asked the right
> question.
> it depends on your use-case(item 2,3,4,5) . For example "hashes" can be
> use to define key-value pair or java bean. In this case probably it is
> better to keep that hash at region-entry level. But if you want to know
> top 10 tweets which are trending then probably you want use
> partition-region for "sorted-set".
>
>
> From: Jason Huynh <jh...@pivotal.io>
> To: dev@geode.apache.org; "user@geode.apache.org" <us...@geode.apache.org>;
> Hitesh Khamesra <hi...@yahoo.com>
> Sent: Tuesday, February 14, 2017 3:15 PM
> Subject: Re: GeodeRedisAdapter improvments/feedback
>
> Hi Hitesh,
>
> Not sure about everyone else, but I had a hard time reading this, however
> I think I figured out what you were describing... the only part I still am
> unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> you want feedback and voting on whether both behaviors are desired? As in
> old implementation and new implementation?
>
> 2,3,4) The new implementation would mean all the data for a specific data
> structure is contained in a single bucket. So the individual data
> structures are not quite scalable. How would you allow scaling of a single
> data structure?
>
> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
>
> > In what format do you want the feedback Hitesh? For now I’ll just
> comment:
> >
> > 1. Redis Type String
> > No comments except that a future Geode value-add would be to extend the
> > Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> > will work. The tradeoff of this is that the data cannot be read by a
> > native redis client but for Geode users it’s great. Call the new client
> > Geodis.
> >
> > 2. List/ Hash/ Set/ SortedSet
> > Creating a separate region for each creates a constraint that the keys
> are
> > limited to the characters for region names, which are A-z/0-9/ - and _.
> > Everything else is out. Redis users might start asking questions why
> their
> > list named ++^^/## throws an error. Your suggestion to make it a key
> rather
> > than a region solves this. Furthermore, creating a new region every time
> a
> > new Redis collection is created is going to be slow. I’m not sure why a
> > region was created but I’m sure it made sense to the developer at the
> time.
> >
> > 7. Default Config
> > Can’t we configure a gfsh option to default to the region types we want?
> > Customer A will want PARTITION but Customer B will want
> > PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider
> a
> > geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_
> PERSISTENT
> > that makes _all_ Redis regions of that type?
> >
> >
> >
> > On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com
> <mailto:
> > hiteshk25@yahoo.com>> wrote:
> >
> > Current GeodeRedisAdapter implementation is based on
> > https://cwiki.apache.org/confluence/display/GEODE/
> Geode+Redis+Adapter+Proposal
> > .
> > We are looking for some feedback on Redis commands and their mapping to
> > geode region.
> >
> > 1. Redis Type String
> > a. Usage Set k1 v1
> > b. Current implementation creates "STRING_REGION" geode-partition-region
> > upfront
> > c. This k1/v1 are geode-region key/value
> > d. Any feedback?
> >
> > 2. List Type
> > a. usage "rpush mylist A"
> > b. Current implementation maps each list to geode-partition-region(i.e.
> > mylist is geode-partition-region); with the ability to get item from
> > head/tail
> > c. Feedback/vote
> > -- List type operation at region-entry level;
> > -- region-key = "mylist"
> > -- region-value = Arraylist (will support all redis list ops)
> > d. Feedback/vote: both behavior is desirable
> >
> >
> > 3. Hashes
> > a. this represents field-value or java bean object
> > b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> > c. Current implementation maps each hashes to
> > geode-partition-region(i.e. user1000 is geode-partition-region)
> > d. Feedback/vote
> > -- Should we map hashes to region-entry
> > -- region-key = user1000
> > -- region-value = map
> > -- This will provide java bean sort to behaviour with 10s of
> > field-value
> > -- Personally I would prefer this..
> > e. Feedback/vote: both behaviour is desirable
> >
> > 4. Sets
> > a. This represents unique keys in set
> > b. usage "sadd myset 1 2 3"
> > c. Current implementation maps each sadd to geode-partition-region(i.e.
> > myset is geode-partition-region)
> > d. Feedback/vote
> > -- Should we map set to region-entry
> > -- region-key = myset
> > -- region-value = Hashset
> > e. Feedback/vote: both behaviour is desirable
> >
> > 5. SortedSets
> > a. This represents unique keys in set with score (usecase Query top-10)
> > b. usage "zadd hackers 1940 "Alan Kay""
> > c. Current implementation maps each zadd to geode-partition-region(i.e.
> > hackers is geode-partition-region)
> > d. Feedback/vote
> > -- Should we map set to region-entry
> > -- region-key = hackers
> > -- region-value = Sorted Hashset
> > e. Feedback/vote: both behaviour is desirable
> >
> > 6. HyperLogLogs
> > a. A HyperLogLog is a probabilistic data structure used in order to
> > count unique things (technically this is referred to estimating the
> > cardinality of a set).
> > b. usage "pfadd hll a b c d"
> > c. Current implementation creates "HLL_REGION" geode-partition-region
> > upfront
> > d. hll becomes region-key and value is HLL object
> > e. any feedback?
> >
> > 7. Default config for geode-region (vote)
> > a. partition region
> > b. 1 redundant copy
> > c. Persistence
> > d. Eviction
> > e. Expiration
> > f. ?
> >
> > 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> > Thus for each operation we need to make sure type of key. In current
> > implementation we have different region for each redis type. Thus we have
> > another region(metaTypeRegion) which keeps type for each key. This makes
> > any operation in geode slow as it needs to verify that type. For
> instance,
> > creating new key need to make sure its already there or not. Whether we
> > should allow type change or not.
> > a. Feedback/vote
> > -- type change of key
> > -- Can we allow two key with same name but two differnt type (as it
> > will endup in two different geode-region)
> > String type "key1" in string region
> > HLL type "key1" in HLL region
> > b. any other feedback
> >
> > 9. Transactions:
> > a. we will not support transaction in redisAdapter as geode transaction
> > are limited to single node.
> > b. feedback?
> >
> > 10. Redis COMMAND (https://redis.io/commands/command)
> > a. should we implement this "COMMAND" ?
> >
> > 11. Any other redis command we should consider?
> >
> >
> > Thanks.
> > Hitesh
> >
> >
> >
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Galen M O'Sullivan <go...@pivotal.io>.
If we put them in separate regions, we'll have the overhead of looking up
in two regions added to each and every operation, and the overhead of
creating all these regions.
If we really wanted to we could have some threshold at which we spill
collections over into their own regions, and have something like the best
of both worlds. It's more complex, though, and I don't know how many people
actually use truly huge collections.
On Tue, Feb 14, 2017 at 4:21 PM, Hitesh Khamesra <
hiteshk25@yahoo.com.invalid> wrote:
> Jason/Dan: Sorry to hear about that. But both of you have asked the right
> question.
> it depends on your use-case(item 2,3,4,5) . For example "hashes" can be
> use to define key-value pair or java bean. In this case probably it is
> better to keep that hash at region-entry level. But if you want to know
> top 10 tweets which are trending then probably you want use
> partition-region for "sorted-set".
>
>
> From: Jason Huynh <jh...@pivotal.io>
> To: dev@geode.apache.org; "user@geode.apache.org" <us...@geode.apache.org>;
> Hitesh Khamesra <hi...@yahoo.com>
> Sent: Tuesday, February 14, 2017 3:15 PM
> Subject: Re: GeodeRedisAdapter improvments/feedback
>
> Hi Hitesh,
>
> Not sure about everyone else, but I had a hard time reading this, however
> I think I figured out what you were describing... the only part I still am
> unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> you want feedback and voting on whether both behaviors are desired? As in
> old implementation and new implementation?
>
> 2,3,4) The new implementation would mean all the data for a specific data
> structure is contained in a single bucket. So the individual data
> structures are not quite scalable. How would you allow scaling of a single
> data structure?
>
> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
>
> > In what format do you want the feedback Hitesh? For now I’ll just
> comment:
> >
> > 1. Redis Type String
> > No comments except that a future Geode value-add would be to extend the
> > Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> > will work. The tradeoff of this is that the data cannot be read by a
> > native redis client but for Geode users it’s great. Call the new client
> > Geodis.
> >
> > 2. List/ Hash/ Set/ SortedSet
> > Creating a separate region for each creates a constraint that the keys
> are
> > limited to the characters for region names, which are A-z/0-9/ - and _.
> > Everything else is out. Redis users might start asking questions why
> their
> > list named ++^^/## throws an error. Your suggestion to make it a key
> rather
> > than a region solves this. Furthermore, creating a new region every time
> a
> > new Redis collection is created is going to be slow. I’m not sure why a
> > region was created but I’m sure it made sense to the developer at the
> time.
> >
> > 7. Default Config
> > Can’t we configure a gfsh option to default to the region types we want?
> > Customer A will want PARTITION but Customer B will want
> > PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider
> a
> > geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_
> PERSISTENT
> > that makes _all_ Redis regions of that type?
> >
> >
> >
> > On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com
> <mailto:
> > hiteshk25@yahoo.com>> wrote:
> >
> > Current GeodeRedisAdapter implementation is based on
> > https://cwiki.apache.org/confluence/display/GEODE/
> Geode+Redis+Adapter+Proposal
> > .
> > We are looking for some feedback on Redis commands and their mapping to
> > geode region.
> >
> > 1. Redis Type String
> > a. Usage Set k1 v1
> > b. Current implementation creates "STRING_REGION" geode-partition-region
> > upfront
> > c. This k1/v1 are geode-region key/value
> > d. Any feedback?
> >
> > 2. List Type
> > a. usage "rpush mylist A"
> > b. Current implementation maps each list to geode-partition-region(i.e.
> > mylist is geode-partition-region); with the ability to get item from
> > head/tail
> > c. Feedback/vote
> > -- List type operation at region-entry level;
> > -- region-key = "mylist"
> > -- region-value = Arraylist (will support all redis list ops)
> > d. Feedback/vote: both behavior is desirable
> >
> >
> > 3. Hashes
> > a. this represents field-value or java bean object
> > b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> > c. Current implementation maps each hashes to
> > geode-partition-region(i.e. user1000 is geode-partition-region)
> > d. Feedback/vote
> > -- Should we map hashes to region-entry
> > -- region-key = user1000
> > -- region-value = map
> > -- This will provide java bean sort to behaviour with 10s of
> > field-value
> > -- Personally I would prefer this..
> > e. Feedback/vote: both behaviour is desirable
> >
> > 4. Sets
> > a. This represents unique keys in set
> > b. usage "sadd myset 1 2 3"
> > c. Current implementation maps each sadd to geode-partition-region(i.e.
> > myset is geode-partition-region)
> > d. Feedback/vote
> > -- Should we map set to region-entry
> > -- region-key = myset
> > -- region-value = Hashset
> > e. Feedback/vote: both behaviour is desirable
> >
> > 5. SortedSets
> > a. This represents unique keys in set with score (usecase Query top-10)
> > b. usage "zadd hackers 1940 "Alan Kay""
> > c. Current implementation maps each zadd to geode-partition-region(i.e.
> > hackers is geode-partition-region)
> > d. Feedback/vote
> > -- Should we map set to region-entry
> > -- region-key = hackers
> > -- region-value = Sorted Hashset
> > e. Feedback/vote: both behaviour is desirable
> >
> > 6. HyperLogLogs
> > a. A HyperLogLog is a probabilistic data structure used in order to
> > count unique things (technically this is referred to estimating the
> > cardinality of a set).
> > b. usage "pfadd hll a b c d"
> > c. Current implementation creates "HLL_REGION" geode-partition-region
> > upfront
> > d. hll becomes region-key and value is HLL object
> > e. any feedback?
> >
> > 7. Default config for geode-region (vote)
> > a. partition region
> > b. 1 redundant copy
> > c. Persistence
> > d. Eviction
> > e. Expiration
> > f. ?
> >
> > 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> > Thus for each operation we need to make sure type of key. In current
> > implementation we have different region for each redis type. Thus we have
> > another region(metaTypeRegion) which keeps type for each key. This makes
> > any operation in geode slow as it needs to verify that type. For
> instance,
> > creating new key need to make sure its already there or not. Whether we
> > should allow type change or not.
> > a. Feedback/vote
> > -- type change of key
> > -- Can we allow two key with same name but two differnt type (as it
> > will endup in two different geode-region)
> > String type "key1" in string region
> > HLL type "key1" in HLL region
> > b. any other feedback
> >
> > 9. Transactions:
> > a. we will not support transaction in redisAdapter as geode transaction
> > are limited to single node.
> > b. feedback?
> >
> > 10. Redis COMMAND (https://redis.io/commands/command)
> > a. should we implement this "COMMAND" ?
> >
> > 11. Any other redis command we should consider?
> >
> >
> > Thanks.
> > Hitesh
> >
> >
> >
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Hitesh Khamesra <hi...@yahoo.com.INVALID>.
Jason/Dan: Sorry to hear about that. But both of you have asked the right question.
it depends on your use-case(item 2,3,4,5) . For example "hashes" can be use to define key-value pair or java bean. In this case probably it is better to keep that hash at region-entry level. But if you want to know top 10 tweets which are trending then probably you want use partition-region for "sorted-set".
From: Jason Huynh <jh...@pivotal.io>
To: dev@geode.apache.org; "user@geode.apache.org" <us...@geode.apache.org>; Hitesh Khamesra <hi...@yahoo.com>
Sent: Tuesday, February 14, 2017 3:15 PM
Subject: Re: GeodeRedisAdapter improvments/feedback
Hi Hitesh,
Not sure about everyone else, but I had a hard time reading this, however
I think I figured out what you were describing... the only part I still am
unsure about is Feedback/vote: both behaviour is desirable. Do you mean
you want feedback and voting on whether both behaviors are desired? As in
old implementation and new implementation?
2,3,4) The new implementation would mean all the data for a specific data
structure is contained in a single bucket. So the individual data
structures are not quite scalable. How would you allow scaling of a single
data structure?
On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
> In what format do you want the feedback Hitesh? For now I’ll just comment:
>
> 1. Redis Type String
> No comments except that a future Geode value-add would be to extend the
> Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> will work. The tradeoff of this is that the data cannot be read by a
> native redis client but for Geode users it’s great. Call the new client
> Geodis.
>
> 2. List/ Hash/ Set/ SortedSet
> Creating a separate region for each creates a constraint that the keys are
> limited to the characters for region names, which are A-z/0-9/ - and _.
> Everything else is out. Redis users might start asking questions why their
> list named ++^^/## throws an error. Your suggestion to make it a key rather
> than a region solves this. Furthermore, creating a new region every time a
> new Redis collection is created is going to be slow. I’m not sure why a
> region was created but I’m sure it made sense to the developer at the time.
>
> 7. Default Config
> Can’t we configure a gfsh option to default to the region types we want?
> Customer A will want PARTITION but Customer B will want
> PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a
> geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT
> that makes _all_ Redis regions of that type?
>
>
>
> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com<mailto:
> hiteshk25@yahoo.com>> wrote:
>
> Current GeodeRedisAdapter implementation is based on
> https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal
> .
> We are looking for some feedback on Redis commands and their mapping to
> geode region.
>
> 1. Redis Type String
> a. Usage Set k1 v1
> b. Current implementation creates "STRING_REGION" geode-partition-region
> upfront
> c. This k1/v1 are geode-region key/value
> d. Any feedback?
>
> 2. List Type
> a. usage "rpush mylist A"
> b. Current implementation maps each list to geode-partition-region(i.e.
> mylist is geode-partition-region); with the ability to get item from
> head/tail
> c. Feedback/vote
> -- List type operation at region-entry level;
> -- region-key = "mylist"
> -- region-value = Arraylist (will support all redis list ops)
> d. Feedback/vote: both behavior is desirable
>
>
> 3. Hashes
> a. this represents field-value or java bean object
> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> c. Current implementation maps each hashes to
> geode-partition-region(i.e. user1000 is geode-partition-region)
> d. Feedback/vote
> -- Should we map hashes to region-entry
> -- region-key = user1000
> -- region-value = map
> -- This will provide java bean sort to behaviour with 10s of
> field-value
> -- Personally I would prefer this..
> e. Feedback/vote: both behaviour is desirable
>
> 4. Sets
> a. This represents unique keys in set
> b. usage "sadd myset 1 2 3"
> c. Current implementation maps each sadd to geode-partition-region(i.e.
> myset is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = myset
> -- region-value = Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 5. SortedSets
> a. This represents unique keys in set with score (usecase Query top-10)
> b. usage "zadd hackers 1940 "Alan Kay""
> c. Current implementation maps each zadd to geode-partition-region(i.e.
> hackers is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = hackers
> -- region-value = Sorted Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 6. HyperLogLogs
> a. A HyperLogLog is a probabilistic data structure used in order to
> count unique things (technically this is referred to estimating the
> cardinality of a set).
> b. usage "pfadd hll a b c d"
> c. Current implementation creates "HLL_REGION" geode-partition-region
> upfront
> d. hll becomes region-key and value is HLL object
> e. any feedback?
>
> 7. Default config for geode-region (vote)
> a. partition region
> b. 1 redundant copy
> c. Persistence
> d. Eviction
> e. Expiration
> f. ?
>
> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> Thus for each operation we need to make sure type of key. In current
> implementation we have different region for each redis type. Thus we have
> another region(metaTypeRegion) which keeps type for each key. This makes
> any operation in geode slow as it needs to verify that type. For instance,
> creating new key need to make sure its already there or not. Whether we
> should allow type change or not.
> a. Feedback/vote
> -- type change of key
> -- Can we allow two key with same name but two differnt type (as it
> will endup in two different geode-region)
> String type "key1" in string region
> HLL type "key1" in HLL region
> b. any other feedback
>
> 9. Transactions:
> a. we will not support transaction in redisAdapter as geode transaction
> are limited to single node.
> b. feedback?
>
> 10. Redis COMMAND (https://redis.io/commands/command)
> a. should we implement this "COMMAND" ?
>
> 11. Any other redis command we should consider?
>
>
> Thanks.
> Hitesh
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Hitesh Khamesra <hi...@yahoo.com>.
Jason/Dan: Sorry to hear about that. But both of you have asked the right question.
it depends on your use-case(item 2,3,4,5) . For example "hashes" can be use to define key-value pair or java bean. In this case probably it is better to keep that hash at region-entry level. But if you want to know top 10 tweets which are trending then probably you want use partition-region for "sorted-set".
From: Jason Huynh <jh...@pivotal.io>
To: dev@geode.apache.org; "user@geode.apache.org" <us...@geode.apache.org>; Hitesh Khamesra <hi...@yahoo.com>
Sent: Tuesday, February 14, 2017 3:15 PM
Subject: Re: GeodeRedisAdapter improvments/feedback
Hi Hitesh,
Not sure about everyone else, but I had a hard time reading this, however
I think I figured out what you were describing... the only part I still am
unsure about is Feedback/vote: both behaviour is desirable. Do you mean
you want feedback and voting on whether both behaviors are desired? As in
old implementation and new implementation?
2,3,4) The new implementation would mean all the data for a specific data
structure is contained in a single bucket. So the individual data
structures are not quite scalable. How would you allow scaling of a single
data structure?
On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
> In what format do you want the feedback Hitesh? For now I’ll just comment:
>
> 1. Redis Type String
> No comments except that a future Geode value-add would be to extend the
> Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> will work. The tradeoff of this is that the data cannot be read by a
> native redis client but for Geode users it’s great. Call the new client
> Geodis.
>
> 2. List/ Hash/ Set/ SortedSet
> Creating a separate region for each creates a constraint that the keys are
> limited to the characters for region names, which are A-z/0-9/ - and _.
> Everything else is out. Redis users might start asking questions why their
> list named ++^^/## throws an error. Your suggestion to make it a key rather
> than a region solves this. Furthermore, creating a new region every time a
> new Redis collection is created is going to be slow. I’m not sure why a
> region was created but I’m sure it made sense to the developer at the time.
>
> 7. Default Config
> Can’t we configure a gfsh option to default to the region types we want?
> Customer A will want PARTITION but Customer B will want
> PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a
> geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT
> that makes _all_ Redis regions of that type?
>
>
>
> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com<mailto:
> hiteshk25@yahoo.com>> wrote:
>
> Current GeodeRedisAdapter implementation is based on
> https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal
> .
> We are looking for some feedback on Redis commands and their mapping to
> geode region.
>
> 1. Redis Type String
> a. Usage Set k1 v1
> b. Current implementation creates "STRING_REGION" geode-partition-region
> upfront
> c. This k1/v1 are geode-region key/value
> d. Any feedback?
>
> 2. List Type
> a. usage "rpush mylist A"
> b. Current implementation maps each list to geode-partition-region(i.e.
> mylist is geode-partition-region); with the ability to get item from
> head/tail
> c. Feedback/vote
> -- List type operation at region-entry level;
> -- region-key = "mylist"
> -- region-value = Arraylist (will support all redis list ops)
> d. Feedback/vote: both behavior is desirable
>
>
> 3. Hashes
> a. this represents field-value or java bean object
> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> c. Current implementation maps each hashes to
> geode-partition-region(i.e. user1000 is geode-partition-region)
> d. Feedback/vote
> -- Should we map hashes to region-entry
> -- region-key = user1000
> -- region-value = map
> -- This will provide java bean sort to behaviour with 10s of
> field-value
> -- Personally I would prefer this..
> e. Feedback/vote: both behaviour is desirable
>
> 4. Sets
> a. This represents unique keys in set
> b. usage "sadd myset 1 2 3"
> c. Current implementation maps each sadd to geode-partition-region(i.e.
> myset is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = myset
> -- region-value = Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 5. SortedSets
> a. This represents unique keys in set with score (usecase Query top-10)
> b. usage "zadd hackers 1940 "Alan Kay""
> c. Current implementation maps each zadd to geode-partition-region(i.e.
> hackers is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = hackers
> -- region-value = Sorted Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 6. HyperLogLogs
> a. A HyperLogLog is a probabilistic data structure used in order to
> count unique things (technically this is referred to estimating the
> cardinality of a set).
> b. usage "pfadd hll a b c d"
> c. Current implementation creates "HLL_REGION" geode-partition-region
> upfront
> d. hll becomes region-key and value is HLL object
> e. any feedback?
>
> 7. Default config for geode-region (vote)
> a. partition region
> b. 1 redundant copy
> c. Persistence
> d. Eviction
> e. Expiration
> f. ?
>
> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> Thus for each operation we need to make sure type of key. In current
> implementation we have different region for each redis type. Thus we have
> another region(metaTypeRegion) which keeps type for each key. This makes
> any operation in geode slow as it needs to verify that type. For instance,
> creating new key need to make sure its already there or not. Whether we
> should allow type change or not.
> a. Feedback/vote
> -- type change of key
> -- Can we allow two key with same name but two differnt type (as it
> will endup in two different geode-region)
> String type "key1" in string region
> HLL type "key1" in HLL region
> b. any other feedback
>
> 9. Transactions:
> a. we will not support transaction in redisAdapter as geode transaction
> are limited to single node.
> b. feedback?
>
> 10. Redis COMMAND (https://redis.io/commands/command)
> a. should we implement this "COMMAND" ?
>
> 11. Any other redis command we should consider?
>
>
> Thanks.
> Hitesh
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Swapnil Bawaskar <sb...@pivotal.io>.
The Redis adapter was designed so that we can scale all the Redis data
structures horizontally. If you bring the data structures to region entry
level, there is no reason for anyone to use our implementation over Redis.
On Tue, Feb 14, 2017 at 3:15 PM Jason Huynh <jh...@pivotal.io> wrote:
> Hi Hitesh,
>
> Not sure about everyone else, but I had a hard time reading this, however
> I think I figured out what you were describing... the only part I still am
> unsure about is Feedback/vote: both behaviour is desirable. Do you mean
> you want feedback and voting on whether both behaviors are desired? As in
> old implementation and new implementation?
>
> 2,3,4) The new implementation would mean all the data for a specific data
> structure is contained in a single bucket. So the individual data
> structures are not quite scalable. How would you allow scaling of a single
> data structure?
>
> On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
>
> In what format do you want the feedback Hitesh? For now I’ll just comment:
>
> 1. Redis Type String
> No comments except that a future Geode value-add would be to extend the
> Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> will work. The tradeoff of this is that the data cannot be read by a
> native redis client but for Geode users it’s great. Call the new client
> Geodis.
>
> 2. List/ Hash/ Set/ SortedSet
> Creating a separate region for each creates a constraint that the keys are
> limited to the characters for region names, which are A-z/0-9/ - and _.
> Everything else is out. Redis users might start asking questions why their
> list named ++^^/## throws an error. Your suggestion to make it a key rather
> than a region solves this. Furthermore, creating a new region every time a
> new Redis collection is created is going to be slow. I’m not sure why a
> region was created but I’m sure it made sense to the developer at the time.
>
> 7. Default Config
> Can’t we configure a gfsh option to default to the region types we want?
> Customer A will want PARTITION but Customer B will want
> PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a
> geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT
> that makes _all_ Redis regions of that type?
>
>
>
> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com<mailto:
> hiteshk25@yahoo.com>> wrote:
>
> Current GeodeRedisAdapter implementation is based on
> https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal
> .
> We are looking for some feedback on Redis commands and their mapping to
> geode region.
>
> 1. Redis Type String
> a. Usage Set k1 v1
> b. Current implementation creates "STRING_REGION" geode-partition-region
> upfront
> c. This k1/v1 are geode-region key/value
> d. Any feedback?
>
> 2. List Type
> a. usage "rpush mylist A"
> b. Current implementation maps each list to geode-partition-region(i.e.
> mylist is geode-partition-region); with the ability to get item from
> head/tail
> c. Feedback/vote
> -- List type operation at region-entry level;
> -- region-key = "mylist"
> -- region-value = Arraylist (will support all redis list ops)
> d. Feedback/vote: both behavior is desirable
>
>
> 3. Hashes
> a. this represents field-value or java bean object
> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> c. Current implementation maps each hashes to
> geode-partition-region(i.e. user1000 is geode-partition-region)
> d. Feedback/vote
> -- Should we map hashes to region-entry
> -- region-key = user1000
> -- region-value = map
> -- This will provide java bean sort to behaviour with 10s of
> field-value
> -- Personally I would prefer this..
> e. Feedback/vote: both behaviour is desirable
>
> 4. Sets
> a. This represents unique keys in set
> b. usage "sadd myset 1 2 3"
> c. Current implementation maps each sadd to geode-partition-region(i.e.
> myset is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = myset
> -- region-value = Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 5. SortedSets
> a. This represents unique keys in set with score (usecase Query top-10)
> b. usage "zadd hackers 1940 "Alan Kay""
> c. Current implementation maps each zadd to geode-partition-region(i.e.
> hackers is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = hackers
> -- region-value = Sorted Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 6. HyperLogLogs
> a. A HyperLogLog is a probabilistic data structure used in order to
> count unique things (technically this is referred to estimating the
> cardinality of a set).
> b. usage "pfadd hll a b c d"
> c. Current implementation creates "HLL_REGION" geode-partition-region
> upfront
> d. hll becomes region-key and value is HLL object
> e. any feedback?
>
> 7. Default config for geode-region (vote)
> a. partition region
> b. 1 redundant copy
> c. Persistence
> d. Eviction
> e. Expiration
> f. ?
>
> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> Thus for each operation we need to make sure type of key. In current
> implementation we have different region for each redis type. Thus we have
> another region(metaTypeRegion) which keeps type for each key. This makes
> any operation in geode slow as it needs to verify that type. For instance,
> creating new key need to make sure its already there or not. Whether we
> should allow type change or not.
> a. Feedback/vote
> -- type change of key
> -- Can we allow two key with same name but two differnt type (as it
> will endup in two different geode-region)
> String type "key1" in string region
> HLL type "key1" in HLL region
> b. any other feedback
>
> 9. Transactions:
> a. we will not support transaction in redisAdapter as geode transaction
> are limited to single node.
> b. feedback?
>
> 10. Redis COMMAND (https://redis.io/commands/command)
> a. should we implement this "COMMAND" ?
>
> 11. Any other redis command we should consider?
>
>
> Thanks.
> Hitesh
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Jason Huynh <jh...@pivotal.io>.
Hi Hitesh,
Not sure about everyone else, but I had a hard time reading this, however
I think I figured out what you were describing... the only part I still am
unsure about is Feedback/vote: both behaviour is desirable. Do you mean
you want feedback and voting on whether both behaviors are desired? As in
old implementation and new implementation?
2,3,4) The new implementation would mean all the data for a specific data
structure is contained in a single bucket. So the individual data
structures are not quite scalable. How would you allow scaling of a single
data structure?
On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
> In what format do you want the feedback Hitesh? For now I’ll just comment:
>
> 1. Redis Type String
> No comments except that a future Geode value-add would be to extend the
> Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> will work. The tradeoff of this is that the data cannot be read by a
> native redis client but for Geode users it’s great. Call the new client
> Geodis.
>
> 2. List/ Hash/ Set/ SortedSet
> Creating a separate region for each creates a constraint that the keys are
> limited to the characters for region names, which are A-z/0-9/ - and _.
> Everything else is out. Redis users might start asking questions why their
> list named ++^^/## throws an error. Your suggestion to make it a key rather
> than a region solves this. Furthermore, creating a new region every time a
> new Redis collection is created is going to be slow. I’m not sure why a
> region was created but I’m sure it made sense to the developer at the time.
>
> 7. Default Config
> Can’t we configure a gfsh option to default to the region types we want?
> Customer A will want PARTITION but Customer B will want
> PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a
> geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT
> that makes _all_ Redis regions of that type?
>
>
>
> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com<mailto:
> hiteshk25@yahoo.com>> wrote:
>
> Current GeodeRedisAdapter implementation is based on
> https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal
> .
> We are looking for some feedback on Redis commands and their mapping to
> geode region.
>
> 1. Redis Type String
> a. Usage Set k1 v1
> b. Current implementation creates "STRING_REGION" geode-partition-region
> upfront
> c. This k1/v1 are geode-region key/value
> d. Any feedback?
>
> 2. List Type
> a. usage "rpush mylist A"
> b. Current implementation maps each list to geode-partition-region(i.e.
> mylist is geode-partition-region); with the ability to get item from
> head/tail
> c. Feedback/vote
> -- List type operation at region-entry level;
> -- region-key = "mylist"
> -- region-value = Arraylist (will support all redis list ops)
> d. Feedback/vote: both behavior is desirable
>
>
> 3. Hashes
> a. this represents field-value or java bean object
> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> c. Current implementation maps each hashes to
> geode-partition-region(i.e. user1000 is geode-partition-region)
> d. Feedback/vote
> -- Should we map hashes to region-entry
> -- region-key = user1000
> -- region-value = map
> -- This will provide java bean sort to behaviour with 10s of
> field-value
> -- Personally I would prefer this..
> e. Feedback/vote: both behaviour is desirable
>
> 4. Sets
> a. This represents unique keys in set
> b. usage "sadd myset 1 2 3"
> c. Current implementation maps each sadd to geode-partition-region(i.e.
> myset is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = myset
> -- region-value = Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 5. SortedSets
> a. This represents unique keys in set with score (usecase Query top-10)
> b. usage "zadd hackers 1940 "Alan Kay""
> c. Current implementation maps each zadd to geode-partition-region(i.e.
> hackers is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = hackers
> -- region-value = Sorted Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 6. HyperLogLogs
> a. A HyperLogLog is a probabilistic data structure used in order to
> count unique things (technically this is referred to estimating the
> cardinality of a set).
> b. usage "pfadd hll a b c d"
> c. Current implementation creates "HLL_REGION" geode-partition-region
> upfront
> d. hll becomes region-key and value is HLL object
> e. any feedback?
>
> 7. Default config for geode-region (vote)
> a. partition region
> b. 1 redundant copy
> c. Persistence
> d. Eviction
> e. Expiration
> f. ?
>
> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> Thus for each operation we need to make sure type of key. In current
> implementation we have different region for each redis type. Thus we have
> another region(metaTypeRegion) which keeps type for each key. This makes
> any operation in geode slow as it needs to verify that type. For instance,
> creating new key need to make sure its already there or not. Whether we
> should allow type change or not.
> a. Feedback/vote
> -- type change of key
> -- Can we allow two key with same name but two differnt type (as it
> will endup in two different geode-region)
> String type "key1" in string region
> HLL type "key1" in HLL region
> b. any other feedback
>
> 9. Transactions:
> a. we will not support transaction in redisAdapter as geode transaction
> are limited to single node.
> b. feedback?
>
> 10. Redis COMMAND (https://redis.io/commands/command)
> a. should we implement this "COMMAND" ?
>
> 11. Any other redis command we should consider?
>
>
> Thanks.
> Hitesh
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Jason Huynh <jh...@pivotal.io>.
Hi Hitesh,
Not sure about everyone else, but I had a hard time reading this, however
I think I figured out what you were describing... the only part I still am
unsure about is Feedback/vote: both behaviour is desirable. Do you mean
you want feedback and voting on whether both behaviors are desired? As in
old implementation and new implementation?
2,3,4) The new implementation would mean all the data for a specific data
structure is contained in a single bucket. So the individual data
structures are not quite scalable. How would you allow scaling of a single
data structure?
On Tue, Feb 14, 2017 at 3:05 PM Real Wes <Th...@outlook.com> wrote:
> In what format do you want the feedback Hitesh? For now I’ll just comment:
>
> 1. Redis Type String
> No comments except that a future Geode value-add would be to extend the
> Jedis client so that the K/V’s are not compressed. In this way OQL and CQ
> will work. The tradeoff of this is that the data cannot be read by a
> native redis client but for Geode users it’s great. Call the new client
> Geodis.
>
> 2. List/ Hash/ Set/ SortedSet
> Creating a separate region for each creates a constraint that the keys are
> limited to the characters for region names, which are A-z/0-9/ - and _.
> Everything else is out. Redis users might start asking questions why their
> list named ++^^/## throws an error. Your suggestion to make it a key rather
> than a region solves this. Furthermore, creating a new region every time a
> new Redis collection is created is going to be slow. I’m not sure why a
> region was created but I’m sure it made sense to the developer at the time.
>
> 7. Default Config
> Can’t we configure a gfsh option to default to the region types we want?
> Customer A will want PARTITION but Customer B will want
> PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a
> geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT
> that makes _all_ Redis regions of that type?
>
>
>
> On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hiteshk25@yahoo.com<mailto:
> hiteshk25@yahoo.com>> wrote:
>
> Current GeodeRedisAdapter implementation is based on
> https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal
> .
> We are looking for some feedback on Redis commands and their mapping to
> geode region.
>
> 1. Redis Type String
> a. Usage Set k1 v1
> b. Current implementation creates "STRING_REGION" geode-partition-region
> upfront
> c. This k1/v1 are geode-region key/value
> d. Any feedback?
>
> 2. List Type
> a. usage "rpush mylist A"
> b. Current implementation maps each list to geode-partition-region(i.e.
> mylist is geode-partition-region); with the ability to get item from
> head/tail
> c. Feedback/vote
> -- List type operation at region-entry level;
> -- region-key = "mylist"
> -- region-value = Arraylist (will support all redis list ops)
> d. Feedback/vote: both behavior is desirable
>
>
> 3. Hashes
> a. this represents field-value or java bean object
> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> c. Current implementation maps each hashes to
> geode-partition-region(i.e. user1000 is geode-partition-region)
> d. Feedback/vote
> -- Should we map hashes to region-entry
> -- region-key = user1000
> -- region-value = map
> -- This will provide java bean sort to behaviour with 10s of
> field-value
> -- Personally I would prefer this..
> e. Feedback/vote: both behaviour is desirable
>
> 4. Sets
> a. This represents unique keys in set
> b. usage "sadd myset 1 2 3"
> c. Current implementation maps each sadd to geode-partition-region(i.e.
> myset is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = myset
> -- region-value = Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 5. SortedSets
> a. This represents unique keys in set with score (usecase Query top-10)
> b. usage "zadd hackers 1940 "Alan Kay""
> c. Current implementation maps each zadd to geode-partition-region(i.e.
> hackers is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = hackers
> -- region-value = Sorted Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 6. HyperLogLogs
> a. A HyperLogLog is a probabilistic data structure used in order to
> count unique things (technically this is referred to estimating the
> cardinality of a set).
> b. usage "pfadd hll a b c d"
> c. Current implementation creates "HLL_REGION" geode-partition-region
> upfront
> d. hll becomes region-key and value is HLL object
> e. any feedback?
>
> 7. Default config for geode-region (vote)
> a. partition region
> b. 1 redundant copy
> c. Persistence
> d. Eviction
> e. Expiration
> f. ?
>
> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> Thus for each operation we need to make sure type of key. In current
> implementation we have different region for each redis type. Thus we have
> another region(metaTypeRegion) which keeps type for each key. This makes
> any operation in geode slow as it needs to verify that type. For instance,
> creating new key need to make sure its already there or not. Whether we
> should allow type change or not.
> a. Feedback/vote
> -- type change of key
> -- Can we allow two key with same name but two differnt type (as it
> will endup in two different geode-region)
> String type "key1" in string region
> HLL type "key1" in HLL region
> b. any other feedback
>
> 9. Transactions:
> a. we will not support transaction in redisAdapter as geode transaction
> are limited to single node.
> b. feedback?
>
> 10. Redis COMMAND (https://redis.io/commands/command)
> a. should we implement this "COMMAND" ?
>
> 11. Any other redis command we should consider?
>
>
> Thanks.
> Hitesh
>
>
>
Re: GeodeRedisAdapter improvments/feedback
Posted by Real Wes <Th...@outlook.com>.
In what format do you want the feedback Hitesh? For now I’ll just comment:
1. Redis Type String
No comments except that a future Geode value-add would be to extend the Jedis client so that the K/V’s are not compressed. In this way OQL and CQ will work. The tradeoff of this is that the data cannot be read by a native redis client but for Geode users it’s great. Call the new client Geodis.
2. List/ Hash/ Set/ SortedSet
Creating a separate region for each creates a constraint that the keys are limited to the characters for region names, which are A-z/0-9/ - and _. Everything else is out. Redis users might start asking questions why their list named ++^^/## throws an error. Your suggestion to make it a key rather than a region solves this. Furthermore, creating a new region every time a new Redis collection is created is going to be slow. I’m not sure why a region was created but I’m sure it made sense to the developer at the time.
7. Default Config
Can’t we configure a gfsh option to default to the region types we want? Customer A will want PARTITION but Customer B will want PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT that makes _all_ Redis regions of that type?
On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hi...@yahoo.com>> wrote:
Current GeodeRedisAdapter implementation is based on https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal.
We are looking for some feedback on Redis commands and their mapping to geode region.
1. Redis Type String
a. Usage Set k1 v1
b. Current implementation creates "STRING_REGION" geode-partition-region upfront
c. This k1/v1 are geode-region key/value
d. Any feedback?
2. List Type
a. usage "rpush mylist A"
b. Current implementation maps each list to geode-partition-region(i.e. mylist is geode-partition-region); with the ability to get item from head/tail
c. Feedback/vote
-- List type operation at region-entry level;
-- region-key = "mylist"
-- region-value = Arraylist (will support all redis list ops)
d. Feedback/vote: both behavior is desirable
3. Hashes
a. this represents field-value or java bean object
b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
c. Current implementation maps each hashes to geode-partition-region(i.e. user1000 is geode-partition-region)
d. Feedback/vote
-- Should we map hashes to region-entry
-- region-key = user1000
-- region-value = map
-- This will provide java bean sort to behaviour with 10s of field-value
-- Personally I would prefer this..
e. Feedback/vote: both behaviour is desirable
4. Sets
a. This represents unique keys in set
b. usage "sadd myset 1 2 3"
c. Current implementation maps each sadd to geode-partition-region(i.e. myset is geode-partition-region)
d. Feedback/vote
-- Should we map set to region-entry
-- region-key = myset
-- region-value = Hashset
e. Feedback/vote: both behaviour is desirable
5. SortedSets
a. This represents unique keys in set with score (usecase Query top-10)
b. usage "zadd hackers 1940 "Alan Kay""
c. Current implementation maps each zadd to geode-partition-region(i.e. hackers is geode-partition-region)
d. Feedback/vote
-- Should we map set to region-entry
-- region-key = hackers
-- region-value = Sorted Hashset
e. Feedback/vote: both behaviour is desirable
6. HyperLogLogs
a. A HyperLogLog is a probabilistic data structure used in order to count unique things (technically this is referred to estimating the cardinality of a set).
b. usage "pfadd hll a b c d"
c. Current implementation creates "HLL_REGION" geode-partition-region upfront
d. hll becomes region-key and value is HLL object
e. any feedback?
7. Default config for geode-region (vote)
a. partition region
b. 1 redundant copy
c. Persistence
d. Eviction
e. Expiration
f. ?
8. It seems; redis knows type(list, hashes, string ,set ..) of each key. Thus for each operation we need to make sure type of key. In current implementation we have different region for each redis type. Thus we have another region(metaTypeRegion) which keeps type for each key. This makes any operation in geode slow as it needs to verify that type. For instance, creating new key need to make sure its already there or not. Whether we should allow type change or not.
a. Feedback/vote
-- type change of key
-- Can we allow two key with same name but two differnt type (as it will endup in two different geode-region)
String type "key1" in string region
HLL type "key1" in HLL region
b. any other feedback
9. Transactions:
a. we will not support transaction in redisAdapter as geode transaction are limited to single node.
b. feedback?
10. Redis COMMAND (https://redis.io/commands/command)
a. should we implement this "COMMAND" ?
11. Any other redis command we should consider?
Thanks.
Hitesh
Re: GeodeRedisAdapter improvments/feedback
Posted by Real Wes <Th...@outlook.com>.
In what format do you want the feedback Hitesh? For now I’ll just comment:
1. Redis Type String
No comments except that a future Geode value-add would be to extend the Jedis client so that the K/V’s are not compressed. In this way OQL and CQ will work. The tradeoff of this is that the data cannot be read by a native redis client but for Geode users it’s great. Call the new client Geodis.
2. List/ Hash/ Set/ SortedSet
Creating a separate region for each creates a constraint that the keys are limited to the characters for region names, which are A-z/0-9/ - and _. Everything else is out. Redis users might start asking questions why their list named ++^^/## throws an error. Your suggestion to make it a key rather than a region solves this. Furthermore, creating a new region every time a new Redis collection is created is going to be slow. I’m not sure why a region was created but I’m sure it made sense to the developer at the time.
7. Default Config
Can’t we configure a gfsh option to default to the region types we want? Customer A will want PARTITION but Customer B will want PARTITION_REDUNDANT_EXPIRATION_PERSISTENT. I wonder if we can consider a geode> create region —redisType=PARTITION_REDUNDANT_EXPIRATION_PERSISTENT that makes _all_ Redis regions of that type?
On Feb 14, 2017, at 5:36 PM, Hitesh Khamesra <hi...@yahoo.com>> wrote:
Current GeodeRedisAdapter implementation is based on https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter+Proposal.
We are looking for some feedback on Redis commands and their mapping to geode region.
1. Redis Type String
a. Usage Set k1 v1
b. Current implementation creates "STRING_REGION" geode-partition-region upfront
c. This k1/v1 are geode-region key/value
d. Any feedback?
2. List Type
a. usage "rpush mylist A"
b. Current implementation maps each list to geode-partition-region(i.e. mylist is geode-partition-region); with the ability to get item from head/tail
c. Feedback/vote
-- List type operation at region-entry level;
-- region-key = "mylist"
-- region-value = Arraylist (will support all redis list ops)
d. Feedback/vote: both behavior is desirable
3. Hashes
a. this represents field-value or java bean object
b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
c. Current implementation maps each hashes to geode-partition-region(i.e. user1000 is geode-partition-region)
d. Feedback/vote
-- Should we map hashes to region-entry
-- region-key = user1000
-- region-value = map
-- This will provide java bean sort to behaviour with 10s of field-value
-- Personally I would prefer this..
e. Feedback/vote: both behaviour is desirable
4. Sets
a. This represents unique keys in set
b. usage "sadd myset 1 2 3"
c. Current implementation maps each sadd to geode-partition-region(i.e. myset is geode-partition-region)
d. Feedback/vote
-- Should we map set to region-entry
-- region-key = myset
-- region-value = Hashset
e. Feedback/vote: both behaviour is desirable
5. SortedSets
a. This represents unique keys in set with score (usecase Query top-10)
b. usage "zadd hackers 1940 "Alan Kay""
c. Current implementation maps each zadd to geode-partition-region(i.e. hackers is geode-partition-region)
d. Feedback/vote
-- Should we map set to region-entry
-- region-key = hackers
-- region-value = Sorted Hashset
e. Feedback/vote: both behaviour is desirable
6. HyperLogLogs
a. A HyperLogLog is a probabilistic data structure used in order to count unique things (technically this is referred to estimating the cardinality of a set).
b. usage "pfadd hll a b c d"
c. Current implementation creates "HLL_REGION" geode-partition-region upfront
d. hll becomes region-key and value is HLL object
e. any feedback?
7. Default config for geode-region (vote)
a. partition region
b. 1 redundant copy
c. Persistence
d. Eviction
e. Expiration
f. ?
8. It seems; redis knows type(list, hashes, string ,set ..) of each key. Thus for each operation we need to make sure type of key. In current implementation we have different region for each redis type. Thus we have another region(metaTypeRegion) which keeps type for each key. This makes any operation in geode slow as it needs to verify that type. For instance, creating new key need to make sure its already there or not. Whether we should allow type change or not.
a. Feedback/vote
-- type change of key
-- Can we allow two key with same name but two differnt type (as it will endup in two different geode-region)
String type "key1" in string region
HLL type "key1" in HLL region
b. any other feedback
9. Transactions:
a. we will not support transaction in redisAdapter as geode transaction are limited to single node.
b. feedback?
10. Redis COMMAND (https://redis.io/commands/command)
a. should we implement this "COMMAND" ?
11. Any other redis command we should consider?
Thanks.
Hitesh
Re: GeodeRedisAdapter improvments/feedback
Posted by Gregory Green <gg...@pivotal.io>.
Hitesh and Team,
Also, I think geospatial support in core Gemfire that can be exposed thru
the following Redis GEO... commands would be great
GEOADD
GEODIST
GEOHASH
GEOPOS
GEORADIUS
GEORADIUSBYMEMBER
On Wed, Feb 15, 2017 at 10:48 AM, Gregory Green <gg...@pivotal.io> wrote:
> Hello Hitesh,
>
> The following is my feedback.
>
> *1. Redis Type String*
> I like the idea of creating a region upfront. If we are still using the
> convention that internal region names start with "__" , then I would
> suggest something like a region named "__RedisString region"
>
> *2. List Type*
>
> I propose using a single partition region (ex: "__RedisList") for the List
> commands.
>
> Region<byteArrayWrapper,ArrayList<byteArrayWrapper>> region;
>
> //Note: ByteArrayWrapper is what the current RedisAdapter uses as its data
> type. It converts strings to bytes using UTF8 encoding
>
> Example Redis commands
>
> RPUSH mylist A =>
>
> Region<ByteArrayWrapper,List<ByteArrayWrapper>> region = getRegion("__RedisList")
> List list = getOrCreateList(mylist);
> list.add(A)
> region.put(mylist,list)
>
> *3. Hashes*
>
> Based on my Spring Data Redis testing for Hash/object support.
>
> HMSET and similar Hash commands are submitted in the following format:
> HMSET region:key [field value]+ I proposed creating regions with the
> following format:
>
> Region<ByteArrayWrapper,Map<ByteArrayWrapper,ByteArrayWrapper>> region;
>
> Also see Hashes section at the following URL https://redis.io/topics/da
> ta-types
>
> Example Redis command:
>
> HMSET companies:100 _class io.pivotal.redis.gemfire.example.repository.Company
> id 100 name nylaInc email info@nylaInc.io website nylaInc.io taxID id:1
> address.address1 address1 address.address2 address2 address.cityTown
> cityTown address.stateProvince stateProvince address.zip zip
> address.country country
>
> =>
>
> //Pseudo Access code
> Region<ByteArrayWrapper,Map<ByteArrayWrapper,ByteArrayWrapper>> companiesRegion = getRegion("companies")
> companiesRegion.put(100, toMap(fieldValues))
>
> //------
>
> // HGETALL region:key
>
> HGETALL companies:100 =>
>
> Region<key,Map<field,value>> companiesRegion = getRegion("companies")
> return companiesRegion.get(100)
>
> //HSET region:key field value
>
> HSET companies:100 email updated@pivotal.io =>
>
> Region<key,Map<field,value>> companiesRegion = getRegion("companies");
> Map map = companiesRegion.get(100)
> map.set(email,updated@pivotal.io)
> companiesRegion.put(100,map);
>
> FYI - I started to implement this and hope to submit a pull request soon
> related to GEODE-2469.
>
>
> *4. Set*
>
> I propose using a single partition region (ex: __RedisSET) for the SET
> commands.
>
> Region<byteArrayWrapper,HashSet<byteArrayWrapper>> region;
>
> Example Redis commands
>
> SADD myset "Hello" =>
>
> Region<ByteArrayWrapper,Set<ByteArrayWrapper>> region = getRegion("__RedisSET");
> Set set = region(myset)
> boolean bool = set.add(Hello)
> if(bool){
> region.put(myset,set)
> }
> return bool;
>
> SMEMBERS myset "Hello" =>
>
> Region<ByteArrayWrapper,Set<ByteArrayWrapper>> region = getRegion("_RedisSET");
> Set set = region(myset)
> return set.contains(Hello)s
>
> FYI - I started to implement this and hope to submit a pull request soon
> related to GEODE-2469.
>
>
> *5. SortedSets *
>
> I propose using a single partition region for the SET commands.
>
> Region<byteArrayWrapper,TreeSet<byteArrayWrapper>> region;
>
> 6. Default config for geode-region (vote)
>
> I think the default setting should be partitioned with persistence and no
> redundant copies.
>
> 7. It seems; redis knows type(list, hashes, string ,set ..) of each key...
>
> I suggested most operations can assume all keys are strings in UTF8 byte
> encoding, not sure if there are any mathematical number based Redis
> commands that need numbers.
>
> *8. Transactions:*
>
> +1 I agree to not support transactions
>
> *9. Redis COMMAND* (https://redis.io/commands/comman
> <https://redis.io/commands/command>
>
> +1 for implementing the "COMMAND"
>
>
> ---------- Forwarded message ----------
> From: Hitesh Khamesra <hi...@yahoo.com.invalid>
> Date: Tue, Feb 14, 2017 at 5:36 PM
> Subject: GeodeRedisAdapter improvments/feedback
> To: Geode <de...@geode.apache.org>, "user@geode.apache.org" <
> user@geode.apache.org>
>
>
> Current GeodeRedisAdapter implementation is based on
> https://cwiki.apache.org/confluence/display/GEODE/Geode+Redi
> s+Adapter+Proposal.
> We are looking for some feedback on Redis commands and their mapping to
> geode region.
>
> 1. Redis Type String
> a. Usage Set k1 v1
> b. Current implementation creates "STRING_REGION" geode-partition-region
> upfront
> c. This k1/v1 are geode-region key/value
> d. Any feedback?
>
> 2. List Type
> a. usage "rpush mylist A"
> b. Current implementation maps each list to geode-partition-region(i.e.
> mylist is geode-partition-region); with the ability to get item from
> head/tail
> c. Feedback/vote
> -- List type operation at region-entry level;
> -- region-key = "mylist"
> -- region-value = Arraylist (will support all redis list ops)
> d. Feedback/vote: both behavior is desirable
>
>
> 3. Hashes
> a. this represents field-value or java bean object
> b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
> c. Current implementation maps each hashes to
> geode-partition-region(i.e. user1000 is geode-partition-region)
> d. Feedback/vote
> -- Should we map hashes to region-entry
> -- region-key = user1000
> -- region-value = map
> -- This will provide java bean sort to behaviour with 10s of
> field-value
> -- Personally I would prefer this..
> e. Feedback/vote: both behaviour is desirable
>
> 4. Sets
> a. This represents unique keys in set
> b. usage "sadd myset 1 2 3"
> c. Current implementation maps each sadd to geode-partition-region(i.e.
> myset is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = myset
> -- region-value = Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 5. SortedSets
> a. This represents unique keys in set with score (usecase Query top-10)
> b. usage "zadd hackers 1940 "Alan Kay""
> c. Current implementation maps each zadd to geode-partition-region(i.e.
> hackers is geode-partition-region)
> d. Feedback/vote
> -- Should we map set to region-entry
> -- region-key = hackers
> -- region-value = Sorted Hashset
> e. Feedback/vote: both behaviour is desirable
>
> 6. HyperLogLogs
> a. A HyperLogLog is a probabilistic data structure used in order to
> count unique things (technically this is referred to estimating the
> cardinality of a set).
> b. usage "pfadd hll a b c d"
> c. Current implementation creates "HLL_REGION" geode-partition-region
> upfront
> d. hll becomes region-key and value is HLL object
> e. any feedback?
>
> 7. Default config for geode-region (vote)
> a. partition region
> b. 1 redundant copy
> c. Persistence
> d. Eviction
> e. Expiration
> f. ?
>
> 8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
> Thus for each operation we need to make sure type of key. In current
> implementation we have different region for each redis type. Thus we have
> another region(metaTypeRegion) which keeps type for each key. This makes
> any operation in geode slow as it needs to verify that type. For instance,
> creating new key need to make sure its already there or not. Whether we
> should allow type change or not.
> a. Feedback/vote
> -- type change of key
> -- Can we allow two key with same name but two differnt type (as it
> will endup in two different geode-region)
> String type "key1" in string region
> HLL type "key1" in HLL region
> b. any other feedback
>
> 9. Transactions:
> a. we will not support transaction in redisAdapter as geode transaction
> are limited to single node.
> b. feedback?
>
> 10. Redis COMMAND (https://redis.io/commands/command)
> a. should we implement this "COMMAND" ?
>
> 11. Any other redis command we should consider?
>
>
> Thanks.Hitesh
>
>
>
>
> --
> *Gregory Green* (Senior Data Engineer)
> ggreen@pivotal.io
>
--
*Gregory Green* (Senior Data Engineer)
ggreen@pivotal.io
201.736.1016
PTO Alert: Feb 14
Fwd: GeodeRedisAdapter improvments/feedback
Posted by Gregory Green <gg...@pivotal.io>.
Hello Hitesh,
The following is my feedback.
*1. Redis Type String*
I like the idea of creating a region upfront. If we are still using the
convention that internal region names start with "__" , then I would
suggest something like a region named "__RedisString region"
*2. List Type*
I propose using a single partition region (ex: "__RedisList") for the List
commands.
Region<byteArrayWrapper,ArrayList<byteArrayWrapper>> region;
//Note: ByteArrayWrapper is what the current RedisAdapter uses as its data
type. It converts strings to bytes using UTF8 encoding
Example Redis commands
RPUSH mylist A =>
Region<ByteArrayWrapper,List<ByteArrayWrapper>> region =
getRegion("__RedisList")
List list = getOrCreateList(mylist);
list.add(A)
region.put(mylist,list)
*3. Hashes*
Based on my Spring Data Redis testing for Hash/object support.
HMSET and similar Hash commands are submitted in the following format:
HMSET region:key [field value]+ I proposed creating regions with the
following format:
Region<ByteArrayWrapper,Map<ByteArrayWrapper,ByteArrayWrapper>> region;
Also see Hashes section at the following URL https://redis.io/topics/
data-types
Example Redis command:
HMSET companies:100 _class io.pivotal.redis.gemfire.example.repository.Company
id 100 name nylaInc email info@nylaInc.io website nylaInc.io taxID id:1
address.address1 address1 address.address2 address2 address.cityTown
cityTown address.stateProvince stateProvince address.zip zip
address.country country
=>
//Pseudo Access code
Region<ByteArrayWrapper,Map<ByteArrayWrapper,ByteArrayWrapper>>
companiesRegion = getRegion("companies")
companiesRegion.put(100, toMap(fieldValues))
//------
// HGETALL region:key
HGETALL companies:100 =>
Region<key,Map<field,value>> companiesRegion = getRegion("companies")
return companiesRegion.get(100)
//HSET region:key field value
HSET companies:100 email updated@pivotal.io =>
Region<key,Map<field,value>> companiesRegion = getRegion("companies");
Map map = companiesRegion.get(100)
map.set(email,updated@pivotal.io)
companiesRegion.put(100,map);
FYI - I started to implement this and hope to submit a pull request soon
related to GEODE-2469.
*4. Set*
I propose using a single partition region (ex: __RedisSET) for the SET
commands.
Region<byteArrayWrapper,HashSet<byteArrayWrapper>> region;
Example Redis commands
SADD myset "Hello" =>
Region<ByteArrayWrapper,Set<ByteArrayWrapper>> region = getRegion("__RedisSET");
Set set = region(myset)
boolean bool = set.add(Hello)
if(bool){
region.put(myset,set)
}
return bool;
SMEMBERS myset "Hello" =>
Region<ByteArrayWrapper,Set<ByteArrayWrapper>> region =
getRegion("_RedisSET");
Set set = region(myset)
return set.contains(Hello)s
FYI - I started to implement this and hope to submit a pull request soon
related to GEODE-2469.
*5. SortedSets *
I propose using a single partition region for the SET commands.
Region<byteArrayWrapper,TreeSet<byteArrayWrapper>> region;
6. Default config for geode-region (vote)
I think the default setting should be partitioned with persistence and no
redundant copies.
7. It seems; redis knows type(list, hashes, string ,set ..) of each key...
I suggested most operations can assume all keys are strings in UTF8 byte
encoding, not sure if there are any mathematical number based Redis
commands that need numbers.
*8. Transactions:*
+1 I agree to not support transactions
*9. Redis COMMAND* (https://redis.io/commands/comman
<https://redis.io/commands/command>
+1 for implementing the "COMMAND"
---------- Forwarded message ----------
From: Hitesh Khamesra <hi...@yahoo.com.invalid>
Date: Tue, Feb 14, 2017 at 5:36 PM
Subject: GeodeRedisAdapter improvments/feedback
To: Geode <de...@geode.apache.org>, "user@geode.apache.org" <
user@geode.apache.org>
Current GeodeRedisAdapter implementation is based on
https://cwiki.apache.org/confluence/display/GEODE/Geode+Redi
s+Adapter+Proposal.
We are looking for some feedback on Redis commands and their mapping to
geode region.
1. Redis Type String
a. Usage Set k1 v1
b. Current implementation creates "STRING_REGION" geode-partition-region
upfront
c. This k1/v1 are geode-region key/value
d. Any feedback?
2. List Type
a. usage "rpush mylist A"
b. Current implementation maps each list to geode-partition-region(i.e.
mylist is geode-partition-region); with the ability to get item from
head/tail
c. Feedback/vote
-- List type operation at region-entry level;
-- region-key = "mylist"
-- region-value = Arraylist (will support all redis list ops)
d. Feedback/vote: both behavior is desirable
3. Hashes
a. this represents field-value or java bean object
b. usage "hmset user1000 username antirez birthyear 1977 verified 1"
c. Current implementation maps each hashes to geode-partition-region(i.e.
user1000 is geode-partition-region)
d. Feedback/vote
-- Should we map hashes to region-entry
-- region-key = user1000
-- region-value = map
-- This will provide java bean sort to behaviour with 10s of field-value
-- Personally I would prefer this..
e. Feedback/vote: both behaviour is desirable
4. Sets
a. This represents unique keys in set
b. usage "sadd myset 1 2 3"
c. Current implementation maps each sadd to geode-partition-region(i.e.
myset is geode-partition-region)
d. Feedback/vote
-- Should we map set to region-entry
-- region-key = myset
-- region-value = Hashset
e. Feedback/vote: both behaviour is desirable
5. SortedSets
a. This represents unique keys in set with score (usecase Query top-10)
b. usage "zadd hackers 1940 "Alan Kay""
c. Current implementation maps each zadd to geode-partition-region(i.e.
hackers is geode-partition-region)
d. Feedback/vote
-- Should we map set to region-entry
-- region-key = hackers
-- region-value = Sorted Hashset
e. Feedback/vote: both behaviour is desirable
6. HyperLogLogs
a. A HyperLogLog is a probabilistic data structure used in order to count
unique things (technically this is referred to estimating the cardinality
of a set).
b. usage "pfadd hll a b c d"
c. Current implementation creates "HLL_REGION" geode-partition-region
upfront
d. hll becomes region-key and value is HLL object
e. any feedback?
7. Default config for geode-region (vote)
a. partition region
b. 1 redundant copy
c. Persistence
d. Eviction
e. Expiration
f. ?
8. It seems; redis knows type(list, hashes, string ,set ..) of each key.
Thus for each operation we need to make sure type of key. In current
implementation we have different region for each redis type. Thus we have
another region(metaTypeRegion) which keeps type for each key. This makes
any operation in geode slow as it needs to verify that type. For instance,
creating new key need to make sure its already there or not. Whether we
should allow type change or not.
a. Feedback/vote
-- type change of key
-- Can we allow two key with same name but two differnt type (as it
will endup in two different geode-region)
String type "key1" in string region
HLL type "key1" in HLL region
b. any other feedback
9. Transactions:
a. we will not support transaction in redisAdapter as geode transaction
are limited to single node.
b. feedback?
10. Redis COMMAND (https://redis.io/commands/command)
a. should we implement this "COMMAND" ?
11. Any other redis command we should consider?
Thanks.Hitesh
--
*Gregory Green* (Senior Data Engineer)
ggreen@pivotal.io