You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by Real Wes Williams <Th...@outlook.com> on 2015/11/28 05:36:28 UTC

Region Changes Reference in Function Scatter-Gather Processing

My function returns a different partitioned region depending on if I pass a _single_ key into the .withFilter() set or I pass _multiple_ keys into .withFilter().  What’s going on?

When I pass in a single key into .onRegion(r).withFilter(one key), the region in the local context. In this condition, keySet returns 10,500 keys.

Function:
    Region<String, Trade> tradesRegion = PartitionRegionHelper.getLocalDataForContext(context);
    Object object = tradesRegion.get(aKey);
    log.info("keySet returns " + tradesRegion.keySet().size());


But when I pass in MULTIPLE keys, the region.get returns a null and there are only 53 keys in the region.  When I list all 53 keys, they are the partition keys and not the actual keys of the region.

Can someone explain what’s going on and what is the optimal way to process scatter-gather keys when a partition-resolver is in play? The partition resolver is in the classpath on both client and server.

SINGLE KEY: [info 2015/11/27 23:09:48.299 EST server1 <ServerConnection on port 50920 Thread 6> tid=0x6b] keySet returns 10614

MULTIPLE KEYS: [info 2015/11/27 22:52:46.969 EST server1 <Function Execution Processor1> tid=0x45] keySet returns 53

Thanks,
Wes

Client:
     Set<String> filter = new HashSet<String>();
     filter.add("161231XYZ|AAB3H 01");
 
     /*
      * calls the function across the nodes
      */
     ResultCollector<?, ?> rc = FunctionService
         .onRegion(region)
         .withFilter(filter)
         .execute(CalculateDurationFunction.ID);

Re: Region Changes Reference in Function Scatter-Gather Processing

Posted by Barry Oglesby <bo...@pivotal.io>.
If the filter set contains 1 element, and the server in which the function
is being executed is the single-hop server, then the function is executed
in-line in the ServerConnection thread itself. Otherwise, the function
execution is handled by the FunctionExecution thread pool.

Also, the PartitionRegionHelper.getLocalDataForContext method returns a
LocalDataSet wrapper on the appropriate buckets. The LocalDataSet keySet
method returns a wrapper on the keys of those buckets. Also, the
LocalDataSet get method returns a value if it is contained in the buckets;
otherwise it returns null.


Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at
http://support.pivotal.io/


On Fri, Nov 27, 2015 at 10:23 PM, Real Wes Williams <Th...@outlook.com>
wrote:

> The answer to the mysterious number differences is that the client region
> was a co-located region but not the region I was interested in processing.
>
> Notwithstanding, why the difference in processing between “Server
> Connection” and “Function Processor” when one or multiple keys are passed
> in?
>
> Thanks,
> Wes
>
>
> On Nov 27, 2015, at 11:36 PM, Real Wes Williams <Th...@outlook.com>
> wrote:
>
>
> My function returns a different partitioned region depending on if I pass
> a _single_ key into the .withFilter() set or I pass _multiple_ keys into
> .withFilter().  What’s going on?
>
> When I pass in a single key into .onRegion(r).withFilter(one key), the
> region in the local context. In this condition, keySet returns 10,500 keys.
>
> Function:
>     Region<String, Trade> tradesRegion =
> PartitionRegionHelper.getLocalDataForContext(context);
>     Object object = tradesRegion.get(aKey);
>     log.info("keySet returns " + tradesRegion.keySet().size());
>
>
> But when I pass in MULTIPLE keys, the region.get returns a null and there
> are only 53 keys in the region.  When I list all 53 keys, they are the
> partition keys and not the actual keys of the region.
>
> Can someone explain what’s going on and what is the optimal way to process
> scatter-gather keys when a partition-resolver is in play? The partition
> resolver is in the classpath on both client and server.
>
> SINGLE KEY: [info 2015/11/27 23:09:48.299 EST server1 <ServerConnection
> on port 50920 Thread 6> tid=0x6b] keySet returns 10614
>
> MULTIPLE KEYS: [info 2015/11/27 22:52:46.969 EST server1 <Function
> Execution Processor1> tid=0x45] keySet returns 53
>
> Thanks,
> Wes
>
> Client:
>      Set<String> filter = new HashSet<String>();
>      filter.add("161231XYZ|AAB3H 01");
>
>      /*
>       * calls the function across the nodes
>       */
>      ResultCollector<?, ?> rc = FunctionService
>          .onRegion(region)
>          .withFilter(filter)
>          .execute(CalculateDurationFunction.ID);
>
>
>
>

Re: Region Changes Reference in Function Scatter-Gather Processing

Posted by Real Wes Williams <Th...@outlook.com>.
The answer to the mysterious number differences is that the client region was a co-located region but not the region I was interested in processing. 

Notwithstanding, why the difference in processing between “Server Connection” and “Function Processor” when one or multiple keys are passed in?

Thanks,
Wes


> On Nov 27, 2015, at 11:36 PM, Real Wes Williams <Th...@outlook.com> wrote:
> 
> 
> My function returns a different partitioned region depending on if I pass a _single_ key into the .withFilter() set or I pass _multiple_ keys into .withFilter().  What’s going on?
> 
> When I pass in a single key into .onRegion(r).withFilter(one key), the region in the local context. In this condition, keySet returns 10,500 keys.
> 
> Function:
>     Region<String, Trade> tradesRegion = PartitionRegionHelper.getLocalDataForContext(context);
>     Object object = tradesRegion.get(aKey);
>     log.info("keySet returns " + tradesRegion.keySet().size());
> 
> 
> But when I pass in MULTIPLE keys, the region.get returns a null and there are only 53 keys in the region.  When I list all 53 keys, they are the partition keys and not the actual keys of the region.
> 
> Can someone explain what’s going on and what is the optimal way to process scatter-gather keys when a partition-resolver is in play? The partition resolver is in the classpath on both client and server.
> 
> SINGLE KEY: [info 2015/11/27 23:09:48.299 EST server1 <ServerConnection on port 50920 Thread 6> tid=0x6b] keySet returns 10614
> 
> MULTIPLE KEYS: [info 2015/11/27 22:52:46.969 EST server1 <Function Execution Processor1> tid=0x45] keySet returns 53
> 
> Thanks,
> Wes
> 
> Client:
>      Set<String> filter = new HashSet<String>();
>      filter.add("161231XYZ|AAB3H 01");
>  
>      /*
>       * calls the function across the nodes
>       */
>      ResultCollector<?, ?> rc = FunctionService
>          .onRegion(region)
>          .withFilter(filter)
>          .execute(CalculateDurationFunction.ID);
> 
>