You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by Amit Pandey <am...@gmail.com> on 2016/12/14 17:04:01 UTC
Quesntion about Geode functions
Hey All,
I want to know does Geode functions execute on collocated data. My usecase
is that I have portioned cache and I want to do some computations in
parallel in all the nodes.
Now does Geode automatically distribute functions so that the code only
works on data in a particular node?
Or is there mechanism to do so?
I think an example will make it clear. Lets say I have some stocks and
their prices in a cache. Now I want to do some calculations , so can I
redirect code to be executed on the node holding the data for Pivotal stock
automatically?
Re: Quesntion about Geode functions
Posted by Amit Pandey <am...@gmail.com>.
Also I didn't mean region sorry for the confusion I meant server node. It
was a typo.
On Wed, Dec 14, 2016 at 11:49 PM, Amit Pandey <am...@gmail.com>
wrote:
> Hi Dan and Anil ,
> Thanks for the clarification.
>
> Dan,
>
> I think your explanation makes it clear to me how to go about it.
>
> Just as a clarification in the geode doc there are mentions of you can
> execute functions "*On a specific server or set of servers", *which I
> imagine is pretty useful, but in my usecase I don't know which actual node
> has the data but want to execute code on collocated data. So it appears to
> me that the filer example will work and that is great for my usecase.
>
> Regards
>
>
>
> On Wed, Dec 14, 2016 at 11:26 PM, Dan Smith <ds...@pivotal.io> wrote:
>
>> Hi Amit,
>>
>> You can target a particular stock or stocks by using the filter arguments
>> of a function. For example FunctionService.onRegion(myReg
>> ion).withFilter(myStock).execute(...). Your function will then be sent
>> to the node or nodes that have that particular stock. If you pass more than
>> one stock, you function will be executed in parallel on different nodes.
>> Within each execution, you can use RegionFunctionContext.getFilter() to
>> get the list of stocks that should be handled on that node.
>>
>> I'm not quite sure I understand what you mean when you say you don't know
>> what region the data is in. Do you have multiple, colocated regions that
>> you need to examine? If the data is colocated then the above technique
>> should still work, you can access all of the colocated regions from within
>> your function as well.
>>
>> -Dan
>>
>> On Wed, Dec 14, 2016 at 9:52 AM, Anilkumar Gingade <ag...@pivotal.io>
>> wrote:
>>
>>> Amit,
>>>
>>> >> But I don't know which regions has the data.
>>>
>>> The data aware function routing is supported using the region name and
>>> keys...In your case, since you don't know the region, you have to execute
>>> the function on all nodes and identify the region.
>>>
>>> Just curious, how are injecting data into the regions? can you identify
>>> the region names using keys?
>>>
>>> -Anil.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Dec 14, 2016 at 9:39 AM, Amit Pandey <am...@gmail.com>
>>> wrote:
>>>
>>>> hey Anil,
>>>>
>>>> Thanks. My case is data dependent. But I don't know which regions has
>>>> the data. But I would like the code to go and execute to the node where it
>>>> is located.
>>>>
>>>> I see this in that doc :- "* a region—*If you are executing a
>>>> data-dependent function, specify a region and, optionally, a set of keys on
>>>> which to run the function. The method FunctionService.onRegion()
>>>> directs a data-dependent function to execute on a specific region."
>>>>
>>>> So does it mean if I specify a set of keys, the code will be executed
>>>> at the place where the data is located?
>>>>
>>>> Regards
>>>>
>>>> On Wed, Dec 14, 2016 at 10:49 PM, Anilkumar Gingade <
>>>> agingade@pivotal.io> wrote:
>>>>
>>>>> Amit,
>>>>>
>>>>> Yes, it does what you are looking for...
>>>>>
>>>>> You can find more about function execution at:
>>>>> http://geode.apache.org/docs/guide/developing/function_exec/
>>>>> how_function_execution_works.html
>>>>>
>>>>> If you have further question don't hesitate to ask.
>>>>>
>>>>> -Anil.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Dec 14, 2016 at 9:04 AM, Amit Pandey <
>>>>> amit.pandey2103@gmail.com> wrote:
>>>>>
>>>>>> Hey All,
>>>>>>
>>>>>> I want to know does Geode functions execute on collocated data. My
>>>>>> usecase is that I have portioned cache and I want to do some computations
>>>>>> in parallel in all the nodes.
>>>>>>
>>>>>> Now does Geode automatically distribute functions so that the code
>>>>>> only works on data in a particular node?
>>>>>>
>>>>>> Or is there mechanism to do so?
>>>>>>
>>>>>> I think an example will make it clear. Lets say I have some stocks
>>>>>> and their prices in a cache. Now I want to do some calculations , so can I
>>>>>> redirect code to be executed on the node holding the data for Pivotal stock
>>>>>> automatically?
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Quesntion about Geode functions
Posted by Amit Pandey <am...@gmail.com>.
Hi Dan and Anil ,
Thanks for the clarification.
Dan,
I think your explanation makes it clear to me how to go about it.
Just as a clarification in the geode doc there are mentions of you can
execute functions "*On a specific server or set of servers", *which I
imagine is pretty useful, but in my usecase I don't know which actual node
has the data but want to execute code on collocated data. So it appears to
me that the filer example will work and that is great for my usecase.
Regards
On Wed, Dec 14, 2016 at 11:26 PM, Dan Smith <ds...@pivotal.io> wrote:
> Hi Amit,
>
> You can target a particular stock or stocks by using the filter arguments
> of a function. For example FunctionService.onRegion(myReg
> ion).withFilter(myStock).execute(...). Your function will then be sent to
> the node or nodes that have that particular stock. If you pass more than
> one stock, you function will be executed in parallel on different nodes.
> Within each execution, you can use RegionFunctionContext.getFilter() to
> get the list of stocks that should be handled on that node.
>
> I'm not quite sure I understand what you mean when you say you don't know
> what region the data is in. Do you have multiple, colocated regions that
> you need to examine? If the data is colocated then the above technique
> should still work, you can access all of the colocated regions from within
> your function as well.
>
> -Dan
>
> On Wed, Dec 14, 2016 at 9:52 AM, Anilkumar Gingade <ag...@pivotal.io>
> wrote:
>
>> Amit,
>>
>> >> But I don't know which regions has the data.
>>
>> The data aware function routing is supported using the region name and
>> keys...In your case, since you don't know the region, you have to execute
>> the function on all nodes and identify the region.
>>
>> Just curious, how are injecting data into the regions? can you identify
>> the region names using keys?
>>
>> -Anil.
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Dec 14, 2016 at 9:39 AM, Amit Pandey <am...@gmail.com>
>> wrote:
>>
>>> hey Anil,
>>>
>>> Thanks. My case is data dependent. But I don't know which regions has
>>> the data. But I would like the code to go and execute to the node where it
>>> is located.
>>>
>>> I see this in that doc :- "* a region—*If you are executing a
>>> data-dependent function, specify a region and, optionally, a set of keys on
>>> which to run the function. The method FunctionService.onRegion()
>>> directs a data-dependent function to execute on a specific region."
>>>
>>> So does it mean if I specify a set of keys, the code will be executed at
>>> the place where the data is located?
>>>
>>> Regards
>>>
>>> On Wed, Dec 14, 2016 at 10:49 PM, Anilkumar Gingade <agingade@pivotal.io
>>> > wrote:
>>>
>>>> Amit,
>>>>
>>>> Yes, it does what you are looking for...
>>>>
>>>> You can find more about function execution at:
>>>> http://geode.apache.org/docs/guide/developing/function_exec/
>>>> how_function_execution_works.html
>>>>
>>>> If you have further question don't hesitate to ask.
>>>>
>>>> -Anil.
>>>>
>>>>
>>>>
>>>> On Wed, Dec 14, 2016 at 9:04 AM, Amit Pandey <amit.pandey2103@gmail.com
>>>> > wrote:
>>>>
>>>>> Hey All,
>>>>>
>>>>> I want to know does Geode functions execute on collocated data. My
>>>>> usecase is that I have portioned cache and I want to do some computations
>>>>> in parallel in all the nodes.
>>>>>
>>>>> Now does Geode automatically distribute functions so that the code
>>>>> only works on data in a particular node?
>>>>>
>>>>> Or is there mechanism to do so?
>>>>>
>>>>> I think an example will make it clear. Lets say I have some stocks and
>>>>> their prices in a cache. Now I want to do some calculations , so can I
>>>>> redirect code to be executed on the node holding the data for Pivotal stock
>>>>> automatically?
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Quesntion about Geode functions
Posted by Dan Smith <ds...@pivotal.io>.
Hi Amit,
You can target a particular stock or stocks by using the filter arguments
of a function. For example FunctionService.onRegion(
myRegion).withFilter(myStock).execute(...). Your function will then be sent
to the node or nodes that have that particular stock. If you pass more than
one stock, you function will be executed in parallel on different nodes.
Within each execution, you can use RegionFunctionContext.getFilter() to get
the list of stocks that should be handled on that node.
I'm not quite sure I understand what you mean when you say you don't know
what region the data is in. Do you have multiple, colocated regions that
you need to examine? If the data is colocated then the above technique
should still work, you can access all of the colocated regions from within
your function as well.
-Dan
On Wed, Dec 14, 2016 at 9:52 AM, Anilkumar Gingade <ag...@pivotal.io>
wrote:
> Amit,
>
> >> But I don't know which regions has the data.
>
> The data aware function routing is supported using the region name and
> keys...In your case, since you don't know the region, you have to execute
> the function on all nodes and identify the region.
>
> Just curious, how are injecting data into the regions? can you identify
> the region names using keys?
>
> -Anil.
>
>
>
>
>
>
>
> On Wed, Dec 14, 2016 at 9:39 AM, Amit Pandey <am...@gmail.com>
> wrote:
>
>> hey Anil,
>>
>> Thanks. My case is data dependent. But I don't know which regions has the
>> data. But I would like the code to go and execute to the node where it is
>> located.
>>
>> I see this in that doc :- "* a region—*If you are executing a
>> data-dependent function, specify a region and, optionally, a set of keys on
>> which to run the function. The method FunctionService.onRegion() directs
>> a data-dependent function to execute on a specific region."
>>
>> So does it mean if I specify a set of keys, the code will be executed at
>> the place where the data is located?
>>
>> Regards
>>
>> On Wed, Dec 14, 2016 at 10:49 PM, Anilkumar Gingade <ag...@pivotal.io>
>> wrote:
>>
>>> Amit,
>>>
>>> Yes, it does what you are looking for...
>>>
>>> You can find more about function execution at:
>>> http://geode.apache.org/docs/guide/developing/function_exec/
>>> how_function_execution_works.html
>>>
>>> If you have further question don't hesitate to ask.
>>>
>>> -Anil.
>>>
>>>
>>>
>>> On Wed, Dec 14, 2016 at 9:04 AM, Amit Pandey <am...@gmail.com>
>>> wrote:
>>>
>>>> Hey All,
>>>>
>>>> I want to know does Geode functions execute on collocated data. My
>>>> usecase is that I have portioned cache and I want to do some computations
>>>> in parallel in all the nodes.
>>>>
>>>> Now does Geode automatically distribute functions so that the code only
>>>> works on data in a particular node?
>>>>
>>>> Or is there mechanism to do so?
>>>>
>>>> I think an example will make it clear. Lets say I have some stocks and
>>>> their prices in a cache. Now I want to do some calculations , so can I
>>>> redirect code to be executed on the node holding the data for Pivotal stock
>>>> automatically?
>>>>
>>>
>>>
>>
>
Re: Quesntion about Geode functions
Posted by Anilkumar Gingade <ag...@pivotal.io>.
Amit,
>> But I don't know which regions has the data.
The data aware function routing is supported using the region name and
keys...In your case, since you don't know the region, you have to execute
the function on all nodes and identify the region.
Just curious, how are injecting data into the regions? can you identify the
region names using keys?
-Anil.
On Wed, Dec 14, 2016 at 9:39 AM, Amit Pandey <am...@gmail.com>
wrote:
> hey Anil,
>
> Thanks. My case is data dependent. But I don't know which regions has the
> data. But I would like the code to go and execute to the node where it is
> located.
>
> I see this in that doc :- "* a region—*If you are executing a
> data-dependent function, specify a region and, optionally, a set of keys on
> which to run the function. The method FunctionService.onRegion() directs
> a data-dependent function to execute on a specific region."
>
> So does it mean if I specify a set of keys, the code will be executed at
> the place where the data is located?
>
> Regards
>
> On Wed, Dec 14, 2016 at 10:49 PM, Anilkumar Gingade <ag...@pivotal.io>
> wrote:
>
>> Amit,
>>
>> Yes, it does what you are looking for...
>>
>> You can find more about function execution at:
>> http://geode.apache.org/docs/guide/developing/function_exec/
>> how_function_execution_works.html
>>
>> If you have further question don't hesitate to ask.
>>
>> -Anil.
>>
>>
>>
>> On Wed, Dec 14, 2016 at 9:04 AM, Amit Pandey <am...@gmail.com>
>> wrote:
>>
>>> Hey All,
>>>
>>> I want to know does Geode functions execute on collocated data. My
>>> usecase is that I have portioned cache and I want to do some computations
>>> in parallel in all the nodes.
>>>
>>> Now does Geode automatically distribute functions so that the code only
>>> works on data in a particular node?
>>>
>>> Or is there mechanism to do so?
>>>
>>> I think an example will make it clear. Lets say I have some stocks and
>>> their prices in a cache. Now I want to do some calculations , so can I
>>> redirect code to be executed on the node holding the data for Pivotal stock
>>> automatically?
>>>
>>
>>
>
Re: Quesntion about Geode functions
Posted by Amit Pandey <am...@gmail.com>.
hey Anil,
Thanks. My case is data dependent. But I don't know which regions has the
data. But I would like the code to go and execute to the node where it is
located.
I see this in that doc :- "* a region—*If you are executing a
data-dependent function, specify a region and, optionally, a set of keys on
which to run the function. The method FunctionService.onRegion() directs a
data-dependent function to execute on a specific region."
So does it mean if I specify a set of keys, the code will be executed at
the place where the data is located?
Regards
On Wed, Dec 14, 2016 at 10:49 PM, Anilkumar Gingade <ag...@pivotal.io>
wrote:
> Amit,
>
> Yes, it does what you are looking for...
>
> You can find more about function execution at:
> http://geode.apache.org/docs/guide/developing/function_
> exec/how_function_execution_works.html
>
> If you have further question don't hesitate to ask.
>
> -Anil.
>
>
>
> On Wed, Dec 14, 2016 at 9:04 AM, Amit Pandey <am...@gmail.com>
> wrote:
>
>> Hey All,
>>
>> I want to know does Geode functions execute on collocated data. My
>> usecase is that I have portioned cache and I want to do some computations
>> in parallel in all the nodes.
>>
>> Now does Geode automatically distribute functions so that the code only
>> works on data in a particular node?
>>
>> Or is there mechanism to do so?
>>
>> I think an example will make it clear. Lets say I have some stocks and
>> their prices in a cache. Now I want to do some calculations , so can I
>> redirect code to be executed on the node holding the data for Pivotal stock
>> automatically?
>>
>
>
Re: Quesntion about Geode functions
Posted by Anilkumar Gingade <ag...@pivotal.io>.
Amit,
Yes, it does what you are looking for...
You can find more about function execution at:
http://geode.apache.org/docs/guide/developing/function_exec/how_function_execution_works.html
If you have further question don't hesitate to ask.
-Anil.
On Wed, Dec 14, 2016 at 9:04 AM, Amit Pandey <am...@gmail.com>
wrote:
> Hey All,
>
> I want to know does Geode functions execute on collocated data. My usecase
> is that I have portioned cache and I want to do some computations in
> parallel in all the nodes.
>
> Now does Geode automatically distribute functions so that the code only
> works on data in a particular node?
>
> Or is there mechanism to do so?
>
> I think an example will make it clear. Lets say I have some stocks and
> their prices in a cache. Now I want to do some calculations , so can I
> redirect code to be executed on the node holding the data for Pivotal stock
> automatically?
>