You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Darshan Singh <da...@gmail.com> on 2018/04/04 19:31:55 UTC
Getting runtime context from scalar and table functions
Hi,
I would like to use accumulators with table /scalar functions. However, I
am not able to figure out how to get the runtime context from inside of
scalar function open method.
Only thing i see is function context which can not provide the runtime
context.
I tried using AbstractRichFunction.getRuntimeContext()
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/api/java/org/apache/flink/api/common/functions/AbstractRichFunction.html#getRuntimeContext-->
. which I guess is not allowed from open of scalar or table functions.
Any help will be appreciated.
Thanks
Re: Getting runtime context from scalar and table functions
Posted by Fabian Hueske <fh...@gmail.com>.
Yes, metrics won't work for this use case.
Before we had proper metrics support, accumulators were often used as a
work around.
In general, the Table API & SQL in Flink are designed to keep all data in
tables and not "leak" data on the side.
Best, Fabian
2018-04-04 22:08 GMT+02:00 Darshan Singh <da...@gmail.com>:
> I doubt the metrics will work as I will need to get the String output. I
> will need to use figure out something else.
>
> Basically I will pass my function a string and will get some columns back
> but if my string is something special. Then I will need to get extra
> information. I am using the data set so I think I wont be able to use
> side-output.
>
> As of now I return my POJO along-with the String(only if it is special)
> and then i select these from table itself. I was just wondering if there
> are other better alternate where my existing table return my POJO and I
> somehow find those special strings once job is done. The number of these
> strings is very small. I thought I could create a custom accumulator which
> will work more like list and in the end I will get all values.
>
> I guess I will need to keep trying something.
>
> Thanks
>
> On Wed, Apr 4, 2018 at 9:01 PM, Darshan Singh <da...@gmail.com>
> wrote:
>
>> Thanks for getting back to me. I was trying to see various options to
>> accumulate(sort of list append) some of my data. The amount of appended
>> data is quite small so I wanted to see if I can use accumulators and once
>> the job is done I can get all the data and then use the way I want to use.
>>
>> I guess I will try the metrics as well to see how that goes.
>>
>> Thanks
>>
>> On Wed, Apr 4, 2018 at 8:51 PM, Fabian Hueske <fh...@gmail.com> wrote:
>>
>>> Hi Darshan,
>>>
>>> Accumulators are not exposed to UDFs of the Table API / SQL.
>>> What's your use case for these? Would metrics do the job as well?
>>>
>>> Best, Fabian
>>>
>>> 2018-04-04 21:31 GMT+02:00 Darshan Singh <da...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> I would like to use accumulators with table /scalar functions. However,
>>>> I am not able to figure out how to get the runtime context from inside of
>>>> scalar function open method.
>>>> Only thing i see is function context which can not provide the runtime
>>>> context.
>>>>
>>>> I tried using AbstractRichFunction.getRuntimeContext()
>>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/api/java/org/apache/flink/api/common/functions/AbstractRichFunction.html#getRuntimeContext-->
>>>> . which I guess is not allowed from open of scalar or table functions.
>>>>
>>>> Any help will be appreciated.
>>>>
>>>> Thanks
>>>>
>>>
>>>
>>
>
Re: Getting runtime context from scalar and table functions
Posted by Darshan Singh <da...@gmail.com>.
I doubt the metrics will work as I will need to get the String output. I
will need to use figure out something else.
Basically I will pass my function a string and will get some columns back
but if my string is something special. Then I will need to get extra
information. I am using the data set so I think I wont be able to use
side-output.
As of now I return my POJO along-with the String(only if it is special) and
then i select these from table itself. I was just wondering if there are
other better alternate where my existing table return my POJO and I somehow
find those special strings once job is done. The number of these strings is
very small. I thought I could create a custom accumulator which will work
more like list and in the end I will get all values.
I guess I will need to keep trying something.
Thanks
On Wed, Apr 4, 2018 at 9:01 PM, Darshan Singh <da...@gmail.com>
wrote:
> Thanks for getting back to me. I was trying to see various options to
> accumulate(sort of list append) some of my data. The amount of appended
> data is quite small so I wanted to see if I can use accumulators and once
> the job is done I can get all the data and then use the way I want to use.
>
> I guess I will try the metrics as well to see how that goes.
>
> Thanks
>
> On Wed, Apr 4, 2018 at 8:51 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> Hi Darshan,
>>
>> Accumulators are not exposed to UDFs of the Table API / SQL.
>> What's your use case for these? Would metrics do the job as well?
>>
>> Best, Fabian
>>
>> 2018-04-04 21:31 GMT+02:00 Darshan Singh <da...@gmail.com>:
>>
>>> Hi,
>>>
>>> I would like to use accumulators with table /scalar functions. However,
>>> I am not able to figure out how to get the runtime context from inside of
>>> scalar function open method.
>>> Only thing i see is function context which can not provide the runtime
>>> context.
>>>
>>> I tried using AbstractRichFunction.getRuntimeContext()
>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/api/java/org/apache/flink/api/common/functions/AbstractRichFunction.html#getRuntimeContext-->
>>> . which I guess is not allowed from open of scalar or table functions.
>>>
>>> Any help will be appreciated.
>>>
>>> Thanks
>>>
>>
>>
>
Re: Getting runtime context from scalar and table functions
Posted by Darshan Singh <da...@gmail.com>.
Thanks for getting back to me. I was trying to see various options to
accumulate(sort of list append) some of my data. The amount of appended
data is quite small so I wanted to see if I can use accumulators and once
the job is done I can get all the data and then use the way I want to use.
I guess I will try the metrics as well to see how that goes.
Thanks
On Wed, Apr 4, 2018 at 8:51 PM, Fabian Hueske <fh...@gmail.com> wrote:
> Hi Darshan,
>
> Accumulators are not exposed to UDFs of the Table API / SQL.
> What's your use case for these? Would metrics do the job as well?
>
> Best, Fabian
>
> 2018-04-04 21:31 GMT+02:00 Darshan Singh <da...@gmail.com>:
>
>> Hi,
>>
>> I would like to use accumulators with table /scalar functions. However, I
>> am not able to figure out how to get the runtime context from inside of
>> scalar function open method.
>> Only thing i see is function context which can not provide the runtime
>> context.
>>
>> I tried using AbstractRichFunction.getRuntimeContext()
>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/api/java/org/apache/flink/api/common/functions/AbstractRichFunction.html#getRuntimeContext-->
>> . which I guess is not allowed from open of scalar or table functions.
>>
>> Any help will be appreciated.
>>
>> Thanks
>>
>
>
Re: Getting runtime context from scalar and table functions
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Darshan,
Accumulators are not exposed to UDFs of the Table API / SQL.
What's your use case for these? Would metrics do the job as well?
Best, Fabian
2018-04-04 21:31 GMT+02:00 Darshan Singh <da...@gmail.com>:
> Hi,
>
> I would like to use accumulators with table /scalar functions. However, I
> am not able to figure out how to get the runtime context from inside of
> scalar function open method.
> Only thing i see is function context which can not provide the runtime
> context.
>
> I tried using AbstractRichFunction.getRuntimeContext()
> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/api/java/org/apache/flink/api/common/functions/AbstractRichFunction.html#getRuntimeContext-->
> . which I guess is not allowed from open of scalar or table functions.
>
> Any help will be appreciated.
>
> Thanks
>