You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Samuel Doyle <sa...@gmail.com> on 2017/05/08 22:27:02 UTC
Joining on multiple row values produced by TableFunction
I want to do something like the following
.join("fields(fields) as (name, content)")
.where("text = 'password for user' && name='text' &&
!content.like('%accepted%') && name='appname' && content.like('%hostd%')")
Fields collects 4 rows in this case which contain those values
This doesn't work with flink 1.2. Is there a way to accomplish this?
…
--
Sent from my phone
Re: Joining on multiple row values produced by TableFunction
Posted by Samuel Doyle <sa...@gmail.com>.
Great, thanks for the information! Looking forward to the new release.
S.D.
On Tue, May 9, 2017 at 1:40 PM Fabian Hueske <fh...@gmail.com> wrote:
> Flink 1.2 does not support windows in SQL.
> The upcoming Flink 1.3, which is currently in testing phase and should be
> released in a few weeks, allows you define several types of windows in SQL.
>
> There are three types of group windows [1] (note that the syntax for the
> time indicators changed a bit, check the test cases for example [2]) and
> OVER windows as know from standard SQL (limitations are all aggregations in
> a SELECT stmt have to operate on the same window, order by only on time,
> and no FOLLOWING rows/range). Examples for OVER windows are in the tests as
> well [3].
>
> Hope this helps,
> Fabian
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/table_api.html#group-windows
> [2]
> https://github.com/apache/flink/blob/release-1.3/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/scala/stream/sql/WindowAggregateTest.scala
> [3]
> https://github.com/apache/flink/blob/release-1.3/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/scala/stream/sql/OverWindowITCase.scala
>
> 2017-05-09 21:59 GMT+02:00 Samuel Doyle <sa...@gmail.com>:
>
>> Is it possible to define windows in such a way that I can use straight
>> sql like with the table environment vs the sql 'like' syntax you use when
>> using the table api?
>> My project is based off Java.
>>
>> Thanks
>>
>> On Tue, May 9, 2017 at 10:34 AM Samuel Doyle <sa...@gmail.com>
>> wrote:
>>
>>> I was able to get it work, albeit probably not the best solution, by
>>> calling the table function twice and assigning different column aliases
>>>
>>> On Tue, May 9, 2017, 12:43 AM Fabian Hueske <fh...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it might be a problem, that your TableFunction is called like an
>>>> attribute (both are called fields).
>>>> If that's not the case, can you provide more information?
>>>>
>>>> - Is there an error message?
>>>> - Yes: what's the error message
>>>> - No: what's the output and what would you expect?
>>>> - Is this for batch or streaming?
>>>>
>>>> Thanks, Fabian
>>>>
>>>> 2017-05-09 0:27 GMT+02:00 Samuel Doyle <sa...@gmail.com>:
>>>>
>>>>> I want to do something like the following
>>>>>
>>>>> .join("fields(fields) as (name, content)")
>>>>> .where("text = 'password for user' && name='text' &&
>>>>> !content.like('%accepted%') && name='appname' && content.like('%hostd%')")
>>>>>
>>>>> Fields collects 4 rows in this case which contain those values
>>>>>
>>>>> This doesn't work with flink 1.2. Is there a way to accomplish this?
>>>>> …
>>>>>
>>>>> --
>>>>>
>>>>> Sent from my phone
>>>>>
>>>>
>>>> --
>>>
>>> Sent from my phone
>>>
>> --
>>
>> Sent from my phone
>>
>
> --
Sent from my phone
Re: Joining on multiple row values produced by TableFunction
Posted by Fabian Hueske <fh...@gmail.com>.
Flink 1.2 does not support windows in SQL.
The upcoming Flink 1.3, which is currently in testing phase and should be
released in a few weeks, allows you define several types of windows in SQL.
There are three types of group windows [1] (note that the syntax for the
time indicators changed a bit, check the test cases for example [2]) and
OVER windows as know from standard SQL (limitations are all aggregations in
a SELECT stmt have to operate on the same window, order by only on time,
and no FOLLOWING rows/range). Examples for OVER windows are in the tests as
well [3].
Hope this helps,
Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/table_api.html#group-windows
[2]
https://github.com/apache/flink/blob/release-1.3/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/scala/stream/sql/WindowAggregateTest.scala
[3]
https://github.com/apache/flink/blob/release-1.3/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/scala/stream/sql/OverWindowITCase.scala
2017-05-09 21:59 GMT+02:00 Samuel Doyle <sa...@gmail.com>:
> Is it possible to define windows in such a way that I can use straight sql
> like with the table environment vs the sql 'like' syntax you use when using
> the table api?
> My project is based off Java.
>
> Thanks
>
> On Tue, May 9, 2017 at 10:34 AM Samuel Doyle <sa...@gmail.com>
> wrote:
>
>> I was able to get it work, albeit probably not the best solution, by
>> calling the table function twice and assigning different column aliases
>>
>> On Tue, May 9, 2017, 12:43 AM Fabian Hueske <fh...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it might be a problem, that your TableFunction is called like an
>>> attribute (both are called fields).
>>> If that's not the case, can you provide more information?
>>>
>>> - Is there an error message?
>>> - Yes: what's the error message
>>> - No: what's the output and what would you expect?
>>> - Is this for batch or streaming?
>>>
>>> Thanks, Fabian
>>>
>>> 2017-05-09 0:27 GMT+02:00 Samuel Doyle <sa...@gmail.com>:
>>>
>>>> I want to do something like the following
>>>>
>>>> .join("fields(fields) as (name, content)")
>>>> .where("text = 'password for user' && name='text' &&
>>>> !content.like('%accepted%') && name='appname' && content.like('%hostd%')")
>>>>
>>>> Fields collects 4 rows in this case which contain those values
>>>>
>>>> This doesn't work with flink 1.2. Is there a way to accomplish this?
>>>> …
>>>>
>>>> --
>>>>
>>>> Sent from my phone
>>>>
>>>
>>> --
>>
>> Sent from my phone
>>
> --
>
> Sent from my phone
>
Re: Joining on multiple row values produced by TableFunction
Posted by Samuel Doyle <sa...@gmail.com>.
Is it possible to define windows in such a way that I can use straight sql
like with the table environment vs the sql 'like' syntax you use when using
the table api?
My project is based off Java.
Thanks
On Tue, May 9, 2017 at 10:34 AM Samuel Doyle <sa...@gmail.com> wrote:
> I was able to get it work, albeit probably not the best solution, by
> calling the table function twice and assigning different column aliases
>
> On Tue, May 9, 2017, 12:43 AM Fabian Hueske <fh...@gmail.com> wrote:
>
>> Hi,
>>
>> it might be a problem, that your TableFunction is called like an
>> attribute (both are called fields).
>> If that's not the case, can you provide more information?
>>
>> - Is there an error message?
>> - Yes: what's the error message
>> - No: what's the output and what would you expect?
>> - Is this for batch or streaming?
>>
>> Thanks, Fabian
>>
>> 2017-05-09 0:27 GMT+02:00 Samuel Doyle <sa...@gmail.com>:
>>
>>> I want to do something like the following
>>>
>>> .join("fields(fields) as (name, content)")
>>> .where("text = 'password for user' && name='text' &&
>>> !content.like('%accepted%') && name='appname' && content.like('%hostd%')")
>>>
>>> Fields collects 4 rows in this case which contain those values
>>>
>>> This doesn't work with flink 1.2. Is there a way to accomplish this?
>>> …
>>>
>>> --
>>>
>>> Sent from my phone
>>>
>>
>> --
>
> Sent from my phone
>
--
Sent from my phone
Re: Joining on multiple row values produced by TableFunction
Posted by Samuel Doyle <sa...@gmail.com>.
I was able to get it work, albeit probably not the best solution, by
calling the table function twice and assigning different column aliases
On Tue, May 9, 2017, 12:43 AM Fabian Hueske <fh...@gmail.com> wrote:
> Hi,
>
> it might be a problem, that your TableFunction is called like an attribute
> (both are called fields).
> If that's not the case, can you provide more information?
>
> - Is there an error message?
> - Yes: what's the error message
> - No: what's the output and what would you expect?
> - Is this for batch or streaming?
>
> Thanks, Fabian
>
> 2017-05-09 0:27 GMT+02:00 Samuel Doyle <sa...@gmail.com>:
>
>> I want to do something like the following
>>
>> .join("fields(fields) as (name, content)")
>> .where("text = 'password for user' && name='text' &&
>> !content.like('%accepted%') && name='appname' && content.like('%hostd%')")
>>
>> Fields collects 4 rows in this case which contain those values
>>
>> This doesn't work with flink 1.2. Is there a way to accomplish this?
>> …
>>
>> --
>>
>> Sent from my phone
>>
>
> --
Sent from my phone
Re: Joining on multiple row values produced by TableFunction
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
it might be a problem, that your TableFunction is called like an attribute
(both are called fields).
If that's not the case, can you provide more information?
- Is there an error message?
- Yes: what's the error message
- No: what's the output and what would you expect?
- Is this for batch or streaming?
Thanks, Fabian
2017-05-09 0:27 GMT+02:00 Samuel Doyle <sa...@gmail.com>:
> I want to do something like the following
>
> .join("fields(fields) as (name, content)")
> .where("text = 'password for user' && name='text' &&
> !content.like('%accepted%') && name='appname' && content.like('%hostd%')")
>
> Fields collects 4 rows in this case which contain those values
>
> This doesn't work with flink 1.2. Is there a way to accomplish this?
> …
>
> --
>
> Sent from my phone
>