You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Khai Tran <kh...@linkedin.com.INVALID> on 2017/04/05 22:13:13 UTC
Calcite to Beam
Hi,
Just want to check if there is any ongoing effort to convert Calcite
logical plan into Apache Beam APIs for stream processing?
Thanks,
Khai
Re: Calcite to Beam
Posted by Xiaoyong Deng <yo...@gmail.com>.
Hi there,
It's exciting that all of us have the same idea, Calcite to Beam, and our
team already made it's true for the last few months, including
tumbling/hopping window newly released in Calcite 1.12.0, and we will
release in our product environment at the end of this month if no big delay.
We want to share our work to the community, and make Calcite SQL to Beam
more universal and powerful with you.
We made some extension， as follows:
1) add "create table" grammer
2) add "create UDF/UDTF/UDAF" grammer
3) use "view" to support multi-line sql in one DAG
4) add some functions not exist in Calcite now(
https://issues.apache.org/jira/browse/CALCITE-1733)
5) the table operators for SQL on beam
6) ...
for example:
-- DDL for udafCREATE UDAF avg_test(DOUBLE) RETURNS DOUBLE AS
'com.test.AvgTest';
-- DDL for stream input CREATE STREAM TABLE source_1(
rt TIMESTAMP,
f1 BIGINT,
f2 DOUBLE,
f3 VARCHAR,
TIMESTAMP by rt -- assign rt column as rowtime) WITH (
type='socket',
port='9999');
-- DDL for view with group & filter resultCREATE VIEW view_1 (rt, f3,
f2, f1) AS SELECT
HOP_END(rt, INTERVAL '1' SECOND, INTERVAL '5' SECOND) AS hop_rt,
avg_test(f3) AS avg_f3,
f2 AS f2,
f1 AS f1FROM source_1 GROUP BY HOP(rt, INTERVAL '1' SECOND,
INTERVAL '5' SECOND), f1, f2WHERE f1 % 2 = 1;
-- DDL for dimension table (which used as a static table when
join)CREATE DIM TABLE dim_table(
`key` BIGINT,
`value` VARCHAR) WITH (
type='mysql',
jdbcUrl='jdbc:mysql://127.0.0.1:3306/test',
tableName='test',
username='test',
password='test');
-- DDL for join resultCREATE VIEW view_2 (rt, f1, f2, f3, f4) AS SELECT
t1.rt AS rt,
t1.f1 AS f1,
t1.f2 AS f2,
t1.f3 AS f3,
t2.value AS f4FROM
view_1 AS t1JOIN dim_table AS t2
ON t1.f1 = t2.key
-- DDL for result 1CREATE RESULT TABLE result_1(
console_rt TIMESTAMP,
console_f1 BIGINT,
console_f2 DOUBLE,
console_f3 VARCHAR,
console_f4 VARCHAR) WITH (
type='console');
-- DDL for result 2CREATE RESULT TABLE result_2(
mysql_rt TIMESTAMP,
mysql_f1 BIGINT,
mysql_f2 DOUBLE,) WITH (
type='mysql',
...);
-- DML for result_1INSERT INTO result_1SELECT
*FROM view_2WHERE f2 > 3;
-- DML for result_2INSERT INTO result_2SELECT
rt,
f1,
f2FROM view_2WHERE f2 <= 3;
2017-04-06 7:02 GMT+08:00 Mingmin Xu <mi...@gmail.com>:
> We have some work already, will create a feature branch later soon.
>
>
>
> On Wed, Apr 5, 2017 at 3:37 PM, Khai Tran <kh...@linkedin.com.invalid>
> wrote:
>
> > Thanks, Julian. That's what I need.
> >
> > On Wed, Apr 5, 2017 at 3:26 PM, Julian Hyde <jh...@apache.org> wrote:
> >
> > > + dev@beam (please moderate through!)
> > >
> > > There’s only been talk so far.
> > >
> > > I found https://issues.apache.org/jira/browse/BEAM-301 <
> > > https://issues.apache.org/jira/browse/BEAM-301>. Does that match what
> > you
> > > were thinking? If so, let’s continue the conversation there.
> > >
> > > Julian
> > >
> > > > On Apr 5, 2017, at 3:13 PM, Khai Tran <kh...@linkedin.com.INVALID>
> > > wrote:
> > > >
> > > > Hi,
> > > > Just want to check if there is any ongoing effort to convert Calcite
> > > > logical plan into Apache Beam APIs for stream processing?
> > > >
> > > > Thanks,
> > > > Khai
> > >
> > >
> >
>
>
>
> --
> ----
> Mingmin
>
Re: Calcite to Beam
Posted by Mingmin Xu <mi...@gmail.com>.
We have some work already, will create a feature branch later soon.
On Wed, Apr 5, 2017 at 3:37 PM, Khai Tran <kh...@linkedin.com.invalid>
wrote:
> Thanks, Julian. That's what I need.
>
> On Wed, Apr 5, 2017 at 3:26 PM, Julian Hyde <jh...@apache.org> wrote:
>
> > + dev@beam (please moderate through!)
> >
> > There’s only been talk so far.
> >
> > I found https://issues.apache.org/jira/browse/BEAM-301 <
> > https://issues.apache.org/jira/browse/BEAM-301>. Does that match what
> you
> > were thinking? If so, let’s continue the conversation there.
> >
> > Julian
> >
> > > On Apr 5, 2017, at 3:13 PM, Khai Tran <kh...@linkedin.com.INVALID>
> > wrote:
> > >
> > > Hi,
> > > Just want to check if there is any ongoing effort to convert Calcite
> > > logical plan into Apache Beam APIs for stream processing?
> > >
> > > Thanks,
> > > Khai
> >
> >
>
--
----
Mingmin
Re: Calcite to Beam
Posted by Khai Tran <kh...@linkedin.com.INVALID>.
Thanks, Julian. That's what I need.
On Wed, Apr 5, 2017 at 3:26 PM, Julian Hyde <jh...@apache.org> wrote:
> + dev@beam (please moderate through!)
>
> There’s only been talk so far.
>
> I found https://issues.apache.org/jira/browse/BEAM-301 <
> https://issues.apache.org/jira/browse/BEAM-301>. Does that match what you
> were thinking? If so, let’s continue the conversation there.
>
> Julian
>
> > On Apr 5, 2017, at 3:13 PM, Khai Tran <kh...@linkedin.com.INVALID>
> wrote:
> >
> > Hi,
> > Just want to check if there is any ongoing effort to convert Calcite
> > logical plan into Apache Beam APIs for stream processing?
> >
> > Thanks,
> > Khai
>
>
Re: Calcite to Beam
Posted by Julian Hyde <jh...@apache.org>.
+ dev@beam (please moderate through!)
There’s only been talk so far.
I found https://issues.apache.org/jira/browse/BEAM-301 <https://issues.apache.org/jira/browse/BEAM-301>. Does that match what you were thinking? If so, let’s continue the conversation there.
Julian
> On Apr 5, 2017, at 3:13 PM, Khai Tran <kh...@linkedin.com.INVALID> wrote:
>
> Hi,
> Just want to check if there is any ongoing effort to convert Calcite
> logical plan into Apache Beam APIs for stream processing?
>
> Thanks,
> Khai
Re: Calcite to Beam
Posted by Julian Hyde <jh...@apache.org>.
+ dev@beam (please moderate through!)
There’s only been talk so far.
I found https://issues.apache.org/jira/browse/BEAM-301 <https://issues.apache.org/jira/browse/BEAM-301>. Does that match what you were thinking? If so, let’s continue the conversation there.
Julian
> On Apr 5, 2017, at 3:13 PM, Khai Tran <kh...@linkedin.com.INVALID> wrote:
>
> Hi,
> Just want to check if there is any ongoing effort to convert Calcite
> logical plan into Apache Beam APIs for stream processing?
>
> Thanks,
> Khai