You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Mingmin Xu <mi...@gmail.com> on 2017/07/05 22:33:55 UTC
Re: BeamSQL status and merge to master
Thanks for everybody's effort, we're very close to finish existing tasks.
Here's an status update of SQL DSL, feel free to have a try and share any
comment:
*1. what's done*
DSL feature is done, with basic filter/project/aggregation/union/join,
built-in functions/UDF/UDAF(pending on #3491)
*2. what's on-going*
more unit tests, and documentation of README/Beam web.
*3. open questions*
BEAM-2441 <https://issues.apache.org/jira/browse/BEAM-2441> want to see
any suggestion on the proper module name for SQL work. As mentioned in
task, '*dsl/sql* is for the Java SDK and also prevents alternative language
implementations, however there's another SQL client and not good to be
included as Java SDK extention'.
---
*How to run the example* beam/dsls/sql/example/BeamSqlExample.java
<https://github.com/apache/beam/blob/DSL_SQL/dsls/sql/src/main/java/org/apache/beam/dsls/sql/example/BeamSqlExample.java>
1. run 'mvn install' to avoid the error in #3439
<https://github.com/apache/beam/pull/3439>
2. run 'mvn -pl dsls/sql compile exec:java
-Dexec.mainClass=org.apache.beam.dsls.sql.example.BeamSqlExample
-Dexec.args="--runner=DirectRunner" -Pdirect-runner'
FYI:
1. burn-down list in google doc
https://docs.google.com/document/d/1EHZgSu4Jd75iplYpYT_K_JwSZxL2DWG8kv_EmQzNXFc/edit?usp=sharing
2. JIRA tasks with label 'dsl_sql_merge'
https://issues.apache.org/jira/browse/BEAM-2555?jql=labels%20%3D%20dsl_sql_merge
Mingmin
On Tue, Jun 13, 2017 at 8:51 AM, Lukasz Cwik <lc...@google.com.invalid>
wrote:
> Nevermind, I merged it into #2 about usability.
>
> On Tue, Jun 13, 2017 at 8:50 AM, Lukasz Cwik <lc...@google.com> wrote:
>
> > I added a section about maven module structure/packaging (#6).
> >
> > On Tue, Jun 13, 2017 at 8:30 AM, Tyler Akidau <takidau@google.com.invalid
> >
> > wrote:
> >
> >> Thanks Mingmin. I've copied your list into a doc[1] to make it easier to
> >> collaborate on comments and edits.
> >>
> >> [1] https://s.apache.org/beam-dsl-sql-burndown
> >>
> >> -Tyler
> >>
> >>
> >> On Mon, Jun 12, 2017 at 10:09 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> >> wrote:
> >>
> >> > Hi Mingmin
> >> >
> >> > Sorry, the meeting was in the middle of the night for me and I wasn't
> >> able
> >> > to
> >> > make it.
> >> >
> >> > The timing and checklist look good to me.
> >> >
> >> > We plan to do a Beam release end of June, so, merging in July means we
> >> can
> >> > include it in the next release.
> >> >
> >> > Thanks !
> >> > Regards
> >> > JB
> >> >
> >> > On 06/13/2017 03:06 AM, Mingmin Xu wrote:
> >> > > Hi all,
> >> > >
> >> > > Thanks to join the meeting. As discussed, we're planning to merge
> >> DSL_SQL
> >> > > branch back to master, targeted in the middle of July. A tag
> >> > > 'dsl_sql_merge'[1] is created to track all todo tasks.
> >> > >
> >> > > *What's added in Beam SQL?*
> >> > > BeamSQL provides the capability to execute SQL queries with Beam
> Java
> >> > SDK,
> >> > > either by translating SQL to a PTransform, or run with a standalone
> >> CLI
> >> > > client.
> >> > >
> >> > > *Checklist for merge:*
> >> > > 1. functionality
> >> > > 1.1. SQL grammer:
> >> > > 1.1.1. basic query with SELECT/FILTER/PROJECT;
> >> > > 1.1.2. AGGREGATION with global window;
> >> > > 1.1.3. AGGREGATION with FIX_TIME/SLIDING_TIME/SESSION window;
> >> > > 1.1.4. JOIN
> >> > > 1.2. UDF/UDAF support;
> >> > > 1.3. support predefined String/Math/Date functions, see[2];
> >> > >
> >> > > 2. DSL interface to convert SQL as PTransform;
> >> > >
> >> > > 3. junit test;
> >> > >
> >> > > 4. Java document;
> >> > >
> >> > > 5. Document of SQL feature in website;
> >> > >
> >> > > Any comments/suggestions are very welcomed.
> >> > >
> >> > > Note:
> >> > > [1].
> >> > >
> >> > https://issues.apache.org/jira/browse/BEAM-2436?jql=labels%
> >> 20%3D%20dsl_sql_merge
> >> > >
> >> > > [2]. https://calcite.apache.org/docs/reference.html
> >> > >
> >> >
> >> > --
> >> > Jean-Baptiste Onofré
> >> > jbonofre@apache.org
> >> > http://blog.nanthrax.net
> >> > Talend - http://www.talend.com
> >> >
> >>
> >
> >
>
--
----
Mingmin
Re: BeamSQL status and merge to master
Posted by Tyler Akidau <ta...@google.com.INVALID>.
There are still open items in the merge to master doc. We're close to being
ready, but let's please address those first.
-Tyler
On Thu, Jul 20, 2017 at 9:34 PM Mingmin Xu <mi...@gmail.com> wrote:
> Quick update:
>
> The final PR[1] is open to review now. Please leave your comment or create
> a sub-task in [2] for any question.
>
> [1]. https://github.com/apache/beam/pull/3606
> [2]. https://issues.apache.org/jira/browse/BEAM-2651
>
>
> On Wed, Jul 5, 2017 at 3:34 PM, Jesse Anderson <je...@bigdatainstitute.io>
> wrote:
>
> > So excited to start using this!
> >
> > On Wed, Jul 5, 2017, 3:34 PM Mingmin Xu <mi...@gmail.com> wrote:
> >
> > > Thanks for everybody's effort, we're very close to finish existing
> tasks.
> > > Here's an status update of SQL DSL, feel free to have a try and share
> any
> > > comment:
> > >
> > > *1. what's done*
> > > DSL feature is done, with basic
> filter/project/aggregation/union/join,
> > > built-in functions/UDF/UDAF(pending on #3491)
> > >
> > > *2. what's on-going*
> > > more unit tests, and documentation of README/Beam web.
> > >
> > > *3. open questions*
> > > BEAM-2441 <https://issues.apache.org/jira/browse/BEAM-2441> want to
> > see
> > > any suggestion on the proper module name for SQL work. As mentioned in
> > > task, '*dsl/sql* is for the Java SDK and also prevents alternative
> > language
> > > implementations, however there's another SQL client and not good to be
> > > included as Java SDK extention'.
> > >
> > > ---
> > > *How to run the example* beam/dsls/sql/example/BeamSqlExample.java
> > > <
> > > https://github.com/apache/beam/blob/DSL_SQL/dsls/sql/src/mai
> > n/java/org/apache/beam/dsls/sql/example/BeamSqlExample.java
> > > >
> > > 1. run 'mvn install' to avoid the error in #3439
> > > <https://github.com/apache/beam/pull/3439>
> > > 2. run 'mvn -pl dsls/sql compile exec:java
> > > -Dexec.mainClass=org.apache.beam.dsls.sql.example.BeamSqlExample
> > > -Dexec.args="--runner=DirectRunner" -Pdirect-runner'
> > >
> > > FYI:
> > > 1. burn-down list in google doc
> > >
> > > https://docs.google.com/document/d/1EHZgSu4Jd75iplYpYT_K_JwS
> > ZxL2DWG8kv_EmQzNXFc/edit?usp=sharing
> > > 2. JIRA tasks with label 'dsl_sql_merge'
> > >
> > > https://issues.apache.org/jira/browse/BEAM-2555?jql=labels%2
> > 0%3D%20dsl_sql_merge
> > >
> > >
> > > Mingmin
> > >
> > > On Tue, Jun 13, 2017 at 8:51 AM, Lukasz Cwik <lcwik@google.com.invalid
> >
> > > wrote:
> > >
> > > > Nevermind, I merged it into #2 about usability.
> > > >
> > > > On Tue, Jun 13, 2017 at 8:50 AM, Lukasz Cwik <lc...@google.com>
> wrote:
> > > >
> > > > > I added a section about maven module structure/packaging (#6).
> > > > >
> > > > > On Tue, Jun 13, 2017 at 8:30 AM, Tyler Akidau
> > > <takidau@google.com.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Thanks Mingmin. I've copied your list into a doc[1] to make it
> > easier
> > > to
> > > > >> collaborate on comments and edits.
> > > > >>
> > > > >> [1] https://s.apache.org/beam-dsl-sql-burndown
> > > > >>
> > > > >> -Tyler
> > > > >>
> > > > >>
> > > > >> On Mon, Jun 12, 2017 at 10:09 PM Jean-Baptiste Onofré <
> > > jb@nanthrax.net>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Mingmin
> > > > >> >
> > > > >> > Sorry, the meeting was in the middle of the night for me and I
> > > wasn't
> > > > >> able
> > > > >> > to
> > > > >> > make it.
> > > > >> >
> > > > >> > The timing and checklist look good to me.
> > > > >> >
> > > > >> > We plan to do a Beam release end of June, so, merging in July
> > means
> > > we
> > > > >> can
> > > > >> > include it in the next release.
> > > > >> >
> > > > >> > Thanks !
> > > > >> > Regards
> > > > >> > JB
> > > > >> >
> > > > >> > On 06/13/2017 03:06 AM, Mingmin Xu wrote:
> > > > >> > > Hi all,
> > > > >> > >
> > > > >> > > Thanks to join the meeting. As discussed, we're planning to
> > merge
> > > > >> DSL_SQL
> > > > >> > > branch back to master, targeted in the middle of July. A tag
> > > > >> > > 'dsl_sql_merge'[1] is created to track all todo tasks.
> > > > >> > >
> > > > >> > > *What's added in Beam SQL?*
> > > > >> > > BeamSQL provides the capability to execute SQL queries with
> Beam
> > > > Java
> > > > >> > SDK,
> > > > >> > > either by translating SQL to a PTransform, or run with a
> > > standalone
> > > > >> CLI
> > > > >> > > client.
> > > > >> > >
> > > > >> > > *Checklist for merge:*
> > > > >> > > 1. functionality
> > > > >> > > 1.1. SQL grammer:
> > > > >> > > 1.1.1. basic query with SELECT/FILTER/PROJECT;
> > > > >> > > 1.1.2. AGGREGATION with global window;
> > > > >> > > 1.1.3. AGGREGATION with FIX_TIME/SLIDING_TIME/SESSION
> > window;
> > > > >> > > 1.1.4. JOIN
> > > > >> > > 1.2. UDF/UDAF support;
> > > > >> > > 1.3. support predefined String/Math/Date functions, see[2];
> > > > >> > >
> > > > >> > > 2. DSL interface to convert SQL as PTransform;
> > > > >> > >
> > > > >> > > 3. junit test;
> > > > >> > >
> > > > >> > > 4. Java document;
> > > > >> > >
> > > > >> > > 5. Document of SQL feature in website;
> > > > >> > >
> > > > >> > > Any comments/suggestions are very welcomed.
> > > > >> > >
> > > > >> > > Note:
> > > > >> > > [1].
> > > > >> > >
> > > > >> > https://issues.apache.org/jira/browse/BEAM-2436?jql=labels%
> > > > >> 20%3D%20dsl_sql_merge
> > > > >> > >
> > > > >> > > [2]. https://calcite.apache.org/docs/reference.html
> > > > >> > >
> > > > >> >
> > > > >> > --
> > > > >> > Jean-Baptiste Onofré
> > > > >> > jbonofre@apache.org
> > > > >> > http://blog.nanthrax.net
> > > > >> > Talend - http://www.talend.com
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > ----
> > > Mingmin
> > >
> > --
> > Thanks,
> >
> > Jesse
> >
>
>
>
> --
> ----
> Mingmin
>
Re: BeamSQL status and merge to master
Posted by Mingmin Xu <mi...@gmail.com>.
Quick update:
The final PR[1] is open to review now. Please leave your comment or create
a sub-task in [2] for any question.
[1]. https://github.com/apache/beam/pull/3606
[2]. https://issues.apache.org/jira/browse/BEAM-2651
On Wed, Jul 5, 2017 at 3:34 PM, Jesse Anderson <je...@bigdatainstitute.io>
wrote:
> So excited to start using this!
>
> On Wed, Jul 5, 2017, 3:34 PM Mingmin Xu <mi...@gmail.com> wrote:
>
> > Thanks for everybody's effort, we're very close to finish existing tasks.
> > Here's an status update of SQL DSL, feel free to have a try and share any
> > comment:
> >
> > *1. what's done*
> > DSL feature is done, with basic filter/project/aggregation/union/join,
> > built-in functions/UDF/UDAF(pending on #3491)
> >
> > *2. what's on-going*
> > more unit tests, and documentation of README/Beam web.
> >
> > *3. open questions*
> > BEAM-2441 <https://issues.apache.org/jira/browse/BEAM-2441> want to
> see
> > any suggestion on the proper module name for SQL work. As mentioned in
> > task, '*dsl/sql* is for the Java SDK and also prevents alternative
> language
> > implementations, however there's another SQL client and not good to be
> > included as Java SDK extention'.
> >
> > ---
> > *How to run the example* beam/dsls/sql/example/BeamSqlExample.java
> > <
> > https://github.com/apache/beam/blob/DSL_SQL/dsls/sql/src/mai
> n/java/org/apache/beam/dsls/sql/example/BeamSqlExample.java
> > >
> > 1. run 'mvn install' to avoid the error in #3439
> > <https://github.com/apache/beam/pull/3439>
> > 2. run 'mvn -pl dsls/sql compile exec:java
> > -Dexec.mainClass=org.apache.beam.dsls.sql.example.BeamSqlExample
> > -Dexec.args="--runner=DirectRunner" -Pdirect-runner'
> >
> > FYI:
> > 1. burn-down list in google doc
> >
> > https://docs.google.com/document/d/1EHZgSu4Jd75iplYpYT_K_JwS
> ZxL2DWG8kv_EmQzNXFc/edit?usp=sharing
> > 2. JIRA tasks with label 'dsl_sql_merge'
> >
> > https://issues.apache.org/jira/browse/BEAM-2555?jql=labels%2
> 0%3D%20dsl_sql_merge
> >
> >
> > Mingmin
> >
> > On Tue, Jun 13, 2017 at 8:51 AM, Lukasz Cwik <lc...@google.com.invalid>
> > wrote:
> >
> > > Nevermind, I merged it into #2 about usability.
> > >
> > > On Tue, Jun 13, 2017 at 8:50 AM, Lukasz Cwik <lc...@google.com> wrote:
> > >
> > > > I added a section about maven module structure/packaging (#6).
> > > >
> > > > On Tue, Jun 13, 2017 at 8:30 AM, Tyler Akidau
> > <takidau@google.com.invalid
> > > >
> > > > wrote:
> > > >
> > > >> Thanks Mingmin. I've copied your list into a doc[1] to make it
> easier
> > to
> > > >> collaborate on comments and edits.
> > > >>
> > > >> [1] https://s.apache.org/beam-dsl-sql-burndown
> > > >>
> > > >> -Tyler
> > > >>
> > > >>
> > > >> On Mon, Jun 12, 2017 at 10:09 PM Jean-Baptiste Onofré <
> > jb@nanthrax.net>
> > > >> wrote:
> > > >>
> > > >> > Hi Mingmin
> > > >> >
> > > >> > Sorry, the meeting was in the middle of the night for me and I
> > wasn't
> > > >> able
> > > >> > to
> > > >> > make it.
> > > >> >
> > > >> > The timing and checklist look good to me.
> > > >> >
> > > >> > We plan to do a Beam release end of June, so, merging in July
> means
> > we
> > > >> can
> > > >> > include it in the next release.
> > > >> >
> > > >> > Thanks !
> > > >> > Regards
> > > >> > JB
> > > >> >
> > > >> > On 06/13/2017 03:06 AM, Mingmin Xu wrote:
> > > >> > > Hi all,
> > > >> > >
> > > >> > > Thanks to join the meeting. As discussed, we're planning to
> merge
> > > >> DSL_SQL
> > > >> > > branch back to master, targeted in the middle of July. A tag
> > > >> > > 'dsl_sql_merge'[1] is created to track all todo tasks.
> > > >> > >
> > > >> > > *What's added in Beam SQL?*
> > > >> > > BeamSQL provides the capability to execute SQL queries with Beam
> > > Java
> > > >> > SDK,
> > > >> > > either by translating SQL to a PTransform, or run with a
> > standalone
> > > >> CLI
> > > >> > > client.
> > > >> > >
> > > >> > > *Checklist for merge:*
> > > >> > > 1. functionality
> > > >> > > 1.1. SQL grammer:
> > > >> > > 1.1.1. basic query with SELECT/FILTER/PROJECT;
> > > >> > > 1.1.2. AGGREGATION with global window;
> > > >> > > 1.1.3. AGGREGATION with FIX_TIME/SLIDING_TIME/SESSION
> window;
> > > >> > > 1.1.4. JOIN
> > > >> > > 1.2. UDF/UDAF support;
> > > >> > > 1.3. support predefined String/Math/Date functions, see[2];
> > > >> > >
> > > >> > > 2. DSL interface to convert SQL as PTransform;
> > > >> > >
> > > >> > > 3. junit test;
> > > >> > >
> > > >> > > 4. Java document;
> > > >> > >
> > > >> > > 5. Document of SQL feature in website;
> > > >> > >
> > > >> > > Any comments/suggestions are very welcomed.
> > > >> > >
> > > >> > > Note:
> > > >> > > [1].
> > > >> > >
> > > >> > https://issues.apache.org/jira/browse/BEAM-2436?jql=labels%
> > > >> 20%3D%20dsl_sql_merge
> > > >> > >
> > > >> > > [2]. https://calcite.apache.org/docs/reference.html
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > Jean-Baptiste Onofré
> > > >> > jbonofre@apache.org
> > > >> > http://blog.nanthrax.net
> > > >> > Talend - http://www.talend.com
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > ----
> > Mingmin
> >
> --
> Thanks,
>
> Jesse
>
--
----
Mingmin
Re: BeamSQL status and merge to master
Posted by Jesse Anderson <je...@bigdatainstitute.io>.
So excited to start using this!
On Wed, Jul 5, 2017, 3:34 PM Mingmin Xu <mi...@gmail.com> wrote:
> Thanks for everybody's effort, we're very close to finish existing tasks.
> Here's an status update of SQL DSL, feel free to have a try and share any
> comment:
>
> *1. what's done*
> DSL feature is done, with basic filter/project/aggregation/union/join,
> built-in functions/UDF/UDAF(pending on #3491)
>
> *2. what's on-going*
> more unit tests, and documentation of README/Beam web.
>
> *3. open questions*
> BEAM-2441 <https://issues.apache.org/jira/browse/BEAM-2441> want to see
> any suggestion on the proper module name for SQL work. As mentioned in
> task, '*dsl/sql* is for the Java SDK and also prevents alternative language
> implementations, however there's another SQL client and not good to be
> included as Java SDK extention'.
>
> ---
> *How to run the example* beam/dsls/sql/example/BeamSqlExample.java
> <
> https://github.com/apache/beam/blob/DSL_SQL/dsls/sql/src/main/java/org/apache/beam/dsls/sql/example/BeamSqlExample.java
> >
> 1. run 'mvn install' to avoid the error in #3439
> <https://github.com/apache/beam/pull/3439>
> 2. run 'mvn -pl dsls/sql compile exec:java
> -Dexec.mainClass=org.apache.beam.dsls.sql.example.BeamSqlExample
> -Dexec.args="--runner=DirectRunner" -Pdirect-runner'
>
> FYI:
> 1. burn-down list in google doc
>
> https://docs.google.com/document/d/1EHZgSu4Jd75iplYpYT_K_JwSZxL2DWG8kv_EmQzNXFc/edit?usp=sharing
> 2. JIRA tasks with label 'dsl_sql_merge'
>
> https://issues.apache.org/jira/browse/BEAM-2555?jql=labels%20%3D%20dsl_sql_merge
>
>
> Mingmin
>
> On Tue, Jun 13, 2017 at 8:51 AM, Lukasz Cwik <lc...@google.com.invalid>
> wrote:
>
> > Nevermind, I merged it into #2 about usability.
> >
> > On Tue, Jun 13, 2017 at 8:50 AM, Lukasz Cwik <lc...@google.com> wrote:
> >
> > > I added a section about maven module structure/packaging (#6).
> > >
> > > On Tue, Jun 13, 2017 at 8:30 AM, Tyler Akidau
> <takidau@google.com.invalid
> > >
> > > wrote:
> > >
> > >> Thanks Mingmin. I've copied your list into a doc[1] to make it easier
> to
> > >> collaborate on comments and edits.
> > >>
> > >> [1] https://s.apache.org/beam-dsl-sql-burndown
> > >>
> > >> -Tyler
> > >>
> > >>
> > >> On Mon, Jun 12, 2017 at 10:09 PM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > >> wrote:
> > >>
> > >> > Hi Mingmin
> > >> >
> > >> > Sorry, the meeting was in the middle of the night for me and I
> wasn't
> > >> able
> > >> > to
> > >> > make it.
> > >> >
> > >> > The timing and checklist look good to me.
> > >> >
> > >> > We plan to do a Beam release end of June, so, merging in July means
> we
> > >> can
> > >> > include it in the next release.
> > >> >
> > >> > Thanks !
> > >> > Regards
> > >> > JB
> > >> >
> > >> > On 06/13/2017 03:06 AM, Mingmin Xu wrote:
> > >> > > Hi all,
> > >> > >
> > >> > > Thanks to join the meeting. As discussed, we're planning to merge
> > >> DSL_SQL
> > >> > > branch back to master, targeted in the middle of July. A tag
> > >> > > 'dsl_sql_merge'[1] is created to track all todo tasks.
> > >> > >
> > >> > > *What's added in Beam SQL?*
> > >> > > BeamSQL provides the capability to execute SQL queries with Beam
> > Java
> > >> > SDK,
> > >> > > either by translating SQL to a PTransform, or run with a
> standalone
> > >> CLI
> > >> > > client.
> > >> > >
> > >> > > *Checklist for merge:*
> > >> > > 1. functionality
> > >> > > 1.1. SQL grammer:
> > >> > > 1.1.1. basic query with SELECT/FILTER/PROJECT;
> > >> > > 1.1.2. AGGREGATION with global window;
> > >> > > 1.1.3. AGGREGATION with FIX_TIME/SLIDING_TIME/SESSION window;
> > >> > > 1.1.4. JOIN
> > >> > > 1.2. UDF/UDAF support;
> > >> > > 1.3. support predefined String/Math/Date functions, see[2];
> > >> > >
> > >> > > 2. DSL interface to convert SQL as PTransform;
> > >> > >
> > >> > > 3. junit test;
> > >> > >
> > >> > > 4. Java document;
> > >> > >
> > >> > > 5. Document of SQL feature in website;
> > >> > >
> > >> > > Any comments/suggestions are very welcomed.
> > >> > >
> > >> > > Note:
> > >> > > [1].
> > >> > >
> > >> > https://issues.apache.org/jira/browse/BEAM-2436?jql=labels%
> > >> 20%3D%20dsl_sql_merge
> > >> > >
> > >> > > [2]. https://calcite.apache.org/docs/reference.html
> > >> > >
> > >> >
> > >> > --
> > >> > Jean-Baptiste Onofré
> > >> > jbonofre@apache.org
> > >> > http://blog.nanthrax.net
> > >> > Talend - http://www.talend.com
> > >> >
> > >>
> > >
> > >
> >
>
>
>
> --
> ----
> Mingmin
>
--
Thanks,
Jesse