You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Reuven Lax <re...@google.com> on 2017/12/02 22:40:57 UTC
Apache Beam, version 2.2.0
The Apache Beam community is pleased to announce the availability of the
2.2.0 release.
This release adds support for generic file sources and sinks (beyond TextIO
and AvroIO) using FileIO, including support for dynamic filenames using
readAll; this allows streaming pipelines to now read from files by
continuously monitoring a directory for new filw. Many other IOs are
improved, notably including exactly-once support for the Kafka sink.
Initial support for BEAM-SQL is also included in this release. For a
more-complete list of major changes in the release, please refer to the
release notes [2].
The 2.2.0 release is now the recommended version; we encourage everyone to
upgrade from any earlier releases.
We’d like to invite everyone to try out Apache Beam today and consider
joining our vibrant community. We welcome feedback, contribution and
participation through our mailing lists, issue tracker, pull requests, and
events.
- Reuven Lax, on behalf of the Apache Beam community.
[1] https://beam.apache.org/get-started/downloads/
[2]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
Re: Apache Beam, version 2.2.0
Posted by Vilhelm von Ehrenheim <vo...@gmail.com>.
I'm super excited about this release! Great work everyone involved!
On Mon, Dec 4, 2017 at 10:58 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Just an important note that we forgot to mention.
>
> !! The 2.2.0 release will be the last one supporting Spark 1.x and Java 7
> !!
>
> Starting from Beam 2.3.0, the Spark runner will work only with Spark 2.x
> and we will focus only Java 8.
>
> Regards
> JB
>
>
> On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
>
>> Thanks Reuven !
>>
>> I would like to emphasize on some highlights in 2.2.0 release:
>>
>> - New IOs have been introduced:
>> * TikaIO leveraging Apache Tika, allowing the deal with a lot of
>> different data formats
>> * RedisIO to read and write key/value pairs from a Redis server. This
>> IO will be soon extended to Redis PubSub.
>> * FileIO provides transforms for working with files (raw). Especially,
>> it provides matching file patterns and read on patterns. It can be easily
>> extended for a specific format (like we do in AvroIO or TextIO now).
>> * SolrIO to interact with Apache Solr (Lucene)
>>
>> - On the other hand, improvements have been performed on existing IOs:
>> * We started to introduce readAll pattern in IOs (AvroIO, TextIO,
>> JdbcIO, ...), allowing to pass "request" arguments via an input PCollection.
>> * ElasticsearchIO has an improved support of different Elasticsearch
>> version (including Elasticsearch 5.x). It also now supports SSL/TLS.
>> * HBaseIO is now able to do dynamic work rebalancing
>> * KinesisIO uses a more accurate watermark (based on
>> approximateArrivalTimestamp)
>> * TextIO now supports custom delimiter and like AvroIO, supports the
>> readAll pattern,
>> * Performance improvements on JdbcIO when it has to read lot of rows
>> * Kafka write supports Exactly-Once pattern (introduce in Kafka 0.11.x)
>>
>> - A new DSL has been introduced: the SQL DSL !
>>
>> We are now focus on 2.3.0 release with new improvements and features !
>>
>> Stay tuned !
>>
>> JB on behalf of the Apache Beam community.
>>
>> On 12/02/2017 11:40 PM, Reuven Lax wrote:
>>
>>> The Apache Beam community is pleased to announce the availability of the
>>> 2.2.0 release.
>>>
>>> This release adds support for generic file sources and sinks (beyond
>>> TextIO and AvroIO) using FileIO, including support for dynamic filenames
>>> using readAll; this allows streaming pipelines to now read from files by
>>> continuously monitoring a directory for new filw. Many other IOs are
>>> improved, notably including exactly-once support for the Kafka sink.
>>> Initial support for BEAM-SQL is also included in this release. For a
>>> more-complete list of major changes in the release, please refer to the
>>> release notes [2].
>>>
>>> The 2.2.0 release is now the recommended version; we encourage everyone
>>> to
>>> upgrade from any earlier releases.
>>>
>>> We’d like to invite everyone to try out Apache Beam today and consider
>>> joining our vibrant community. We welcome feedback, contribution and
>>> participation through our mailing lists, issue tracker, pull requests,
>>> and
>>> events.
>>>
>>> - Reuven Lax, on behalf of the Apache Beam community.
>>>
>>> [1] https://beam.apache.org/get-started/downloads/
>>> [2]
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>>> ctId=12319527&version=12341044
>>>
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Apache Beam, version 2.2.0
Posted by Lukasz Cwik <lc...@google.com>.
I also believe we were still in the investigatory phase for dropping
support for Java 7.
On Mon, Dec 4, 2017 at 2:22 PM, Eugene Kirpichov <ki...@google.com>
wrote:
> Thanks JB for sending the detailed notes about new stuff in 2.2.0! A lot
> of exciting things indeed.
>
> Regarding Java 8: I thought our consensus was to have the release notes
> say that we're *considering* going Java8-only, and use that to get more
> opinions from the user community - but I can't find the emails that made me
> think so.
>
> +Ismaël Mejía <ie...@gmail.com> - do you think we should formally
> conclude the vote on the thread [VOTE] [DISCUSSION] Remove support for Java
> 7?
> Or should we take more steps - e.g. perhaps tweet a link to that thread
> from the Beam twitter account, ask people to chime in, and wait for say 2
> weeks before declaring a conclusion?
>
> Let's also have a process JIRA for going Java8. I've filed one:
> https://issues.apache.org/jira/browse/BEAM-3285
>
> On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Just an important note that we forgot to mention.
>>
>> !! The 2.2.0 release will be the last one supporting Spark 1.x and Java 7
>> !!
>>
>> Starting from Beam 2.3.0, the Spark runner will work only with Spark 2.x
>> and we
>> will focus only Java 8.
>>
>> Regards
>> JB
>>
>> On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
>> > Thanks Reuven !
>> >
>> > I would like to emphasize on some highlights in 2.2.0 release:
>> >
>> > - New IOs have been introduced:
>> > * TikaIO leveraging Apache Tika, allowing the deal with a lot of
>> different
>> > data formats
>> > * RedisIO to read and write key/value pairs from a Redis server. This
>> IO will
>> > be soon extended to Redis PubSub.
>> > * FileIO provides transforms for working with files (raw).
>> Especially, it
>> > provides matching file patterns and read on patterns. It can be easily
>> extended
>> > for a specific format (like we do in AvroIO or TextIO now).
>> > * SolrIO to interact with Apache Solr (Lucene)
>> >
>> > - On the other hand, improvements have been performed on existing IOs:
>> > * We started to introduce readAll pattern in IOs (AvroIO, TextIO,
>> JdbcIO,
>> > ...), allowing to pass "request" arguments via an input PCollection.
>> > * ElasticsearchIO has an improved support of different Elasticsearch
>> version
>> > (including Elasticsearch 5.x). It also now supports SSL/TLS.
>> > * HBaseIO is now able to do dynamic work rebalancing
>> > * KinesisIO uses a more accurate watermark (based on
>> approximateArrivalTimestamp)
>> > * TextIO now supports custom delimiter and like AvroIO, supports the
>> readAll
>> > pattern,
>> > * Performance improvements on JdbcIO when it has to read lot of rows
>> > * Kafka write supports Exactly-Once pattern (introduce in Kafka
>> 0.11.x)
>> >
>> > - A new DSL has been introduced: the SQL DSL !
>> >
>> > We are now focus on 2.3.0 release with new improvements and features !
>> >
>> > Stay tuned !
>> >
>> > JB on behalf of the Apache Beam community.
>> >
>> > On 12/02/2017 11:40 PM, Reuven Lax wrote:
>> >> The Apache Beam community is pleased to announce the availability of
>> the
>> >> 2.2.0 release.
>> >>
>> >> This release adds support for generic file sources and sinks (beyond
>> TextIO
>> >> and AvroIO) using FileIO, including support for dynamic filenames using
>> >> readAll; this allows streaming pipelines to now read from files by
>> >> continuously monitoring a directory for new filw. Many other IOs are
>> improved,
>> >> notably including exactly-once support for the Kafka sink. Initial
>> support for
>> >> BEAM-SQL is also included in this release. For a more-complete list of
>> major
>> >> changes in the release, please refer to the release notes [2].
>> >>
>> >> The 2.2.0 release is now the recommended version; we encourage
>> everyone to
>> >> upgrade from any earlier releases.
>> >>
>> >> We’d like to invite everyone to try out Apache Beam today and consider
>> >> joining our vibrant community. We welcome feedback, contribution and
>> >> participation through our mailing lists, issue tracker, pull requests,
>> and
>> >> events.
>> >>
>> >> - Reuven Lax, on behalf of the Apache Beam community.
>> >>
>> >> [1] https://beam.apache.org/get-started/downloads/
>> >> [2]
>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12319527&version=12341044
>> >>
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
Re: Apache Beam, version 2.2.0
Posted by Ahmet Altay <al...@google.com>.
Thank you Reuven! I tweeted the release announcement on Beam's account.
On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <re...@google.com> wrote:
> Technically it's a backwards-incompatible change, however if we are
> convinced the risk is low we could do it.
>
> As mentioned on the original thread, it's not clear that all Beam users
> read user@ - e.g. most Dataflow users definitely do not. I think we need
> to separately reach out to users of each runner through runner-specific
> channels.
>
> Reuven
>
> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov <ki...@google.com>
> wrote:
>
>> On the original thread https://lists.apache.or
>> g/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e3919790
>> 87f725d2e0@%3Cuser.beam.apache.org%3E , Robert and Ismaël were in favor
>> of no major version change [Ismaël said:* Also I am afraid that if we
>> wait*
>> *until we have enough changes to switch Beam to a new major version the
>> switch to Java 8 will happen too late, probably after Java 8's end of life.
>> And I am not exaggerating, Java 8 is planned to EOL next march 2018!*];
>> JB and now Reuven are in favor of a major version change; nobody so far
>> argued against switching to Java8 in general.
>>
>> I'm personally in favor of no major version change (i.e. not waiting
>> until all other large changes for Beam 3.0 converge, which will likely be
>> many months), because:
>> - Reasons Ismaël cited; plus the reason that most people are likely
>> already using Java 8.
>> - Going Java8-only earlier will make other Beam 3.0 APIs better for Java8
>> users, because we (Beam contributors) will have experience working with
>> them within the SDK in Java8 (e.g. writing tests with use of lambdas and
>> noticing whether it's clunky, or whether some other Beam APIs need better
>> Java8 support).
>> - Going Java8 will make it more reasonable to include (mostly or only)
>> Java8 snippets in Beam documentation, which will obviously look more
>> concise and attractive, addressing one of the common concerns of Beam users
>> that it has a heavyweight API compared to functional-style APIs of Spark
>> etc.
>>
>> I think resolving this via a poll of users would be reasonable. I'd
>> suggest e.g. the following phrasing:
>>
>> Apache Beam is considering dropping support for Java 7, and supporting
>> only Java 8 and above in a subsequent release. How would it impact your
>> usage of Beam?
>> - I am already using only Java 8+ for building my Beam code
>> - I am using Java 7 for building my Beam code, but I would have no
>> trouble switching to Java 8
>> - I am using Java 7 for building my Beam code, and dropping Java 7 would
>> be a blocker or hindrance to adopting the new release for me
>>
>> We could tweet this poll on Apache Beam twitter and publish on user@,
>> and, say, if we receive 5% or fewer votes for option 3 after keeping it
>> open for 2 weeks, then adopt Java 8 without a major version change.
>>
>> WDYT?
>>
>> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Good idea ! Definitely +1
>>>
>>> Regards
>>> JB
>>>
>>> On 12/05/2017 05:25 AM, Reuven Lax wrote:
>>> > We should bring this up on the Beam 3.0 thread. Since it's technically
>>> a
>>> > backwards-incompatible change, it might make a good item for Beam 3.0.
>>> >
>>> > Reuven
>>> >
>>> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré <jb@nanthrax.net
>>> > <ma...@nanthrax.net>> wrote:
>>> >
>>> > My apologizes, I thought we had a consensus already.
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
>>> >
>>> > Thanks JB for sending the detailed notes about new stuff in
>>> 2.2.0! A lot
>>> > of exciting things indeed.
>>> >
>>> > Regarding Java 8: I thought our consensus was to have the
>>> release notes
>>> > say that we're *considering* going Java8-only, and use that to
>>> get more
>>> > opinions from the user community - but I can't find the emails
>>> that made
>>> > me think so.
>>> >
>>> > +Ismaël Mejía <mailto:iemejia@gmail.com <mailto:
>>> iemejia@gmail.com>> - do
>>> > you think we should formally conclude the vote on the
>>> thread [VOTE]
>>> > [DISCUSSION] Remove support for Java 7?
>>> > Or should we take more steps - e.g. perhaps tweet a link to
>>> that thread
>>> > from the Beam twitter account, ask people to chime in, and
>>> wait for say
>>> > 2 weeks before declaring a conclusion?
>>> >
>>> > Let's also have a process JIRA for going Java8. I've filed one:
>>> > https://issues.apache.org/jira/browse/BEAM-3285
>>> > <https://issues.apache.org/jira/browse/BEAM-3285>
>>> >
>>> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré <
>>> jb@nanthrax.net
>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>> > <ma...@nanthrax.net>>> wrote:
>>> >
>>> > Just an important note that we forgot to mention.
>>> >
>>> > !! The 2.2.0 release will be the last one supporting
>>> Spark 1.x and
>>> > Java 7 !!
>>> >
>>> > Starting from Beam 2.3.0, the Spark runner will work only
>>> with
>>> > Spark 2.x and we
>>> > will focus only Java 8.
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
>>> > > Thanks Reuven !
>>> > >
>>> > > I would like to emphasize on some highlights in 2.2.0
>>> release:
>>> > >
>>> > > - New IOs have been introduced:
>>> > > * TikaIO leveraging Apache Tika, allowing the deal
>>> with a lot
>>> > of different
>>> > > data formats
>>> > > * RedisIO to read and write key/value pairs from a
>>> Redis
>>> > server. This
>>> > IO will
>>> > > be soon extended to Redis PubSub.
>>> > > * FileIO provides transforms for working with files
>>> (raw).
>>> > Especially, it
>>> > > provides matching file patterns and read on patterns.
>>> It can be
>>> > easily
>>> > extended
>>> > > for a specific format (like we do in AvroIO or TextIO
>>> now).
>>> > > * SolrIO to interact with Apache Solr (Lucene)
>>> > >
>>> > > - On the other hand, improvements have been performed
>>> on
>>> > existing IOs:
>>> > > * We started to introduce readAll pattern in IOs
>>> (AvroIO,
>>> > TextIO, JdbcIO,
>>> > > ...), allowing to pass "request" arguments via an
>>> input PCollection.
>>> > > * ElasticsearchIO has an improved support of
>>> different
>>> > Elasticsearch
>>> > version
>>> > > (including Elasticsearch 5.x). It also now supports
>>> SSL/TLS.
>>> > > * HBaseIO is now able to do dynamic work rebalancing
>>> > > * KinesisIO uses a more accurate watermark (based on
>>> > approximateArrivalTimestamp)
>>> > > * TextIO now supports custom delimiter and like
>>> AvroIO,
>>> > supports the
>>> > readAll
>>> > > pattern,
>>> > > * Performance improvements on JdbcIO when it has to
>>> read lot
>>> > of rows
>>> > > * Kafka write supports Exactly-Once pattern
>>> (introduce in
>>> > Kafka 0.11.x)
>>> > >
>>> > > - A new DSL has been introduced: the SQL DSL !
>>> > >
>>> > > We are now focus on 2.3.0 release with new
>>> improvements and
>>> > features !
>>> > >
>>> > > Stay tuned !
>>> > >
>>> > > JB on behalf of the Apache Beam community.
>>> > >
>>> > > On 12/02/2017 11:40 PM, Reuven Lax wrote:
>>> > >> The Apache Beam community is pleased to announce the
>>> > availability of the
>>> > >> 2.2.0 release.
>>> > >>
>>> > >> This release adds support for generic file sources
>>> and sinks
>>> > (beyond TextIO
>>> > >> and AvroIO) using FileIO, including support for
>>> dynamic
>>> > filenames using
>>> > >> readAll; this allows streaming pipelines to now read
>>> from files by
>>> > >> continuously monitoring a directory for new filw.
>>> Many other
>>> > IOs are
>>> > improved,
>>> > >> notably including exactly-once support for the Kafka
>>> sink. Initial
>>> > support for
>>> > >> BEAM-SQL is also included in this release. For a
>>> more-complete
>>> > list of major
>>> > >> changes in the release, please refer to the release
>>> notes [2].
>>> > >>
>>> > >> The 2.2.0 release is now the recommended version; we
>>> encourage
>>> > everyone to
>>> > >> upgrade from any earlier releases.
>>> > >>
>>> > >> We’d like to invite everyone to try out Apache Beam
>>> today and
>>> > consider
>>> > >> joining our vibrant community. We welcome feedback,
>>> > contribution and
>>> > >> participation through our mailing lists, issue
>>> tracker, pull
>>> > requests, and
>>> > >> events.
>>> > >>
>>> > >> - Reuven Lax, on behalf of the Apache Beam community.
>>> > >>
>>> > >> [1] https://beam.apache.org/get-started/downloads/
>>> > <https://beam.apache.org/get-started/downloads/>
>>> > >> [2]
>>> > >>
>>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?proj
>>> ectId=12319527&version=12341044
>>> > <https://issues.apache.org/jira/secure/ReleaseNote.jspa?pro
>>> jectId=12319527&version=12341044>
>>> > >>
>>> > >
>>> >
>>> > --
>>> > Jean-Baptiste Onofré
>>> > jbonofre@apache.org <ma...@apache.org>
>>> > <mailto:jbonofre@apache.org <ma...@apache.org>>
>>> > http://blog.nanthrax.net
>>> > Talend - http://www.talend.com
>>> >
>>> >
>>> > --
>>> > Jean-Baptiste Onofré
>>> > jbonofre@apache.org <ma...@apache.org>
>>> > http://blog.nanthrax.net
>>> > Talend - http://www.talend.com
>>> >
>>> >
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
>
Re: Apache Beam, version 2.2.0
Posted by Steve Niemitz <sn...@apache.org>.
Wah I see it now, no 404 for me either...nothing to see here, carry on :P
On Wed, Dec 13, 2017 at 5:40 PM, Reuven Lax <re...@google.com> wrote:
> Something strange is going on. We can see it in the dropdown list in the
> UI, but if you click on that tag you get a 404.
>
> On Wed, Dec 13, 2017 at 2:37 PM, Reuven Lax <re...@google.com> wrote:
>
>> I created a v2.2 tag. Let me look to see what happened.
>>
>> On Dec 13, 2017 2:31 PM, "Steve Niemitz" <sn...@apache.org> wrote:
>>
>>> Sorry for resurrecting this thread, but I was trying to build 2.2.0 from
>>> source today and noticed there was no v2.2.0 tag (only v2.2.0-RC4). I
>>> assume that's not intentional?
>>>
>>> On Fri, Dec 8, 2017 at 3:35 AM, Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>>> Thanks Eugene for opening the poll (sorry if I didn't before I was
>>>> quite busy in the last two days but expected to do it today).
>>>>
>>>>
>>>> On Fri, Dec 8, 2017 at 1:27 AM, Ahmet Altay <al...@google.com> wrote:
>>>> >
>>>> > On Thu, Dec 7, 2017 at 3:51 PM, Eugene Kirpichov <
>>>> kirpichov@google.com>
>>>> > wrote:
>>>> >>
>>>> >> I've sent the poll
>>>> >> https://lists.apache.org/thread.html/5bc2e184a24de9dbc8184ff
>>>> d2720d1894010497d47d956b395e037df@%3Cuser.beam.apache.org%3E
>>>> >> Will figure out how to tweet from @ApacheBeam, and sent the Twitter
>>>> poll
>>>> >> as well (or ask someone to).
>>>> >
>>>> >
>>>> > I tweeted the poll.
>>>> >
>>>> >>
>>>> >>
>>>> >> On Wed, Dec 6, 2017 at 1:47 PM Lukasz Cwik <lc...@google.com> wrote:
>>>> >>>
>>>> >>> +1 on moving forward with the plan suggested by kirpichov@
>>>> >>>
>>>> >>> On Wed, Dec 6, 2017 at 9:14 AM, Robert Bradshaw <
>>>> robertwb@google.com>
>>>> >>> wrote:
>>>> >>>>
>>>> >>>> +1 to moving forward with this plan.
>>>> >>>>
>>>> >>>> (FWIW, this seems *less* backwards incompatible than, say, moving
>>>> from
>>>> >>>> Spark 1 to Spark 2, which was decided much quicker. I suppose the
>>>> >>>> Spark change has a lower bound on the number of users it could
>>>> impact
>>>> >>>> though.)
>>>> >>>>
>>>> >>>> On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <
>>>> kirpichov@google.com>
>>>> >>>> wrote:
>>>> >>>> > Okay, then let's go forward. Seems that we should:
>>>> >>>> > - Open a new poll on user@, in light of 2.2 having been released
>>>> >>>> > - Open a twitter poll
>>>> >>>> > - Tweet that there's also a poll going on on user@
>>>> >>>> > - Runner authors will reach out to respective runner user
>>>> communities
>>>> >>>> > - 2 weeks later we gather results and decide
>>>> >>>> > ?
>>>> >>>> >
>>>> >>>> > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <ie...@gmail.com>
>>>> wrote:
>>>> >>>> >>
>>>> >>>> >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far
>>>> away,
>>>> >>>> >> and starting to improve Beam to offer a Java 8 friendly
>>>> experience
>>>> >>>> >> seems like an excellent idea.
>>>> >>>> >>
>>>> >>>> >> I understand the backwards compatibility argument. We should do
>>>> the
>>>> >>>> >> poll in twitter + try to reach more users for comments. If you
>>>> >>>> >> consider that it is worth, I can open a second poll at user@.
>>>> >>>> >>
>>>> >>>> >> In any case we should try to move forward, even if we have more
>>>> than
>>>> >>>> >> 5% of users who want to stay on Java 7 we can consider to
>>>> maintain
>>>> >>>> >> minor releases of a backwards compatible version where we can
>>>> >>>> >> backport
>>>> >>>> >> only critical fixes e.g. security/data related errors but
>>>> nothing
>>>> >>>> >> new,
>>>> >>>> >> in case some user really needs to have them. Of course this can
>>>> be
>>>> >>>> >> some extra work (to be discussed).
>>>> >>>> >>
>>>> >>>> >>
>>>> >>>> >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré
>>>> >>>> >> <jb...@nanthrax.net>
>>>> >>>> >> wrote:
>>>> >>>> >> > +1, and sorry again, I thought we got an consensus.
>>>> >>>> >> >
>>>> >>>> >> > Regards
>>>> >>>> >> > JB
>>>> >>>> >> >
>>>> >>>> >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote:
>>>> >>>> >> >>
>>>> >>>> >> >> +1 to the poll and also to Reuven's point.
>>>> >>>> >> >>
>>>> >>>> >> >> Those without a support contract would have been using JDK 7
>>>> >>>> >> >> without
>>>> >>>> >> >> security updates for years. IMO it seems harmful, as a
>>>> netizen, to
>>>> >>>> >> >> encourage
>>>> >>>> >> >> its use/existence.
>>>> >>>> >> >>
>>>> >>>> >> >> If there's no noise from the prior thread, then I would
>>>> assume no
>>>> >>>> >> >> one
>>>> >>>> >> >> on
>>>> >>>> >> >> user@ has any objection. Anyone else with customers should
>>>> reach
>>>> >>>> >> >> out to
>>>> >>>> >> >> them.
>>>> >>>> >> >>
>>>> >>>> >> >> Kenn
>>>> >>>> >> >>
>>>> >>>> >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <relax@google.com
>>>> >>>> >> >> <ma...@google.com>> wrote:
>>>> >>>> >> >>
>>>> >>>> >> >> Technically it's a backwards-incompatible change,
>>>> however if
>>>> >>>> >> >> we are
>>>> >>>> >> >> convinced the risk is low we could do it.
>>>> >>>> >> >>
>>>> >>>> >> >> As mentioned on the original thread, it's not clear that
>>>> all
>>>> >>>> >> >> Beam
>>>> >>>> >> >> users read
>>>> >>>> >> >> user@ - e.g. most Dataflow users definitely do not. I
>>>> think we
>>>> >>>> >> >> need
>>>> >>>> >> >> to
>>>> >>>> >> >> separately reach out to users of each runner through
>>>> >>>> >> >> runner-specific
>>>> >>>> >> >> channels.
>>>> >>>> >> >>
>>>> >>>> >> >> Reuven
>>>> >>>> >> >>
>>>> >>>> >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov
>>>> >>>> >> >> <kirpichov@google.com
>>>> >>>> >> >> <ma...@google.com>> wrote:
>>>> >>>> >> >>
>>>> >>>> >> >> On the original thread
>>>> >>>> >> >>
>>>> >>>> >> >>
>>>> >>>> >> >>
>>>> >>>> >> >> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9
>>>> f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
>>>> >>>> >> >>
>>>> >>>> >> >>
>>>> >>>> >> >>
>>>> >>>> >> >> <https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e
>>>> 9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E>
>>>> >>>> >> >> ,
>>>> >>>> >> >> Robert and Ismaël were in favor of no major version
>>>> change
>>>> >>>> >> >> [Ismaël
>>>> >>>> >> >> said:/Also I am afraid that if we wait/
>>>> >>>> >> >> /until we have enough changes to switch Beam to a new
>>>> >>>> >> >> major
>>>> >>>> >> >> version the
>>>> >>>> >> >> switch to Java 8 will happen too late, probably
>>>> after Java
>>>> >>>> >> >> 8's
>>>> >>>> >> >> end
>>>> >>>> >> >> of
>>>> >>>> >> >> life. And I am not exaggerating, Java 8 is planned
>>>> to EOL
>>>> >>>> >> >> next
>>>> >>>> >> >> march
>>>> >>>> >> >> 2018!/]; JB and now Reuven are in favor of a major
>>>> version
>>>> >>>> >> >> change;
>>>> >>>> >> >>
>>>> >>>> >> >> nobody so far argued against switching to Java8 in
>>>> >>>> >> >> general.
>>>> >>>> >> >>
>>>> >>>> >> >> I'm personally in favor of no major version change
>>>> (i.e.
>>>> >>>> >> >> not
>>>> >>>> >> >> waiting
>>>> >>>> >> >> until all other large changes for Beam 3.0 converge,
>>>> which
>>>> >>>> >> >> will
>>>> >>>> >> >> likely
>>>> >>>> >> >> be many months), because:
>>>> >>>> >> >> - Reasons Ismaël cited; plus the reason that most
>>>> people
>>>> >>>> >> >> are
>>>> >>>> >> >> likely
>>>> >>>> >> >> already using Java 8.
>>>> >>>> >> >> - Going Java8-only earlier will make other Beam 3.0
>>>> APIs
>>>> >>>> >> >> better
>>>> >>>> >> >> for
>>>> >>>> >> >> Java8 users, because we (Beam contributors) will have
>>>> >>>> >> >> experience
>>>> >>>> >> >> working
>>>> >>>> >> >> with them within the SDK in Java8 (e.g. writing
>>>> tests with
>>>> >>>> >> >> use
>>>> >>>> >> >> of
>>>> >>>> >> >> lambdas and noticing whether it's clunky, or whether
>>>> some
>>>> >>>> >> >> other
>>>> >>>> >> >> Beam
>>>> >>>> >> >> APIs need better Java8 support).
>>>> >>>> >> >> - Going Java8 will make it more reasonable to include
>>>> >>>> >> >> (mostly
>>>> >>>> >> >> or
>>>> >>>> >> >> only)
>>>> >>>> >> >> Java8 snippets in Beam documentation, which will
>>>> obviously
>>>> >>>> >> >> look
>>>> >>>> >> >> more
>>>> >>>> >> >> concise and attractive, addressing one of the common
>>>> >>>> >> >> concerns
>>>> >>>> >> >> of
>>>> >>>> >> >> Beam
>>>> >>>> >> >> users that it has a heavyweight API compared to
>>>> >>>> >> >> functional-style
>>>> >>>> >> >> APIs of
>>>> >>>> >> >> Spark etc.
>>>> >>>> >> >>
>>>> >>>> >> >> I think resolving this via a poll of users would be
>>>> >>>> >> >> reasonable.
>>>> >>>> >> >> I'd
>>>> >>>> >> >> suggest e.g. the following phrasing:
>>>> >>>> >> >>
>>>> >>>> >> >> Apache Beam is considering dropping support for Java
>>>> 7,
>>>> >>>> >> >> and
>>>> >>>> >> >> supporting
>>>> >>>> >> >> only Java 8 and above in a subsequent release. How
>>>> would
>>>> >>>> >> >> it
>>>> >>>> >> >> impact
>>>> >>>> >> >> your
>>>> >>>> >> >> usage of Beam?
>>>> >>>> >> >> - I am already using only Java 8+ for building my
>>>> Beam
>>>> >>>> >> >> code
>>>> >>>> >> >> - I am using Java 7 for building my Beam code, but I
>>>> would
>>>> >>>> >> >> have
>>>> >>>> >> >> no
>>>> >>>> >> >> trouble switching to Java 8
>>>> >>>> >> >> - I am using Java 7 for building my Beam code, and
>>>> >>>> >> >> dropping
>>>> >>>> >> >> Java 7
>>>> >>>> >> >> would
>>>> >>>> >> >> be a blocker or hindrance to adopting the new
>>>> release for
>>>> >>>> >> >> me
>>>> >>>> >> >>
>>>> >>>> >> >> We could tweet this poll on Apache Beam twitter and
>>>> >>>> >> >> publish on
>>>> >>>> >> >> user@,
>>>> >>>> >> >> and, say, if we receive 5% or fewer votes for option
>>>> 3
>>>> >>>> >> >> after
>>>> >>>> >> >> keeping it
>>>> >>>> >> >> open for 2 weeks, then adopt Java 8 without a major
>>>> >>>> >> >> version
>>>> >>>> >> >> change.
>>>> >>>> >> >>
>>>> >>>> >> >> WDYT?
>>>> >>>> >> >>
>>>> >>>> >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré
>>>> >>>> >> >> <jb@nanthrax.net
>>>> >>>> >> >> <ma...@nanthrax.net>> wrote:
>>>> >>>> >> >>
>>>> >>>> >> >> Good idea ! Definitely +1
>>>> >>>> >> >>
>>>> >>>> >> >> Regards
>>>> >>>> >> >> JB
>>>> >>>> >> >>
>>>> >>>> >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote:
>>>> >>>> >> >> > We should bring this up on the Beam 3.0
>>>> thread.
>>>> >>>> >> >> Since
>>>> >>>> >> >> it's
>>>> >>>> >> >> technically a
>>>> >>>> >> >> > backwards-incompatible change, it might make
>>>> a good
>>>> >>>> >> >> item
>>>> >>>> >> >> for Beam
>>>> >>>> >> >> 3.0.
>>>> >>>> >> >> >
>>>> >>>> >> >> > Reuven
>>>> >>>> >> >> >
>>>> >>>> >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste
>>>> >>>> >> >> Onofré
>>>> >>>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
>>>> >>>> >> >> > <mailto:jb@nanthrax.net <mailto:
>>>> jb@nanthrax.net>>>
>>>> >>>> >> >> wrote:
>>>> >>>> >> >> >
>>>> >>>> >> >> > My apologizes, I thought we had a
>>>> consensus
>>>> >>>> >> >> already.
>>>> >>>> >> >> >
>>>> >>>> >> >> > Regards
>>>> >>>> >> >> > JB
>>>> >>>> >> >> >
>>>> >>>> >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov
>>>> wrote:
>>>> >>>> >> >> >
>>>> >>>> >> >> > Thanks JB for sending the detailed
>>>> notes
>>>> >>>> >> >> about
>>>> >>>> >> >> new
>>>> >>>> >> >> stuff
>>>> >>>> >> >> in 2.2.0! A lot
>>>> >>>> >> >> > of exciting things indeed.
>>>> >>>> >> >> >
>>>> >>>> >> >> > Regarding Java 8: I thought our
>>>> consensus
>>>> >>>> >> >> was to
>>>> >>>> >> >> have the
>>>> >>>> >> >> release notes
>>>> >>>> >> >> > say that we're *considering* going
>>>> >>>> >> >> Java8-only,
>>>> >>>> >> >> and
>>>> >>>> >> >> use
>>>> >>>> >> >> that to get more
>>>> >>>> >> >> > opinions from the user community -
>>>> but I
>>>> >>>> >> >> can't
>>>> >>>> >> >> find
>>>> >>>> >> >> the
>>>> >>>> >> >> emails that made
>>>> >>>> >> >> > me think so.
>>>> >>>> >> >> >
>>>> >>>> >> >> > +Ismaël Mejía <mailto:
>>>> iemejia@gmail.com
>>>> >>>> >> >> <ma...@gmail.com> <mailto:
>>>> iemejia@gmail.com
>>>> >>>> >> >> <ma...@gmail.com>>> - do
>>>> >>>> >> >> > you think we should formally conclude
>>>> the
>>>> >>>> >> >> vote
>>>> >>>> >> >> on
>>>> >>>> >> >> the
>>>> >>>> >> >> thread [VOTE]
>>>> >>>> >> >> > [DISCUSSION] Remove support for Java
>>>> 7?
>>>> >>>> >> >> > Or should we take more steps - e.g.
>>>> perhaps
>>>> >>>> >> >> tweet a
>>>> >>>> >> >> link
>>>> >>>> >> >> to that thread
>>>> >>>> >> >> > from the Beam twitter account, ask
>>>> people
>>>> >>>> >> >> to
>>>> >>>> >> >> chime
>>>> >>>> >> >> in,
>>>> >>>> >> >> and wait for say
>>>> >>>> >> >> > 2 weeks before declaring a conclusion?
>>>> >>>> >> >> >
>>>> >>>> >> >> > Let's also have a process JIRA for
>>>> going
>>>> >>>> >> >> Java8.
>>>> >>>> >> >> I've
>>>> >>>> >> >> filed one:
>>>> >>>> >> >> > https://issues.apache.org/jira
>>>> /browse/BEAM-3285
>>>> >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285
>>>> >
>>>> >>>> >> >> >
>>>> >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285
>>>> >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285
>>>> >>
>>>> >>>> >> >> >
>>>> >>>> >> >> > On Mon, Dec 4, 2017 at 1:58 AM
>>>> >>>> >> >> Jean-Baptiste
>>>> >>>> >> >> Onofré
>>>> >>>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
>>>> >>>> >> >> > <mailto:jb@nanthrax.net
>>>> >>>> >> >> <ma...@nanthrax.net>>
>>>> >>>> >> >> <mailto:jb@nanthrax.net <ma...@nanthrax.net>
>>>> >>>> >> >> > <mailto:jb@nanthrax.net
>>>> >>>> >> >> <ma...@nanthrax.net>>>>
>>>> >>>> >> >> wrote:
>>>> >>>> >> >> >
>>>> >>>> >> >> > Just an important note that we
>>>> forgot
>>>> >>>> >> >> to
>>>> >>>> >> >> mention.
>>>> >>>> >> >> >
>>>> >>>> >> >> > !! The 2.2.0 release will be the
>>>> last
>>>> >>>> >> >> one
>>>> >>>> >> >> supporting
>>>> >>>> >> >> Spark 1.x and
>>>> >>>> >> >> > Java 7 !!
>>>> >>>> >> >> >
>>>> >>>> >> >> > Starting from Beam 2.3.0, the
>>>> Spark
>>>> >>>> >> >> runner
>>>> >>>> >> >> will work
>>>> >>>> >> >> only with
>>>> >>>> >> >> > Spark 2.x and we
>>>> >>>> >> >> > will focus only Java 8.
>>>> >>>> >> >> >
>>>> >>>> >> >> > Regards
>>>> >>>> >> >> > JB
>>>> >>>> >> >> >
>>>> >>>> >> >> > On 12/04/2017 10:15 AM,
>>>> Jean-Baptiste
>>>> >>>> >> >> Onofré
>>>> >>>> >> >> wrote:
>>>> >>>> >> >> > > Thanks Reuven !
>>>> >>>> >> >> > >
>>>> >>>> >> >> > > I would like to emphasize on
>>>> some
>>>> >>>> >> >> highlights in
>>>> >>>> >> >> 2.2.0 release:
>>>> >>>> >> >> > >
>>>> >>>> >> >> > > - New IOs have been
>>>> introduced:
>>>> >>>> >> >> > > * TikaIO leveraging Apache
>>>> Tika,
>>>> >>>> >> >> allowing
>>>> >>>> >> >> the
>>>> >>>> >> >> deal with a lot
>>>> >>>> >> >> > of different
>>>> >>>> >> >> > > data formats
>>>> >>>> >> >> > > * RedisIO to read and write
>>>> >>>> >> >> key/value
>>>> >>>> >> >> pairs
>>>> >>>> >> >> from a Redis
>>>> >>>> >> >> > server. This
>>>> >>>> >> >> > IO will
>>>> >>>> >> >> > > be soon extended to Redis
>>>> PubSub.
>>>> >>>> >> >> > > * FileIO provides
>>>> transforms for
>>>> >>>> >> >> working
>>>> >>>> >> >> with
>>>> >>>> >> >> files (raw).
>>>> >>>> >> >> > Especially, it
>>>> >>>> >> >> > > provides matching file
>>>> patterns and
>>>> >>>> >> >> read
>>>> >>>> >> >> on
>>>> >>>> >> >> patterns. It can be
>>>> >>>> >> >> > easily
>>>> >>>> >> >> > extended
>>>> >>>> >> >> > > for a specific format (like
>>>> we do
>>>> >>>> >> >> in
>>>> >>>> >> >> AvroIO
>>>> >>>> >> >> or
>>>> >>>> >> >> TextIO now).
>>>> >>>> >> >> > > * SolrIO to interact with
>>>> Apache
>>>> >>>> >> >> Solr
>>>> >>>> >> >> (Lucene)
>>>> >>>> >> >> > >
>>>> >>>> >> >> > > - On the other hand,
>>>> improvements
>>>> >>>> >> >> have
>>>> >>>> >> >> been
>>>> >>>> >> >> performed on
>>>> >>>> >> >> > existing IOs:
>>>> >>>> >> >> > > * We started to introduce
>>>> readAll
>>>> >>>> >> >> pattern
>>>> >>>> >> >> in
>>>> >>>> >> >> IOs (AvroIO,
>>>> >>>> >> >> > TextIO, JdbcIO,
>>>> >>>> >> >> > > ...), allowing to pass
>>>> "request"
>>>> >>>> >> >> arguments
>>>> >>>> >> >> via an
>>>> >>>> >> >> input PCollection.
>>>> >>>> >> >> > > * ElasticsearchIO has an
>>>> improved
>>>> >>>> >> >> support
>>>> >>>> >> >> of
>>>> >>>> >> >> different
>>>> >>>> >> >> > Elasticsearch
>>>> >>>> >> >> > version
>>>> >>>> >> >> > > (including Elasticsearch
>>>> 5.x). It
>>>> >>>> >> >> also
>>>> >>>> >> >> now
>>>> >>>> >> >> supports SSL/TLS.
>>>> >>>> >> >> > > * HBaseIO is now able to do
>>>> >>>> >> >> dynamic
>>>> >>>> >> >> work
>>>> >>>> >> >> rebalancing
>>>> >>>> >> >> > > * KinesisIO uses a more
>>>> accurate
>>>> >>>> >> >> watermark
>>>> >>>> >> >> (based on
>>>> >>>> >> >> > approximateArrivalTimestamp)
>>>> >>>> >> >> > > * TextIO now supports custom
>>>> >>>> >> >> delimiter
>>>> >>>> >> >> and like
>>>> >>>> >> >> AvroIO,
>>>> >>>> >> >> > supports the
>>>> >>>> >> >> > readAll
>>>> >>>> >> >> > > pattern,
>>>> >>>> >> >> > > * Performance improvements
>>>> on
>>>> >>>> >> >> JdbcIO
>>>> >>>> >> >> when
>>>> >>>> >> >> it
>>>> >>>> >> >> has to read lot
>>>> >>>> >> >> > of rows
>>>> >>>> >> >> > > * Kafka write supports
>>>> >>>> >> >> Exactly-Once
>>>> >>>> >> >> pattern
>>>> >>>> >> >> (introduce in
>>>> >>>> >> >> > Kafka 0.11.x)
>>>> >>>> >> >> > >
>>>> >>>> >> >> > > - A new DSL has been
>>>> introduced:
>>>> >>>> >> >> the SQL
>>>> >>>> >> >> DSL !
>>>> >>>> >> >> > >
>>>> >>>> >> >> > > We are now focus on 2.3.0
>>>> release
>>>> >>>> >> >> with
>>>> >>>> >> >> new
>>>> >>>> >> >> improvements and
>>>> >>>> >> >> > features !
>>>> >>>> >> >> > >
>>>> >>>> >> >> > > Stay tuned !
>>>> >>>> >> >> > >
>>>> >>>> >> >> > > JB on behalf of the Apache
>>>> Beam
>>>> >>>> >> >> community.
>>>> >>>> >> >> > >
>>>> >>>> >> >> > > On 12/02/2017 11:40 PM,
>>>> Reuven Lax
>>>> >>>> >> >> wrote:
>>>> >>>> >> >> > >> The Apache Beam community is
>>>> >>>> >> >> pleased to
>>>> >>>> >> >> announce the
>>>> >>>> >> >> > availability of the
>>>> >>>> >> >> > >> 2.2.0 release.
>>>> >>>> >> >> > >>
>>>> >>>> >> >> > >> This release adds support for
>>>> >>>> >> >> generic
>>>> >>>> >> >> file
>>>> >>>> >> >> sources and sinks
>>>> >>>> >> >> > (beyond TextIO
>>>> >>>> >> >> > >> and AvroIO) using FileIO,
>>>> >>>> >> >> including
>>>> >>>> >> >> support for
>>>> >>>> >> >> dynamic
>>>> >>>> >> >> > filenames using
>>>> >>>> >> >> > >> readAll; this allows
>>>> streaming
>>>> >>>> >> >> pipelines
>>>> >>>> >> >> to now
>>>> >>>> >> >> read from files by
>>>> >>>> >> >> > >> continuously monitoring a
>>>> >>>> >> >> directory for
>>>> >>>> >> >> new
>>>> >>>> >> >> filw. Many other
>>>> >>>> >> >> > IOs are
>>>> >>>> >> >> > improved,
>>>> >>>> >> >> > >> notably including
>>>> exactly-once
>>>> >>>> >> >> support
>>>> >>>> >> >> for
>>>> >>>> >> >> the
>>>> >>>> >> >> Kafka sink. Initial
>>>> >>>> >> >> > support for
>>>> >>>> >> >> > >> BEAM-SQL is also included in
>>>> this
>>>> >>>> >> >> release.
>>>> >>>> >> >> For a
>>>> >>>> >> >> more-complete
>>>> >>>> >> >> > list of major
>>>> >>>> >> >> > >> changes in the release,
>>>> please
>>>> >>>> >> >> refer to
>>>> >>>> >> >> the
>>>> >>>> >> >> release notes [2].
>>>> >>>> >> >> > >>
>>>> >>>> >> >> > >> The 2.2.0 release is now the
>>>> >>>> >> >> recommended
>>>> >>>> >> >> version; we encourage
>>>> >>>> >> >> > everyone to
>>>> >>>> >> >> > >> upgrade from any earlier
>>>> releases.
>>>> >>>> >> >> > >>
>>>> >>>> >> >> > >> We’d like to invite everyone
>>>> to
>>>> >>>> >> >> try out
>>>> >>>> >> >> Apache
>>>> >>>> >> >> Beam today and
>>>> >>>> >> >> > consider
>>>> >>>> >> >> > >> joining our vibrant
>>>> community. We
>>>> >>>> >> >> welcome
>>>> >>>> >> >> feedback,
>>>> >>>> >> >> > contribution and
>>>> >>>> >> >> > >> participation through our
>>>> mailing
>>>> >>>> >> >> lists,
>>>> >>>> >> >> issue
>>>> >>>> >> >> tracker, pull
>>>> >>>> >> >> > requests, and
>>>> >>>> >> >> > >> events.
>>>> >>>> >> >> > >>
>>>> >>>> >> >> > >> - Reuven Lax, on behalf of
>>>> the
>>>> >>>> >> >> Apache
>>>> >>>> >> >> Beam
>>>> >>>> >> >> community.
>>>> >>>> >> >> > >>
>>>> >>>> >> >> > >> [1]
>>>> >>>> >> >> https://beam.apache.org/get-started/downloads/
>>>> >>>> >> >> <https://beam.apache.org/get-started/downloads/>
>>>> >>>> >> >> >
>>>> >>>> >> >> <https://beam.apache.org/get-started/downloads/
>>>> >>>> >> >> <https://beam.apache.org/get-started/downloads/
>>>> >>
>>>> >>>> >> >> > >> [2]
>>>> >>>> >> >> > >>
>>>> >>>> >> >> >
>>>> >>>> >> >>
>>>> >>>> >> >>
>>>> >>>> >> >>
>>>> >>>> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>>>> ctId=12319527&version=12341044
>>>> >>>> >> >>
>>>> >>>> >> >>
>>>> >>>> >> >>
>>>> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?proj
>>>> ectId=12319527&version=12341044>
>>>> >>>> >> >> >
>>>> >>>> >> >>
>>>> >>>> >> >>
>>>> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?proj
>>>> ectId=12319527&version=12341044
>>>> >>>> >> >>
>>>> >>>> >> >>
>>>> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?proj
>>>> ectId=12319527&version=12341044>>
>>>> >>>> >> >> > >>
>>>> >>>> >> >> > >
>>>> >>>> >> >> >
>>>> >>>> >> >> > --
>>>> >>>> >> >> > Jean-Baptiste Onofré
>>>> >>>> >> >> > jbonofre@apache.org <mailto:
>>>> jbonofre@apache.org>
>>>> >>>> >> >> <mailto:jbonofre@apache.org
>>>> >>>> >> >> <ma...@apache.org>>
>>>> >>>> >> >> > <mailto:jbonofre@apache.org
>>>> >>>> >> >> <ma...@apache.org>
>>>> >>>> >> >> <mailto:jbonofre@apache.org
>>>> >>>> >> >> <ma...@apache.org>>>
>>>> >>>> >> >> > http://blog.nanthrax.net
>>>> >>>> >> >> > Talend - http://www.talend.com
>>>> >>>> >> >> >
>>>> >>>> >> >> >
>>>> >>>> >> >> > --
>>>> >>>> >> >> > Jean-Baptiste Onofré
>>>> >>>> >> >> > jbonofre@apache.org <mailto:
>>>> jbonofre@apache.org>
>>>> >>>> >> >> <mailto:jbonofre@apache.org
>>>> >>>> >> >> <ma...@apache.org>>
>>>> >>>> >> >> > http://blog.nanthrax.net
>>>> >>>> >> >> > Talend - http://www.talend.com
>>>> >>>> >> >> >
>>>> >>>> >> >> >
>>>> >>>> >> >>
>>>> >>>> >> >> --
>>>> >>>> >> >> Jean-Baptiste Onofré
>>>> >>>> >> >> jbonofre@apache.org <ma...@apache.org>
>>>> >>>> >> >> http://blog.nanthrax.net
>>>> >>>> >> >> Talend - http://www.talend.com
>>>> >>>> >> >>
>>>> >>>> >> >>
>>>> >>>> >> >>
>>>> >>>> >> >
>>>> >>>> >> > --
>>>> >>>> >> > Jean-Baptiste Onofré
>>>> >>>> >> > jbonofre@apache.org
>>>> >>>> >> > http://blog.nanthrax.net
>>>> >>>> >> > Talend - http://www.talend.com
>>>> >>>
>>>> >>>
>>>> >
>>>>
>>>
>>>
>
Re: Apache Beam, version 2.2.0
Posted by Reuven Lax <re...@google.com>.
Something strange is going on. We can see it in the dropdown list in the
UI, but if you click on that tag you get a 404.
On Wed, Dec 13, 2017 at 2:37 PM, Reuven Lax <re...@google.com> wrote:
> I created a v2.2 tag. Let me look to see what happened.
>
> On Dec 13, 2017 2:31 PM, "Steve Niemitz" <sn...@apache.org> wrote:
>
>> Sorry for resurrecting this thread, but I was trying to build 2.2.0 from
>> source today and noticed there was no v2.2.0 tag (only v2.2.0-RC4). I
>> assume that's not intentional?
>>
>> On Fri, Dec 8, 2017 at 3:35 AM, Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> Thanks Eugene for opening the poll (sorry if I didn't before I was
>>> quite busy in the last two days but expected to do it today).
>>>
>>>
>>> On Fri, Dec 8, 2017 at 1:27 AM, Ahmet Altay <al...@google.com> wrote:
>>> >
>>> > On Thu, Dec 7, 2017 at 3:51 PM, Eugene Kirpichov <kirpichov@google.com
>>> >
>>> > wrote:
>>> >>
>>> >> I've sent the poll
>>> >> https://lists.apache.org/thread.html/5bc2e184a24de9dbc8184ff
>>> d2720d1894010497d47d956b395e037df@%3Cuser.beam.apache.org%3E
>>> >> Will figure out how to tweet from @ApacheBeam, and sent the Twitter
>>> poll
>>> >> as well (or ask someone to).
>>> >
>>> >
>>> > I tweeted the poll.
>>> >
>>> >>
>>> >>
>>> >> On Wed, Dec 6, 2017 at 1:47 PM Lukasz Cwik <lc...@google.com> wrote:
>>> >>>
>>> >>> +1 on moving forward with the plan suggested by kirpichov@
>>> >>>
>>> >>> On Wed, Dec 6, 2017 at 9:14 AM, Robert Bradshaw <robertwb@google.com
>>> >
>>> >>> wrote:
>>> >>>>
>>> >>>> +1 to moving forward with this plan.
>>> >>>>
>>> >>>> (FWIW, this seems *less* backwards incompatible than, say, moving
>>> from
>>> >>>> Spark 1 to Spark 2, which was decided much quicker. I suppose the
>>> >>>> Spark change has a lower bound on the number of users it could
>>> impact
>>> >>>> though.)
>>> >>>>
>>> >>>> On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <
>>> kirpichov@google.com>
>>> >>>> wrote:
>>> >>>> > Okay, then let's go forward. Seems that we should:
>>> >>>> > - Open a new poll on user@, in light of 2.2 having been released
>>> >>>> > - Open a twitter poll
>>> >>>> > - Tweet that there's also a poll going on on user@
>>> >>>> > - Runner authors will reach out to respective runner user
>>> communities
>>> >>>> > - 2 weeks later we gather results and decide
>>> >>>> > ?
>>> >>>> >
>>> >>>> > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <ie...@gmail.com>
>>> wrote:
>>> >>>> >>
>>> >>>> >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far
>>> away,
>>> >>>> >> and starting to improve Beam to offer a Java 8 friendly
>>> experience
>>> >>>> >> seems like an excellent idea.
>>> >>>> >>
>>> >>>> >> I understand the backwards compatibility argument. We should do
>>> the
>>> >>>> >> poll in twitter + try to reach more users for comments. If you
>>> >>>> >> consider that it is worth, I can open a second poll at user@.
>>> >>>> >>
>>> >>>> >> In any case we should try to move forward, even if we have more
>>> than
>>> >>>> >> 5% of users who want to stay on Java 7 we can consider to
>>> maintain
>>> >>>> >> minor releases of a backwards compatible version where we can
>>> >>>> >> backport
>>> >>>> >> only critical fixes e.g. security/data related errors but nothing
>>> >>>> >> new,
>>> >>>> >> in case some user really needs to have them. Of course this can
>>> be
>>> >>>> >> some extra work (to be discussed).
>>> >>>> >>
>>> >>>> >>
>>> >>>> >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré
>>> >>>> >> <jb...@nanthrax.net>
>>> >>>> >> wrote:
>>> >>>> >> > +1, and sorry again, I thought we got an consensus.
>>> >>>> >> >
>>> >>>> >> > Regards
>>> >>>> >> > JB
>>> >>>> >> >
>>> >>>> >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote:
>>> >>>> >> >>
>>> >>>> >> >> +1 to the poll and also to Reuven's point.
>>> >>>> >> >>
>>> >>>> >> >> Those without a support contract would have been using JDK 7
>>> >>>> >> >> without
>>> >>>> >> >> security updates for years. IMO it seems harmful, as a
>>> netizen, to
>>> >>>> >> >> encourage
>>> >>>> >> >> its use/existence.
>>> >>>> >> >>
>>> >>>> >> >> If there's no noise from the prior thread, then I would
>>> assume no
>>> >>>> >> >> one
>>> >>>> >> >> on
>>> >>>> >> >> user@ has any objection. Anyone else with customers should
>>> reach
>>> >>>> >> >> out to
>>> >>>> >> >> them.
>>> >>>> >> >>
>>> >>>> >> >> Kenn
>>> >>>> >> >>
>>> >>>> >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <relax@google.com
>>> >>>> >> >> <ma...@google.com>> wrote:
>>> >>>> >> >>
>>> >>>> >> >> Technically it's a backwards-incompatible change, however
>>> if
>>> >>>> >> >> we are
>>> >>>> >> >> convinced the risk is low we could do it.
>>> >>>> >> >>
>>> >>>> >> >> As mentioned on the original thread, it's not clear that
>>> all
>>> >>>> >> >> Beam
>>> >>>> >> >> users read
>>> >>>> >> >> user@ - e.g. most Dataflow users definitely do not. I
>>> think we
>>> >>>> >> >> need
>>> >>>> >> >> to
>>> >>>> >> >> separately reach out to users of each runner through
>>> >>>> >> >> runner-specific
>>> >>>> >> >> channels.
>>> >>>> >> >>
>>> >>>> >> >> Reuven
>>> >>>> >> >>
>>> >>>> >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov
>>> >>>> >> >> <kirpichov@google.com
>>> >>>> >> >> <ma...@google.com>> wrote:
>>> >>>> >> >>
>>> >>>> >> >> On the original thread
>>> >>>> >> >>
>>> >>>> >> >>
>>> >>>> >> >>
>>> >>>> >> >> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9
>>> f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
>>> >>>> >> >>
>>> >>>> >> >>
>>> >>>> >> >>
>>> >>>> >> >> <https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e
>>> 9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E>
>>> >>>> >> >> ,
>>> >>>> >> >> Robert and Ismaël were in favor of no major version
>>> change
>>> >>>> >> >> [Ismaël
>>> >>>> >> >> said:/Also I am afraid that if we wait/
>>> >>>> >> >> /until we have enough changes to switch Beam to a new
>>> >>>> >> >> major
>>> >>>> >> >> version the
>>> >>>> >> >> switch to Java 8 will happen too late, probably after
>>> Java
>>> >>>> >> >> 8's
>>> >>>> >> >> end
>>> >>>> >> >> of
>>> >>>> >> >> life. And I am not exaggerating, Java 8 is planned to
>>> EOL
>>> >>>> >> >> next
>>> >>>> >> >> march
>>> >>>> >> >> 2018!/]; JB and now Reuven are in favor of a major
>>> version
>>> >>>> >> >> change;
>>> >>>> >> >>
>>> >>>> >> >> nobody so far argued against switching to Java8 in
>>> >>>> >> >> general.
>>> >>>> >> >>
>>> >>>> >> >> I'm personally in favor of no major version change
>>> (i.e.
>>> >>>> >> >> not
>>> >>>> >> >> waiting
>>> >>>> >> >> until all other large changes for Beam 3.0 converge,
>>> which
>>> >>>> >> >> will
>>> >>>> >> >> likely
>>> >>>> >> >> be many months), because:
>>> >>>> >> >> - Reasons Ismaël cited; plus the reason that most
>>> people
>>> >>>> >> >> are
>>> >>>> >> >> likely
>>> >>>> >> >> already using Java 8.
>>> >>>> >> >> - Going Java8-only earlier will make other Beam 3.0
>>> APIs
>>> >>>> >> >> better
>>> >>>> >> >> for
>>> >>>> >> >> Java8 users, because we (Beam contributors) will have
>>> >>>> >> >> experience
>>> >>>> >> >> working
>>> >>>> >> >> with them within the SDK in Java8 (e.g. writing tests
>>> with
>>> >>>> >> >> use
>>> >>>> >> >> of
>>> >>>> >> >> lambdas and noticing whether it's clunky, or whether
>>> some
>>> >>>> >> >> other
>>> >>>> >> >> Beam
>>> >>>> >> >> APIs need better Java8 support).
>>> >>>> >> >> - Going Java8 will make it more reasonable to include
>>> >>>> >> >> (mostly
>>> >>>> >> >> or
>>> >>>> >> >> only)
>>> >>>> >> >> Java8 snippets in Beam documentation, which will
>>> obviously
>>> >>>> >> >> look
>>> >>>> >> >> more
>>> >>>> >> >> concise and attractive, addressing one of the common
>>> >>>> >> >> concerns
>>> >>>> >> >> of
>>> >>>> >> >> Beam
>>> >>>> >> >> users that it has a heavyweight API compared to
>>> >>>> >> >> functional-style
>>> >>>> >> >> APIs of
>>> >>>> >> >> Spark etc.
>>> >>>> >> >>
>>> >>>> >> >> I think resolving this via a poll of users would be
>>> >>>> >> >> reasonable.
>>> >>>> >> >> I'd
>>> >>>> >> >> suggest e.g. the following phrasing:
>>> >>>> >> >>
>>> >>>> >> >> Apache Beam is considering dropping support for Java
>>> 7,
>>> >>>> >> >> and
>>> >>>> >> >> supporting
>>> >>>> >> >> only Java 8 and above in a subsequent release. How
>>> would
>>> >>>> >> >> it
>>> >>>> >> >> impact
>>> >>>> >> >> your
>>> >>>> >> >> usage of Beam?
>>> >>>> >> >> - I am already using only Java 8+ for building my Beam
>>> >>>> >> >> code
>>> >>>> >> >> - I am using Java 7 for building my Beam code, but I
>>> would
>>> >>>> >> >> have
>>> >>>> >> >> no
>>> >>>> >> >> trouble switching to Java 8
>>> >>>> >> >> - I am using Java 7 for building my Beam code, and
>>> >>>> >> >> dropping
>>> >>>> >> >> Java 7
>>> >>>> >> >> would
>>> >>>> >> >> be a blocker or hindrance to adopting the new release
>>> for
>>> >>>> >> >> me
>>> >>>> >> >>
>>> >>>> >> >> We could tweet this poll on Apache Beam twitter and
>>> >>>> >> >> publish on
>>> >>>> >> >> user@,
>>> >>>> >> >> and, say, if we receive 5% or fewer votes for option 3
>>> >>>> >> >> after
>>> >>>> >> >> keeping it
>>> >>>> >> >> open for 2 weeks, then adopt Java 8 without a major
>>> >>>> >> >> version
>>> >>>> >> >> change.
>>> >>>> >> >>
>>> >>>> >> >> WDYT?
>>> >>>> >> >>
>>> >>>> >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré
>>> >>>> >> >> <jb@nanthrax.net
>>> >>>> >> >> <ma...@nanthrax.net>> wrote:
>>> >>>> >> >>
>>> >>>> >> >> Good idea ! Definitely +1
>>> >>>> >> >>
>>> >>>> >> >> Regards
>>> >>>> >> >> JB
>>> >>>> >> >>
>>> >>>> >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote:
>>> >>>> >> >> > We should bring this up on the Beam 3.0 thread.
>>> >>>> >> >> Since
>>> >>>> >> >> it's
>>> >>>> >> >> technically a
>>> >>>> >> >> > backwards-incompatible change, it might make a
>>> good
>>> >>>> >> >> item
>>> >>>> >> >> for Beam
>>> >>>> >> >> 3.0.
>>> >>>> >> >> >
>>> >>>> >> >> > Reuven
>>> >>>> >> >> >
>>> >>>> >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste
>>> >>>> >> >> Onofré
>>> >>>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
>>> >>>> >> >> > <mailto:jb@nanthrax.net <mailto:
>>> jb@nanthrax.net>>>
>>> >>>> >> >> wrote:
>>> >>>> >> >> >
>>> >>>> >> >> > My apologizes, I thought we had a consensus
>>> >>>> >> >> already.
>>> >>>> >> >> >
>>> >>>> >> >> > Regards
>>> >>>> >> >> > JB
>>> >>>> >> >> >
>>> >>>> >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov
>>> wrote:
>>> >>>> >> >> >
>>> >>>> >> >> > Thanks JB for sending the detailed
>>> notes
>>> >>>> >> >> about
>>> >>>> >> >> new
>>> >>>> >> >> stuff
>>> >>>> >> >> in 2.2.0! A lot
>>> >>>> >> >> > of exciting things indeed.
>>> >>>> >> >> >
>>> >>>> >> >> > Regarding Java 8: I thought our
>>> consensus
>>> >>>> >> >> was to
>>> >>>> >> >> have the
>>> >>>> >> >> release notes
>>> >>>> >> >> > say that we're *considering* going
>>> >>>> >> >> Java8-only,
>>> >>>> >> >> and
>>> >>>> >> >> use
>>> >>>> >> >> that to get more
>>> >>>> >> >> > opinions from the user community - but
>>> I
>>> >>>> >> >> can't
>>> >>>> >> >> find
>>> >>>> >> >> the
>>> >>>> >> >> emails that made
>>> >>>> >> >> > me think so.
>>> >>>> >> >> >
>>> >>>> >> >> > +Ismaël Mejía <mailto:
>>> iemejia@gmail.com
>>> >>>> >> >> <ma...@gmail.com> <mailto:
>>> iemejia@gmail.com
>>> >>>> >> >> <ma...@gmail.com>>> - do
>>> >>>> >> >> > you think we should formally conclude
>>> the
>>> >>>> >> >> vote
>>> >>>> >> >> on
>>> >>>> >> >> the
>>> >>>> >> >> thread [VOTE]
>>> >>>> >> >> > [DISCUSSION] Remove support for Java 7?
>>> >>>> >> >> > Or should we take more steps - e.g.
>>> perhaps
>>> >>>> >> >> tweet a
>>> >>>> >> >> link
>>> >>>> >> >> to that thread
>>> >>>> >> >> > from the Beam twitter account, ask
>>> people
>>> >>>> >> >> to
>>> >>>> >> >> chime
>>> >>>> >> >> in,
>>> >>>> >> >> and wait for say
>>> >>>> >> >> > 2 weeks before declaring a conclusion?
>>> >>>> >> >> >
>>> >>>> >> >> > Let's also have a process JIRA for
>>> going
>>> >>>> >> >> Java8.
>>> >>>> >> >> I've
>>> >>>> >> >> filed one:
>>> >>>> >> >> > https://issues.apache.org/jira
>>> /browse/BEAM-3285
>>> >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>
>>> >>>> >> >> >
>>> >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285
>>> >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285
>>> >>
>>> >>>> >> >> >
>>> >>>> >> >> > On Mon, Dec 4, 2017 at 1:58 AM
>>> >>>> >> >> Jean-Baptiste
>>> >>>> >> >> Onofré
>>> >>>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
>>> >>>> >> >> > <mailto:jb@nanthrax.net
>>> >>>> >> >> <ma...@nanthrax.net>>
>>> >>>> >> >> <mailto:jb@nanthrax.net <ma...@nanthrax.net>
>>> >>>> >> >> > <mailto:jb@nanthrax.net
>>> >>>> >> >> <ma...@nanthrax.net>>>>
>>> >>>> >> >> wrote:
>>> >>>> >> >> >
>>> >>>> >> >> > Just an important note that we
>>> forgot
>>> >>>> >> >> to
>>> >>>> >> >> mention.
>>> >>>> >> >> >
>>> >>>> >> >> > !! The 2.2.0 release will be the
>>> last
>>> >>>> >> >> one
>>> >>>> >> >> supporting
>>> >>>> >> >> Spark 1.x and
>>> >>>> >> >> > Java 7 !!
>>> >>>> >> >> >
>>> >>>> >> >> > Starting from Beam 2.3.0, the
>>> Spark
>>> >>>> >> >> runner
>>> >>>> >> >> will work
>>> >>>> >> >> only with
>>> >>>> >> >> > Spark 2.x and we
>>> >>>> >> >> > will focus only Java 8.
>>> >>>> >> >> >
>>> >>>> >> >> > Regards
>>> >>>> >> >> > JB
>>> >>>> >> >> >
>>> >>>> >> >> > On 12/04/2017 10:15 AM,
>>> Jean-Baptiste
>>> >>>> >> >> Onofré
>>> >>>> >> >> wrote:
>>> >>>> >> >> > > Thanks Reuven !
>>> >>>> >> >> > >
>>> >>>> >> >> > > I would like to emphasize on
>>> some
>>> >>>> >> >> highlights in
>>> >>>> >> >> 2.2.0 release:
>>> >>>> >> >> > >
>>> >>>> >> >> > > - New IOs have been introduced:
>>> >>>> >> >> > > * TikaIO leveraging Apache
>>> Tika,
>>> >>>> >> >> allowing
>>> >>>> >> >> the
>>> >>>> >> >> deal with a lot
>>> >>>> >> >> > of different
>>> >>>> >> >> > > data formats
>>> >>>> >> >> > > * RedisIO to read and write
>>> >>>> >> >> key/value
>>> >>>> >> >> pairs
>>> >>>> >> >> from a Redis
>>> >>>> >> >> > server. This
>>> >>>> >> >> > IO will
>>> >>>> >> >> > > be soon extended to Redis
>>> PubSub.
>>> >>>> >> >> > > * FileIO provides transforms
>>> for
>>> >>>> >> >> working
>>> >>>> >> >> with
>>> >>>> >> >> files (raw).
>>> >>>> >> >> > Especially, it
>>> >>>> >> >> > > provides matching file
>>> patterns and
>>> >>>> >> >> read
>>> >>>> >> >> on
>>> >>>> >> >> patterns. It can be
>>> >>>> >> >> > easily
>>> >>>> >> >> > extended
>>> >>>> >> >> > > for a specific format (like we
>>> do
>>> >>>> >> >> in
>>> >>>> >> >> AvroIO
>>> >>>> >> >> or
>>> >>>> >> >> TextIO now).
>>> >>>> >> >> > > * SolrIO to interact with
>>> Apache
>>> >>>> >> >> Solr
>>> >>>> >> >> (Lucene)
>>> >>>> >> >> > >
>>> >>>> >> >> > > - On the other hand,
>>> improvements
>>> >>>> >> >> have
>>> >>>> >> >> been
>>> >>>> >> >> performed on
>>> >>>> >> >> > existing IOs:
>>> >>>> >> >> > > * We started to introduce
>>> readAll
>>> >>>> >> >> pattern
>>> >>>> >> >> in
>>> >>>> >> >> IOs (AvroIO,
>>> >>>> >> >> > TextIO, JdbcIO,
>>> >>>> >> >> > > ...), allowing to pass
>>> "request"
>>> >>>> >> >> arguments
>>> >>>> >> >> via an
>>> >>>> >> >> input PCollection.
>>> >>>> >> >> > > * ElasticsearchIO has an
>>> improved
>>> >>>> >> >> support
>>> >>>> >> >> of
>>> >>>> >> >> different
>>> >>>> >> >> > Elasticsearch
>>> >>>> >> >> > version
>>> >>>> >> >> > > (including Elasticsearch 5.x).
>>> It
>>> >>>> >> >> also
>>> >>>> >> >> now
>>> >>>> >> >> supports SSL/TLS.
>>> >>>> >> >> > > * HBaseIO is now able to do
>>> >>>> >> >> dynamic
>>> >>>> >> >> work
>>> >>>> >> >> rebalancing
>>> >>>> >> >> > > * KinesisIO uses a more
>>> accurate
>>> >>>> >> >> watermark
>>> >>>> >> >> (based on
>>> >>>> >> >> > approximateArrivalTimestamp)
>>> >>>> >> >> > > * TextIO now supports custom
>>> >>>> >> >> delimiter
>>> >>>> >> >> and like
>>> >>>> >> >> AvroIO,
>>> >>>> >> >> > supports the
>>> >>>> >> >> > readAll
>>> >>>> >> >> > > pattern,
>>> >>>> >> >> > > * Performance improvements on
>>> >>>> >> >> JdbcIO
>>> >>>> >> >> when
>>> >>>> >> >> it
>>> >>>> >> >> has to read lot
>>> >>>> >> >> > of rows
>>> >>>> >> >> > > * Kafka write supports
>>> >>>> >> >> Exactly-Once
>>> >>>> >> >> pattern
>>> >>>> >> >> (introduce in
>>> >>>> >> >> > Kafka 0.11.x)
>>> >>>> >> >> > >
>>> >>>> >> >> > > - A new DSL has been
>>> introduced:
>>> >>>> >> >> the SQL
>>> >>>> >> >> DSL !
>>> >>>> >> >> > >
>>> >>>> >> >> > > We are now focus on 2.3.0
>>> release
>>> >>>> >> >> with
>>> >>>> >> >> new
>>> >>>> >> >> improvements and
>>> >>>> >> >> > features !
>>> >>>> >> >> > >
>>> >>>> >> >> > > Stay tuned !
>>> >>>> >> >> > >
>>> >>>> >> >> > > JB on behalf of the Apache Beam
>>> >>>> >> >> community.
>>> >>>> >> >> > >
>>> >>>> >> >> > > On 12/02/2017 11:40 PM, Reuven
>>> Lax
>>> >>>> >> >> wrote:
>>> >>>> >> >> > >> The Apache Beam community is
>>> >>>> >> >> pleased to
>>> >>>> >> >> announce the
>>> >>>> >> >> > availability of the
>>> >>>> >> >> > >> 2.2.0 release.
>>> >>>> >> >> > >>
>>> >>>> >> >> > >> This release adds support for
>>> >>>> >> >> generic
>>> >>>> >> >> file
>>> >>>> >> >> sources and sinks
>>> >>>> >> >> > (beyond TextIO
>>> >>>> >> >> > >> and AvroIO) using FileIO,
>>> >>>> >> >> including
>>> >>>> >> >> support for
>>> >>>> >> >> dynamic
>>> >>>> >> >> > filenames using
>>> >>>> >> >> > >> readAll; this allows streaming
>>> >>>> >> >> pipelines
>>> >>>> >> >> to now
>>> >>>> >> >> read from files by
>>> >>>> >> >> > >> continuously monitoring a
>>> >>>> >> >> directory for
>>> >>>> >> >> new
>>> >>>> >> >> filw. Many other
>>> >>>> >> >> > IOs are
>>> >>>> >> >> > improved,
>>> >>>> >> >> > >> notably including exactly-once
>>> >>>> >> >> support
>>> >>>> >> >> for
>>> >>>> >> >> the
>>> >>>> >> >> Kafka sink. Initial
>>> >>>> >> >> > support for
>>> >>>> >> >> > >> BEAM-SQL is also included in
>>> this
>>> >>>> >> >> release.
>>> >>>> >> >> For a
>>> >>>> >> >> more-complete
>>> >>>> >> >> > list of major
>>> >>>> >> >> > >> changes in the release, please
>>> >>>> >> >> refer to
>>> >>>> >> >> the
>>> >>>> >> >> release notes [2].
>>> >>>> >> >> > >>
>>> >>>> >> >> > >> The 2.2.0 release is now the
>>> >>>> >> >> recommended
>>> >>>> >> >> version; we encourage
>>> >>>> >> >> > everyone to
>>> >>>> >> >> > >> upgrade from any earlier
>>> releases.
>>> >>>> >> >> > >>
>>> >>>> >> >> > >> We’d like to invite everyone
>>> to
>>> >>>> >> >> try out
>>> >>>> >> >> Apache
>>> >>>> >> >> Beam today and
>>> >>>> >> >> > consider
>>> >>>> >> >> > >> joining our vibrant
>>> community. We
>>> >>>> >> >> welcome
>>> >>>> >> >> feedback,
>>> >>>> >> >> > contribution and
>>> >>>> >> >> > >> participation through our
>>> mailing
>>> >>>> >> >> lists,
>>> >>>> >> >> issue
>>> >>>> >> >> tracker, pull
>>> >>>> >> >> > requests, and
>>> >>>> >> >> > >> events.
>>> >>>> >> >> > >>
>>> >>>> >> >> > >> - Reuven Lax, on behalf of the
>>> >>>> >> >> Apache
>>> >>>> >> >> Beam
>>> >>>> >> >> community.
>>> >>>> >> >> > >>
>>> >>>> >> >> > >> [1]
>>> >>>> >> >> https://beam.apache.org/get-started/downloads/
>>> >>>> >> >> <https://beam.apache.org/get-started/downloads/>
>>> >>>> >> >> >
>>> >>>> >> >> <https://beam.apache.org/get-started/downloads/
>>> >>>> >> >> <https://beam.apache.org/get-started/downloads/>>
>>> >>>> >> >> > >> [2]
>>> >>>> >> >> > >>
>>> >>>> >> >> >
>>> >>>> >> >>
>>> >>>> >> >>
>>> >>>> >> >>
>>> >>>> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>>> ctId=12319527&version=12341044
>>> >>>> >> >>
>>> >>>> >> >>
>>> >>>> >> >>
>>> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?proj
>>> ectId=12319527&version=12341044>
>>> >>>> >> >> >
>>> >>>> >> >>
>>> >>>> >> >>
>>> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?proj
>>> ectId=12319527&version=12341044
>>> >>>> >> >>
>>> >>>> >> >>
>>> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?proj
>>> ectId=12319527&version=12341044>>
>>> >>>> >> >> > >>
>>> >>>> >> >> > >
>>> >>>> >> >> >
>>> >>>> >> >> > --
>>> >>>> >> >> > Jean-Baptiste Onofré
>>> >>>> >> >> > jbonofre@apache.org <mailto:
>>> jbonofre@apache.org>
>>> >>>> >> >> <mailto:jbonofre@apache.org
>>> >>>> >> >> <ma...@apache.org>>
>>> >>>> >> >> > <mailto:jbonofre@apache.org
>>> >>>> >> >> <ma...@apache.org>
>>> >>>> >> >> <mailto:jbonofre@apache.org
>>> >>>> >> >> <ma...@apache.org>>>
>>> >>>> >> >> > http://blog.nanthrax.net
>>> >>>> >> >> > Talend - http://www.talend.com
>>> >>>> >> >> >
>>> >>>> >> >> >
>>> >>>> >> >> > --
>>> >>>> >> >> > Jean-Baptiste Onofré
>>> >>>> >> >> > jbonofre@apache.org <mailto:
>>> jbonofre@apache.org>
>>> >>>> >> >> <mailto:jbonofre@apache.org
>>> >>>> >> >> <ma...@apache.org>>
>>> >>>> >> >> > http://blog.nanthrax.net
>>> >>>> >> >> > Talend - http://www.talend.com
>>> >>>> >> >> >
>>> >>>> >> >> >
>>> >>>> >> >>
>>> >>>> >> >> --
>>> >>>> >> >> Jean-Baptiste Onofré
>>> >>>> >> >> jbonofre@apache.org <ma...@apache.org>
>>> >>>> >> >> http://blog.nanthrax.net
>>> >>>> >> >> Talend - http://www.talend.com
>>> >>>> >> >>
>>> >>>> >> >>
>>> >>>> >> >>
>>> >>>> >> >
>>> >>>> >> > --
>>> >>>> >> > Jean-Baptiste Onofré
>>> >>>> >> > jbonofre@apache.org
>>> >>>> >> > http://blog.nanthrax.net
>>> >>>> >> > Talend - http://www.talend.com
>>> >>>
>>> >>>
>>> >
>>>
>>
>>
Re: Apache Beam, version 2.2.0
Posted by Reuven Lax <re...@google.com>.
I created a v2.2 tag. Let me look to see what happened.
On Dec 13, 2017 2:31 PM, "Steve Niemitz" <sn...@apache.org> wrote:
> Sorry for resurrecting this thread, but I was trying to build 2.2.0 from
> source today and noticed there was no v2.2.0 tag (only v2.2.0-RC4). I
> assume that's not intentional?
>
> On Fri, Dec 8, 2017 at 3:35 AM, Ismaël Mejía <ie...@gmail.com> wrote:
>
>> Thanks Eugene for opening the poll (sorry if I didn't before I was
>> quite busy in the last two days but expected to do it today).
>>
>>
>> On Fri, Dec 8, 2017 at 1:27 AM, Ahmet Altay <al...@google.com> wrote:
>> >
>> > On Thu, Dec 7, 2017 at 3:51 PM, Eugene Kirpichov <ki...@google.com>
>> > wrote:
>> >>
>> >> I've sent the poll
>> >> https://lists.apache.org/thread.html/5bc2e184a24de9dbc8184ff
>> d2720d1894010497d47d956b395e037df@%3Cuser.beam.apache.org%3E
>> >> Will figure out how to tweet from @ApacheBeam, and sent the Twitter
>> poll
>> >> as well (or ask someone to).
>> >
>> >
>> > I tweeted the poll.
>> >
>> >>
>> >>
>> >> On Wed, Dec 6, 2017 at 1:47 PM Lukasz Cwik <lc...@google.com> wrote:
>> >>>
>> >>> +1 on moving forward with the plan suggested by kirpichov@
>> >>>
>> >>> On Wed, Dec 6, 2017 at 9:14 AM, Robert Bradshaw <ro...@google.com>
>> >>> wrote:
>> >>>>
>> >>>> +1 to moving forward with this plan.
>> >>>>
>> >>>> (FWIW, this seems *less* backwards incompatible than, say, moving
>> from
>> >>>> Spark 1 to Spark 2, which was decided much quicker. I suppose the
>> >>>> Spark change has a lower bound on the number of users it could impact
>> >>>> though.)
>> >>>>
>> >>>> On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <
>> kirpichov@google.com>
>> >>>> wrote:
>> >>>> > Okay, then let's go forward. Seems that we should:
>> >>>> > - Open a new poll on user@, in light of 2.2 having been released
>> >>>> > - Open a twitter poll
>> >>>> > - Tweet that there's also a poll going on on user@
>> >>>> > - Runner authors will reach out to respective runner user
>> communities
>> >>>> > - 2 weeks later we gather results and decide
>> >>>> > ?
>> >>>> >
>> >>>> > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <ie...@gmail.com>
>> wrote:
>> >>>> >>
>> >>>> >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far
>> away,
>> >>>> >> and starting to improve Beam to offer a Java 8 friendly experience
>> >>>> >> seems like an excellent idea.
>> >>>> >>
>> >>>> >> I understand the backwards compatibility argument. We should do
>> the
>> >>>> >> poll in twitter + try to reach more users for comments. If you
>> >>>> >> consider that it is worth, I can open a second poll at user@.
>> >>>> >>
>> >>>> >> In any case we should try to move forward, even if we have more
>> than
>> >>>> >> 5% of users who want to stay on Java 7 we can consider to maintain
>> >>>> >> minor releases of a backwards compatible version where we can
>> >>>> >> backport
>> >>>> >> only critical fixes e.g. security/data related errors but nothing
>> >>>> >> new,
>> >>>> >> in case some user really needs to have them. Of course this can be
>> >>>> >> some extra work (to be discussed).
>> >>>> >>
>> >>>> >>
>> >>>> >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré
>> >>>> >> <jb...@nanthrax.net>
>> >>>> >> wrote:
>> >>>> >> > +1, and sorry again, I thought we got an consensus.
>> >>>> >> >
>> >>>> >> > Regards
>> >>>> >> > JB
>> >>>> >> >
>> >>>> >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote:
>> >>>> >> >>
>> >>>> >> >> +1 to the poll and also to Reuven's point.
>> >>>> >> >>
>> >>>> >> >> Those without a support contract would have been using JDK 7
>> >>>> >> >> without
>> >>>> >> >> security updates for years. IMO it seems harmful, as a
>> netizen, to
>> >>>> >> >> encourage
>> >>>> >> >> its use/existence.
>> >>>> >> >>
>> >>>> >> >> If there's no noise from the prior thread, then I would assume
>> no
>> >>>> >> >> one
>> >>>> >> >> on
>> >>>> >> >> user@ has any objection. Anyone else with customers should
>> reach
>> >>>> >> >> out to
>> >>>> >> >> them.
>> >>>> >> >>
>> >>>> >> >> Kenn
>> >>>> >> >>
>> >>>> >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <relax@google.com
>> >>>> >> >> <ma...@google.com>> wrote:
>> >>>> >> >>
>> >>>> >> >> Technically it's a backwards-incompatible change, however
>> if
>> >>>> >> >> we are
>> >>>> >> >> convinced the risk is low we could do it.
>> >>>> >> >>
>> >>>> >> >> As mentioned on the original thread, it's not clear that
>> all
>> >>>> >> >> Beam
>> >>>> >> >> users read
>> >>>> >> >> user@ - e.g. most Dataflow users definitely do not. I
>> think we
>> >>>> >> >> need
>> >>>> >> >> to
>> >>>> >> >> separately reach out to users of each runner through
>> >>>> >> >> runner-specific
>> >>>> >> >> channels.
>> >>>> >> >>
>> >>>> >> >> Reuven
>> >>>> >> >>
>> >>>> >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov
>> >>>> >> >> <kirpichov@google.com
>> >>>> >> >> <ma...@google.com>> wrote:
>> >>>> >> >>
>> >>>> >> >> On the original thread
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9
>> f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> <https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e
>> 9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E>
>> >>>> >> >> ,
>> >>>> >> >> Robert and Ismaël were in favor of no major version
>> change
>> >>>> >> >> [Ismaël
>> >>>> >> >> said:/Also I am afraid that if we wait/
>> >>>> >> >> /until we have enough changes to switch Beam to a new
>> >>>> >> >> major
>> >>>> >> >> version the
>> >>>> >> >> switch to Java 8 will happen too late, probably after
>> Java
>> >>>> >> >> 8's
>> >>>> >> >> end
>> >>>> >> >> of
>> >>>> >> >> life. And I am not exaggerating, Java 8 is planned to
>> EOL
>> >>>> >> >> next
>> >>>> >> >> march
>> >>>> >> >> 2018!/]; JB and now Reuven are in favor of a major
>> version
>> >>>> >> >> change;
>> >>>> >> >>
>> >>>> >> >> nobody so far argued against switching to Java8 in
>> >>>> >> >> general.
>> >>>> >> >>
>> >>>> >> >> I'm personally in favor of no major version change
>> (i.e.
>> >>>> >> >> not
>> >>>> >> >> waiting
>> >>>> >> >> until all other large changes for Beam 3.0 converge,
>> which
>> >>>> >> >> will
>> >>>> >> >> likely
>> >>>> >> >> be many months), because:
>> >>>> >> >> - Reasons Ismaël cited; plus the reason that most
>> people
>> >>>> >> >> are
>> >>>> >> >> likely
>> >>>> >> >> already using Java 8.
>> >>>> >> >> - Going Java8-only earlier will make other Beam 3.0
>> APIs
>> >>>> >> >> better
>> >>>> >> >> for
>> >>>> >> >> Java8 users, because we (Beam contributors) will have
>> >>>> >> >> experience
>> >>>> >> >> working
>> >>>> >> >> with them within the SDK in Java8 (e.g. writing tests
>> with
>> >>>> >> >> use
>> >>>> >> >> of
>> >>>> >> >> lambdas and noticing whether it's clunky, or whether
>> some
>> >>>> >> >> other
>> >>>> >> >> Beam
>> >>>> >> >> APIs need better Java8 support).
>> >>>> >> >> - Going Java8 will make it more reasonable to include
>> >>>> >> >> (mostly
>> >>>> >> >> or
>> >>>> >> >> only)
>> >>>> >> >> Java8 snippets in Beam documentation, which will
>> obviously
>> >>>> >> >> look
>> >>>> >> >> more
>> >>>> >> >> concise and attractive, addressing one of the common
>> >>>> >> >> concerns
>> >>>> >> >> of
>> >>>> >> >> Beam
>> >>>> >> >> users that it has a heavyweight API compared to
>> >>>> >> >> functional-style
>> >>>> >> >> APIs of
>> >>>> >> >> Spark etc.
>> >>>> >> >>
>> >>>> >> >> I think resolving this via a poll of users would be
>> >>>> >> >> reasonable.
>> >>>> >> >> I'd
>> >>>> >> >> suggest e.g. the following phrasing:
>> >>>> >> >>
>> >>>> >> >> Apache Beam is considering dropping support for Java 7,
>> >>>> >> >> and
>> >>>> >> >> supporting
>> >>>> >> >> only Java 8 and above in a subsequent release. How
>> would
>> >>>> >> >> it
>> >>>> >> >> impact
>> >>>> >> >> your
>> >>>> >> >> usage of Beam?
>> >>>> >> >> - I am already using only Java 8+ for building my Beam
>> >>>> >> >> code
>> >>>> >> >> - I am using Java 7 for building my Beam code, but I
>> would
>> >>>> >> >> have
>> >>>> >> >> no
>> >>>> >> >> trouble switching to Java 8
>> >>>> >> >> - I am using Java 7 for building my Beam code, and
>> >>>> >> >> dropping
>> >>>> >> >> Java 7
>> >>>> >> >> would
>> >>>> >> >> be a blocker or hindrance to adopting the new release
>> for
>> >>>> >> >> me
>> >>>> >> >>
>> >>>> >> >> We could tweet this poll on Apache Beam twitter and
>> >>>> >> >> publish on
>> >>>> >> >> user@,
>> >>>> >> >> and, say, if we receive 5% or fewer votes for option 3
>> >>>> >> >> after
>> >>>> >> >> keeping it
>> >>>> >> >> open for 2 weeks, then adopt Java 8 without a major
>> >>>> >> >> version
>> >>>> >> >> change.
>> >>>> >> >>
>> >>>> >> >> WDYT?
>> >>>> >> >>
>> >>>> >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré
>> >>>> >> >> <jb@nanthrax.net
>> >>>> >> >> <ma...@nanthrax.net>> wrote:
>> >>>> >> >>
>> >>>> >> >> Good idea ! Definitely +1
>> >>>> >> >>
>> >>>> >> >> Regards
>> >>>> >> >> JB
>> >>>> >> >>
>> >>>> >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote:
>> >>>> >> >> > We should bring this up on the Beam 3.0 thread.
>> >>>> >> >> Since
>> >>>> >> >> it's
>> >>>> >> >> technically a
>> >>>> >> >> > backwards-incompatible change, it might make a
>> good
>> >>>> >> >> item
>> >>>> >> >> for Beam
>> >>>> >> >> 3.0.
>> >>>> >> >> >
>> >>>> >> >> > Reuven
>> >>>> >> >> >
>> >>>> >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste
>> >>>> >> >> Onofré
>> >>>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
>> >>>> >> >> > <mailto:jb@nanthrax.net <mailto:jb@nanthrax.net
>> >>>
>> >>>> >> >> wrote:
>> >>>> >> >> >
>> >>>> >> >> > My apologizes, I thought we had a consensus
>> >>>> >> >> already.
>> >>>> >> >> >
>> >>>> >> >> > Regards
>> >>>> >> >> > JB
>> >>>> >> >> >
>> >>>> >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov
>> wrote:
>> >>>> >> >> >
>> >>>> >> >> > Thanks JB for sending the detailed notes
>> >>>> >> >> about
>> >>>> >> >> new
>> >>>> >> >> stuff
>> >>>> >> >> in 2.2.0! A lot
>> >>>> >> >> > of exciting things indeed.
>> >>>> >> >> >
>> >>>> >> >> > Regarding Java 8: I thought our
>> consensus
>> >>>> >> >> was to
>> >>>> >> >> have the
>> >>>> >> >> release notes
>> >>>> >> >> > say that we're *considering* going
>> >>>> >> >> Java8-only,
>> >>>> >> >> and
>> >>>> >> >> use
>> >>>> >> >> that to get more
>> >>>> >> >> > opinions from the user community - but I
>> >>>> >> >> can't
>> >>>> >> >> find
>> >>>> >> >> the
>> >>>> >> >> emails that made
>> >>>> >> >> > me think so.
>> >>>> >> >> >
>> >>>> >> >> > +Ismaël Mejía <mailto:iemejia@gmail.com
>> >>>> >> >> <ma...@gmail.com> <mailto:
>> iemejia@gmail.com
>> >>>> >> >> <ma...@gmail.com>>> - do
>> >>>> >> >> > you think we should formally conclude
>> the
>> >>>> >> >> vote
>> >>>> >> >> on
>> >>>> >> >> the
>> >>>> >> >> thread [VOTE]
>> >>>> >> >> > [DISCUSSION] Remove support for Java 7?
>> >>>> >> >> > Or should we take more steps - e.g.
>> perhaps
>> >>>> >> >> tweet a
>> >>>> >> >> link
>> >>>> >> >> to that thread
>> >>>> >> >> > from the Beam twitter account, ask
>> people
>> >>>> >> >> to
>> >>>> >> >> chime
>> >>>> >> >> in,
>> >>>> >> >> and wait for say
>> >>>> >> >> > 2 weeks before declaring a conclusion?
>> >>>> >> >> >
>> >>>> >> >> > Let's also have a process JIRA for going
>> >>>> >> >> Java8.
>> >>>> >> >> I've
>> >>>> >> >> filed one:
>> >>>> >> >> > https://issues.apache.org/jira/browse/BEAM-3285
>> >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>
>> >>>> >> >> >
>> >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285
>> >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>>
>> >>>> >> >> >
>> >>>> >> >> > On Mon, Dec 4, 2017 at 1:58 AM
>> >>>> >> >> Jean-Baptiste
>> >>>> >> >> Onofré
>> >>>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
>> >>>> >> >> > <mailto:jb@nanthrax.net
>> >>>> >> >> <ma...@nanthrax.net>>
>> >>>> >> >> <mailto:jb@nanthrax.net <ma...@nanthrax.net>
>> >>>> >> >> > <mailto:jb@nanthrax.net
>> >>>> >> >> <ma...@nanthrax.net>>>>
>> >>>> >> >> wrote:
>> >>>> >> >> >
>> >>>> >> >> > Just an important note that we
>> forgot
>> >>>> >> >> to
>> >>>> >> >> mention.
>> >>>> >> >> >
>> >>>> >> >> > !! The 2.2.0 release will be the
>> last
>> >>>> >> >> one
>> >>>> >> >> supporting
>> >>>> >> >> Spark 1.x and
>> >>>> >> >> > Java 7 !!
>> >>>> >> >> >
>> >>>> >> >> > Starting from Beam 2.3.0, the Spark
>> >>>> >> >> runner
>> >>>> >> >> will work
>> >>>> >> >> only with
>> >>>> >> >> > Spark 2.x and we
>> >>>> >> >> > will focus only Java 8.
>> >>>> >> >> >
>> >>>> >> >> > Regards
>> >>>> >> >> > JB
>> >>>> >> >> >
>> >>>> >> >> > On 12/04/2017 10:15 AM,
>> Jean-Baptiste
>> >>>> >> >> Onofré
>> >>>> >> >> wrote:
>> >>>> >> >> > > Thanks Reuven !
>> >>>> >> >> > >
>> >>>> >> >> > > I would like to emphasize on
>> some
>> >>>> >> >> highlights in
>> >>>> >> >> 2.2.0 release:
>> >>>> >> >> > >
>> >>>> >> >> > > - New IOs have been introduced:
>> >>>> >> >> > > * TikaIO leveraging Apache
>> Tika,
>> >>>> >> >> allowing
>> >>>> >> >> the
>> >>>> >> >> deal with a lot
>> >>>> >> >> > of different
>> >>>> >> >> > > data formats
>> >>>> >> >> > > * RedisIO to read and write
>> >>>> >> >> key/value
>> >>>> >> >> pairs
>> >>>> >> >> from a Redis
>> >>>> >> >> > server. This
>> >>>> >> >> > IO will
>> >>>> >> >> > > be soon extended to Redis
>> PubSub.
>> >>>> >> >> > > * FileIO provides transforms
>> for
>> >>>> >> >> working
>> >>>> >> >> with
>> >>>> >> >> files (raw).
>> >>>> >> >> > Especially, it
>> >>>> >> >> > > provides matching file patterns
>> and
>> >>>> >> >> read
>> >>>> >> >> on
>> >>>> >> >> patterns. It can be
>> >>>> >> >> > easily
>> >>>> >> >> > extended
>> >>>> >> >> > > for a specific format (like we
>> do
>> >>>> >> >> in
>> >>>> >> >> AvroIO
>> >>>> >> >> or
>> >>>> >> >> TextIO now).
>> >>>> >> >> > > * SolrIO to interact with
>> Apache
>> >>>> >> >> Solr
>> >>>> >> >> (Lucene)
>> >>>> >> >> > >
>> >>>> >> >> > > - On the other hand,
>> improvements
>> >>>> >> >> have
>> >>>> >> >> been
>> >>>> >> >> performed on
>> >>>> >> >> > existing IOs:
>> >>>> >> >> > > * We started to introduce
>> readAll
>> >>>> >> >> pattern
>> >>>> >> >> in
>> >>>> >> >> IOs (AvroIO,
>> >>>> >> >> > TextIO, JdbcIO,
>> >>>> >> >> > > ...), allowing to pass "request"
>> >>>> >> >> arguments
>> >>>> >> >> via an
>> >>>> >> >> input PCollection.
>> >>>> >> >> > > * ElasticsearchIO has an
>> improved
>> >>>> >> >> support
>> >>>> >> >> of
>> >>>> >> >> different
>> >>>> >> >> > Elasticsearch
>> >>>> >> >> > version
>> >>>> >> >> > > (including Elasticsearch 5.x).
>> It
>> >>>> >> >> also
>> >>>> >> >> now
>> >>>> >> >> supports SSL/TLS.
>> >>>> >> >> > > * HBaseIO is now able to do
>> >>>> >> >> dynamic
>> >>>> >> >> work
>> >>>> >> >> rebalancing
>> >>>> >> >> > > * KinesisIO uses a more
>> accurate
>> >>>> >> >> watermark
>> >>>> >> >> (based on
>> >>>> >> >> > approximateArrivalTimestamp)
>> >>>> >> >> > > * TextIO now supports custom
>> >>>> >> >> delimiter
>> >>>> >> >> and like
>> >>>> >> >> AvroIO,
>> >>>> >> >> > supports the
>> >>>> >> >> > readAll
>> >>>> >> >> > > pattern,
>> >>>> >> >> > > * Performance improvements on
>> >>>> >> >> JdbcIO
>> >>>> >> >> when
>> >>>> >> >> it
>> >>>> >> >> has to read lot
>> >>>> >> >> > of rows
>> >>>> >> >> > > * Kafka write supports
>> >>>> >> >> Exactly-Once
>> >>>> >> >> pattern
>> >>>> >> >> (introduce in
>> >>>> >> >> > Kafka 0.11.x)
>> >>>> >> >> > >
>> >>>> >> >> > > - A new DSL has been introduced:
>> >>>> >> >> the SQL
>> >>>> >> >> DSL !
>> >>>> >> >> > >
>> >>>> >> >> > > We are now focus on 2.3.0
>> release
>> >>>> >> >> with
>> >>>> >> >> new
>> >>>> >> >> improvements and
>> >>>> >> >> > features !
>> >>>> >> >> > >
>> >>>> >> >> > > Stay tuned !
>> >>>> >> >> > >
>> >>>> >> >> > > JB on behalf of the Apache Beam
>> >>>> >> >> community.
>> >>>> >> >> > >
>> >>>> >> >> > > On 12/02/2017 11:40 PM, Reuven
>> Lax
>> >>>> >> >> wrote:
>> >>>> >> >> > >> The Apache Beam community is
>> >>>> >> >> pleased to
>> >>>> >> >> announce the
>> >>>> >> >> > availability of the
>> >>>> >> >> > >> 2.2.0 release.
>> >>>> >> >> > >>
>> >>>> >> >> > >> This release adds support for
>> >>>> >> >> generic
>> >>>> >> >> file
>> >>>> >> >> sources and sinks
>> >>>> >> >> > (beyond TextIO
>> >>>> >> >> > >> and AvroIO) using FileIO,
>> >>>> >> >> including
>> >>>> >> >> support for
>> >>>> >> >> dynamic
>> >>>> >> >> > filenames using
>> >>>> >> >> > >> readAll; this allows streaming
>> >>>> >> >> pipelines
>> >>>> >> >> to now
>> >>>> >> >> read from files by
>> >>>> >> >> > >> continuously monitoring a
>> >>>> >> >> directory for
>> >>>> >> >> new
>> >>>> >> >> filw. Many other
>> >>>> >> >> > IOs are
>> >>>> >> >> > improved,
>> >>>> >> >> > >> notably including exactly-once
>> >>>> >> >> support
>> >>>> >> >> for
>> >>>> >> >> the
>> >>>> >> >> Kafka sink. Initial
>> >>>> >> >> > support for
>> >>>> >> >> > >> BEAM-SQL is also included in
>> this
>> >>>> >> >> release.
>> >>>> >> >> For a
>> >>>> >> >> more-complete
>> >>>> >> >> > list of major
>> >>>> >> >> > >> changes in the release, please
>> >>>> >> >> refer to
>> >>>> >> >> the
>> >>>> >> >> release notes [2].
>> >>>> >> >> > >>
>> >>>> >> >> > >> The 2.2.0 release is now the
>> >>>> >> >> recommended
>> >>>> >> >> version; we encourage
>> >>>> >> >> > everyone to
>> >>>> >> >> > >> upgrade from any earlier
>> releases.
>> >>>> >> >> > >>
>> >>>> >> >> > >> We’d like to invite everyone to
>> >>>> >> >> try out
>> >>>> >> >> Apache
>> >>>> >> >> Beam today and
>> >>>> >> >> > consider
>> >>>> >> >> > >> joining our vibrant community.
>> We
>> >>>> >> >> welcome
>> >>>> >> >> feedback,
>> >>>> >> >> > contribution and
>> >>>> >> >> > >> participation through our
>> mailing
>> >>>> >> >> lists,
>> >>>> >> >> issue
>> >>>> >> >> tracker, pull
>> >>>> >> >> > requests, and
>> >>>> >> >> > >> events.
>> >>>> >> >> > >>
>> >>>> >> >> > >> - Reuven Lax, on behalf of the
>> >>>> >> >> Apache
>> >>>> >> >> Beam
>> >>>> >> >> community.
>> >>>> >> >> > >>
>> >>>> >> >> > >> [1]
>> >>>> >> >> https://beam.apache.org/get-started/downloads/
>> >>>> >> >> <https://beam.apache.org/get-started/downloads/>
>> >>>> >> >> >
>> >>>> >> >> <https://beam.apache.org/get-started/downloads/
>> >>>> >> >> <https://beam.apache.org/get-started/downloads/>>
>> >>>> >> >> > >> [2]
>> >>>> >> >> > >>
>> >>>> >> >> >
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>> ctId=12319527&version=12341044
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?proj
>> ectId=12319527&version=12341044>
>> >>>> >> >> >
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?proj
>> ectId=12319527&version=12341044
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?proj
>> ectId=12319527&version=12341044>>
>> >>>> >> >> > >>
>> >>>> >> >> > >
>> >>>> >> >> >
>> >>>> >> >> > --
>> >>>> >> >> > Jean-Baptiste Onofré
>> >>>> >> >> > jbonofre@apache.org <mailto:jbonofre@apache.org
>> >
>> >>>> >> >> <mailto:jbonofre@apache.org
>> >>>> >> >> <ma...@apache.org>>
>> >>>> >> >> > <mailto:jbonofre@apache.org
>> >>>> >> >> <ma...@apache.org>
>> >>>> >> >> <mailto:jbonofre@apache.org
>> >>>> >> >> <ma...@apache.org>>>
>> >>>> >> >> > http://blog.nanthrax.net
>> >>>> >> >> > Talend - http://www.talend.com
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> > --
>> >>>> >> >> > Jean-Baptiste Onofré
>> >>>> >> >> > jbonofre@apache.org <mailto:jbonofre@apache.org
>> >
>> >>>> >> >> <mailto:jbonofre@apache.org
>> >>>> >> >> <ma...@apache.org>>
>> >>>> >> >> > http://blog.nanthrax.net
>> >>>> >> >> > Talend - http://www.talend.com
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >>
>> >>>> >> >> --
>> >>>> >> >> Jean-Baptiste Onofré
>> >>>> >> >> jbonofre@apache.org <ma...@apache.org>
>> >>>> >> >> http://blog.nanthrax.net
>> >>>> >> >> Talend - http://www.talend.com
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >
>> >>>> >> > --
>> >>>> >> > Jean-Baptiste Onofré
>> >>>> >> > jbonofre@apache.org
>> >>>> >> > http://blog.nanthrax.net
>> >>>> >> > Talend - http://www.talend.com
>> >>>
>> >>>
>> >
>>
>
>
Re: Apache Beam, version 2.2.0
Posted by Steve Niemitz <sn...@apache.org>.
Sorry for resurrecting this thread, but I was trying to build 2.2.0 from
source today and noticed there was no v2.2.0 tag (only v2.2.0-RC4). I
assume that's not intentional?
On Fri, Dec 8, 2017 at 3:35 AM, Ismaël Mejía <ie...@gmail.com> wrote:
> Thanks Eugene for opening the poll (sorry if I didn't before I was
> quite busy in the last two days but expected to do it today).
>
>
> On Fri, Dec 8, 2017 at 1:27 AM, Ahmet Altay <al...@google.com> wrote:
> >
> > On Thu, Dec 7, 2017 at 3:51 PM, Eugene Kirpichov <ki...@google.com>
> > wrote:
> >>
> >> I've sent the poll
> >> https://lists.apache.org/thread.html/5bc2e184a24de9dbc8184ffd2720d1
> 894010497d47d956b395e037df@%3Cuser.beam.apache.org%3E
> >> Will figure out how to tweet from @ApacheBeam, and sent the Twitter poll
> >> as well (or ask someone to).
> >
> >
> > I tweeted the poll.
> >
> >>
> >>
> >> On Wed, Dec 6, 2017 at 1:47 PM Lukasz Cwik <lc...@google.com> wrote:
> >>>
> >>> +1 on moving forward with the plan suggested by kirpichov@
> >>>
> >>> On Wed, Dec 6, 2017 at 9:14 AM, Robert Bradshaw <ro...@google.com>
> >>> wrote:
> >>>>
> >>>> +1 to moving forward with this plan.
> >>>>
> >>>> (FWIW, this seems *less* backwards incompatible than, say, moving from
> >>>> Spark 1 to Spark 2, which was decided much quicker. I suppose the
> >>>> Spark change has a lower bound on the number of users it could impact
> >>>> though.)
> >>>>
> >>>> On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <
> kirpichov@google.com>
> >>>> wrote:
> >>>> > Okay, then let's go forward. Seems that we should:
> >>>> > - Open a new poll on user@, in light of 2.2 having been released
> >>>> > - Open a twitter poll
> >>>> > - Tweet that there's also a poll going on on user@
> >>>> > - Runner authors will reach out to respective runner user
> communities
> >>>> > - 2 weeks later we gather results and decide
> >>>> > ?
> >>>> >
> >>>> > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <ie...@gmail.com>
> wrote:
> >>>> >>
> >>>> >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far
> away,
> >>>> >> and starting to improve Beam to offer a Java 8 friendly experience
> >>>> >> seems like an excellent idea.
> >>>> >>
> >>>> >> I understand the backwards compatibility argument. We should do the
> >>>> >> poll in twitter + try to reach more users for comments. If you
> >>>> >> consider that it is worth, I can open a second poll at user@.
> >>>> >>
> >>>> >> In any case we should try to move forward, even if we have more
> than
> >>>> >> 5% of users who want to stay on Java 7 we can consider to maintain
> >>>> >> minor releases of a backwards compatible version where we can
> >>>> >> backport
> >>>> >> only critical fixes e.g. security/data related errors but nothing
> >>>> >> new,
> >>>> >> in case some user really needs to have them. Of course this can be
> >>>> >> some extra work (to be discussed).
> >>>> >>
> >>>> >>
> >>>> >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré
> >>>> >> <jb...@nanthrax.net>
> >>>> >> wrote:
> >>>> >> > +1, and sorry again, I thought we got an consensus.
> >>>> >> >
> >>>> >> > Regards
> >>>> >> > JB
> >>>> >> >
> >>>> >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote:
> >>>> >> >>
> >>>> >> >> +1 to the poll and also to Reuven's point.
> >>>> >> >>
> >>>> >> >> Those without a support contract would have been using JDK 7
> >>>> >> >> without
> >>>> >> >> security updates for years. IMO it seems harmful, as a netizen,
> to
> >>>> >> >> encourage
> >>>> >> >> its use/existence.
> >>>> >> >>
> >>>> >> >> If there's no noise from the prior thread, then I would assume
> no
> >>>> >> >> one
> >>>> >> >> on
> >>>> >> >> user@ has any objection. Anyone else with customers should
> reach
> >>>> >> >> out to
> >>>> >> >> them.
> >>>> >> >>
> >>>> >> >> Kenn
> >>>> >> >>
> >>>> >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <relax@google.com
> >>>> >> >> <ma...@google.com>> wrote:
> >>>> >> >>
> >>>> >> >> Technically it's a backwards-incompatible change, however if
> >>>> >> >> we are
> >>>> >> >> convinced the risk is low we could do it.
> >>>> >> >>
> >>>> >> >> As mentioned on the original thread, it's not clear that all
> >>>> >> >> Beam
> >>>> >> >> users read
> >>>> >> >> user@ - e.g. most Dataflow users definitely do not. I
> think we
> >>>> >> >> need
> >>>> >> >> to
> >>>> >> >> separately reach out to users of each runner through
> >>>> >> >> runner-specific
> >>>> >> >> channels.
> >>>> >> >>
> >>>> >> >> Reuven
> >>>> >> >>
> >>>> >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov
> >>>> >> >> <kirpichov@google.com
> >>>> >> >> <ma...@google.com>> wrote:
> >>>> >> >>
> >>>> >> >> On the original thread
> >>>> >> >>
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> https://lists.apache.org/thread.html/
> 2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%
> 3Cuser.beam.apache.org%3E
> >>>> >> >>
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> <https://lists.apache.org/thread.html/
> 2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%
> 3Cuser.beam.apache.org%3E>
> >>>> >> >> ,
> >>>> >> >> Robert and Ismaël were in favor of no major version
> change
> >>>> >> >> [Ismaël
> >>>> >> >> said:/Also I am afraid that if we wait/
> >>>> >> >> /until we have enough changes to switch Beam to a new
> >>>> >> >> major
> >>>> >> >> version the
> >>>> >> >> switch to Java 8 will happen too late, probably after
> Java
> >>>> >> >> 8's
> >>>> >> >> end
> >>>> >> >> of
> >>>> >> >> life. And I am not exaggerating, Java 8 is planned to
> EOL
> >>>> >> >> next
> >>>> >> >> march
> >>>> >> >> 2018!/]; JB and now Reuven are in favor of a major
> version
> >>>> >> >> change;
> >>>> >> >>
> >>>> >> >> nobody so far argued against switching to Java8 in
> >>>> >> >> general.
> >>>> >> >>
> >>>> >> >> I'm personally in favor of no major version change (i.e.
> >>>> >> >> not
> >>>> >> >> waiting
> >>>> >> >> until all other large changes for Beam 3.0 converge,
> which
> >>>> >> >> will
> >>>> >> >> likely
> >>>> >> >> be many months), because:
> >>>> >> >> - Reasons Ismaël cited; plus the reason that most people
> >>>> >> >> are
> >>>> >> >> likely
> >>>> >> >> already using Java 8.
> >>>> >> >> - Going Java8-only earlier will make other Beam 3.0 APIs
> >>>> >> >> better
> >>>> >> >> for
> >>>> >> >> Java8 users, because we (Beam contributors) will have
> >>>> >> >> experience
> >>>> >> >> working
> >>>> >> >> with them within the SDK in Java8 (e.g. writing tests
> with
> >>>> >> >> use
> >>>> >> >> of
> >>>> >> >> lambdas and noticing whether it's clunky, or whether
> some
> >>>> >> >> other
> >>>> >> >> Beam
> >>>> >> >> APIs need better Java8 support).
> >>>> >> >> - Going Java8 will make it more reasonable to include
> >>>> >> >> (mostly
> >>>> >> >> or
> >>>> >> >> only)
> >>>> >> >> Java8 snippets in Beam documentation, which will
> obviously
> >>>> >> >> look
> >>>> >> >> more
> >>>> >> >> concise and attractive, addressing one of the common
> >>>> >> >> concerns
> >>>> >> >> of
> >>>> >> >> Beam
> >>>> >> >> users that it has a heavyweight API compared to
> >>>> >> >> functional-style
> >>>> >> >> APIs of
> >>>> >> >> Spark etc.
> >>>> >> >>
> >>>> >> >> I think resolving this via a poll of users would be
> >>>> >> >> reasonable.
> >>>> >> >> I'd
> >>>> >> >> suggest e.g. the following phrasing:
> >>>> >> >>
> >>>> >> >> Apache Beam is considering dropping support for Java 7,
> >>>> >> >> and
> >>>> >> >> supporting
> >>>> >> >> only Java 8 and above in a subsequent release. How would
> >>>> >> >> it
> >>>> >> >> impact
> >>>> >> >> your
> >>>> >> >> usage of Beam?
> >>>> >> >> - I am already using only Java 8+ for building my Beam
> >>>> >> >> code
> >>>> >> >> - I am using Java 7 for building my Beam code, but I
> would
> >>>> >> >> have
> >>>> >> >> no
> >>>> >> >> trouble switching to Java 8
> >>>> >> >> - I am using Java 7 for building my Beam code, and
> >>>> >> >> dropping
> >>>> >> >> Java 7
> >>>> >> >> would
> >>>> >> >> be a blocker or hindrance to adopting the new release
> for
> >>>> >> >> me
> >>>> >> >>
> >>>> >> >> We could tweet this poll on Apache Beam twitter and
> >>>> >> >> publish on
> >>>> >> >> user@,
> >>>> >> >> and, say, if we receive 5% or fewer votes for option 3
> >>>> >> >> after
> >>>> >> >> keeping it
> >>>> >> >> open for 2 weeks, then adopt Java 8 without a major
> >>>> >> >> version
> >>>> >> >> change.
> >>>> >> >>
> >>>> >> >> WDYT?
> >>>> >> >>
> >>>> >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré
> >>>> >> >> <jb@nanthrax.net
> >>>> >> >> <ma...@nanthrax.net>> wrote:
> >>>> >> >>
> >>>> >> >> Good idea ! Definitely +1
> >>>> >> >>
> >>>> >> >> Regards
> >>>> >> >> JB
> >>>> >> >>
> >>>> >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote:
> >>>> >> >> > We should bring this up on the Beam 3.0 thread.
> >>>> >> >> Since
> >>>> >> >> it's
> >>>> >> >> technically a
> >>>> >> >> > backwards-incompatible change, it might make a
> good
> >>>> >> >> item
> >>>> >> >> for Beam
> >>>> >> >> 3.0.
> >>>> >> >> >
> >>>> >> >> > Reuven
> >>>> >> >> >
> >>>> >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste
> >>>> >> >> Onofré
> >>>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
> >>>> >> >> > <mailto:jb@nanthrax.net <mailto:jb@nanthrax.net
> >>>
> >>>> >> >> wrote:
> >>>> >> >> >
> >>>> >> >> > My apologizes, I thought we had a consensus
> >>>> >> >> already.
> >>>> >> >> >
> >>>> >> >> > Regards
> >>>> >> >> > JB
> >>>> >> >> >
> >>>> >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov
> wrote:
> >>>> >> >> >
> >>>> >> >> > Thanks JB for sending the detailed notes
> >>>> >> >> about
> >>>> >> >> new
> >>>> >> >> stuff
> >>>> >> >> in 2.2.0! A lot
> >>>> >> >> > of exciting things indeed.
> >>>> >> >> >
> >>>> >> >> > Regarding Java 8: I thought our consensus
> >>>> >> >> was to
> >>>> >> >> have the
> >>>> >> >> release notes
> >>>> >> >> > say that we're *considering* going
> >>>> >> >> Java8-only,
> >>>> >> >> and
> >>>> >> >> use
> >>>> >> >> that to get more
> >>>> >> >> > opinions from the user community - but I
> >>>> >> >> can't
> >>>> >> >> find
> >>>> >> >> the
> >>>> >> >> emails that made
> >>>> >> >> > me think so.
> >>>> >> >> >
> >>>> >> >> > +Ismaël Mejía <mailto:iemejia@gmail.com
> >>>> >> >> <ma...@gmail.com> <mailto:
> iemejia@gmail.com
> >>>> >> >> <ma...@gmail.com>>> - do
> >>>> >> >> > you think we should formally conclude the
> >>>> >> >> vote
> >>>> >> >> on
> >>>> >> >> the
> >>>> >> >> thread [VOTE]
> >>>> >> >> > [DISCUSSION] Remove support for Java 7?
> >>>> >> >> > Or should we take more steps - e.g.
> perhaps
> >>>> >> >> tweet a
> >>>> >> >> link
> >>>> >> >> to that thread
> >>>> >> >> > from the Beam twitter account, ask people
> >>>> >> >> to
> >>>> >> >> chime
> >>>> >> >> in,
> >>>> >> >> and wait for say
> >>>> >> >> > 2 weeks before declaring a conclusion?
> >>>> >> >> >
> >>>> >> >> > Let's also have a process JIRA for going
> >>>> >> >> Java8.
> >>>> >> >> I've
> >>>> >> >> filed one:
> >>>> >> >> > https://issues.apache.org/jira/browse/BEAM-3285
> >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>
> >>>> >> >> >
> >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285
> >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>>
> >>>> >> >> >
> >>>> >> >> > On Mon, Dec 4, 2017 at 1:58 AM
> >>>> >> >> Jean-Baptiste
> >>>> >> >> Onofré
> >>>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
> >>>> >> >> > <mailto:jb@nanthrax.net
> >>>> >> >> <ma...@nanthrax.net>>
> >>>> >> >> <mailto:jb@nanthrax.net <ma...@nanthrax.net>
> >>>> >> >> > <mailto:jb@nanthrax.net
> >>>> >> >> <ma...@nanthrax.net>>>>
> >>>> >> >> wrote:
> >>>> >> >> >
> >>>> >> >> > Just an important note that we
> forgot
> >>>> >> >> to
> >>>> >> >> mention.
> >>>> >> >> >
> >>>> >> >> > !! The 2.2.0 release will be the
> last
> >>>> >> >> one
> >>>> >> >> supporting
> >>>> >> >> Spark 1.x and
> >>>> >> >> > Java 7 !!
> >>>> >> >> >
> >>>> >> >> > Starting from Beam 2.3.0, the Spark
> >>>> >> >> runner
> >>>> >> >> will work
> >>>> >> >> only with
> >>>> >> >> > Spark 2.x and we
> >>>> >> >> > will focus only Java 8.
> >>>> >> >> >
> >>>> >> >> > Regards
> >>>> >> >> > JB
> >>>> >> >> >
> >>>> >> >> > On 12/04/2017 10:15 AM,
> Jean-Baptiste
> >>>> >> >> Onofré
> >>>> >> >> wrote:
> >>>> >> >> > > Thanks Reuven !
> >>>> >> >> > >
> >>>> >> >> > > I would like to emphasize on some
> >>>> >> >> highlights in
> >>>> >> >> 2.2.0 release:
> >>>> >> >> > >
> >>>> >> >> > > - New IOs have been introduced:
> >>>> >> >> > > * TikaIO leveraging Apache
> Tika,
> >>>> >> >> allowing
> >>>> >> >> the
> >>>> >> >> deal with a lot
> >>>> >> >> > of different
> >>>> >> >> > > data formats
> >>>> >> >> > > * RedisIO to read and write
> >>>> >> >> key/value
> >>>> >> >> pairs
> >>>> >> >> from a Redis
> >>>> >> >> > server. This
> >>>> >> >> > IO will
> >>>> >> >> > > be soon extended to Redis PubSub.
> >>>> >> >> > > * FileIO provides transforms
> for
> >>>> >> >> working
> >>>> >> >> with
> >>>> >> >> files (raw).
> >>>> >> >> > Especially, it
> >>>> >> >> > > provides matching file patterns
> and
> >>>> >> >> read
> >>>> >> >> on
> >>>> >> >> patterns. It can be
> >>>> >> >> > easily
> >>>> >> >> > extended
> >>>> >> >> > > for a specific format (like we do
> >>>> >> >> in
> >>>> >> >> AvroIO
> >>>> >> >> or
> >>>> >> >> TextIO now).
> >>>> >> >> > > * SolrIO to interact with
> Apache
> >>>> >> >> Solr
> >>>> >> >> (Lucene)
> >>>> >> >> > >
> >>>> >> >> > > - On the other hand, improvements
> >>>> >> >> have
> >>>> >> >> been
> >>>> >> >> performed on
> >>>> >> >> > existing IOs:
> >>>> >> >> > > * We started to introduce
> readAll
> >>>> >> >> pattern
> >>>> >> >> in
> >>>> >> >> IOs (AvroIO,
> >>>> >> >> > TextIO, JdbcIO,
> >>>> >> >> > > ...), allowing to pass "request"
> >>>> >> >> arguments
> >>>> >> >> via an
> >>>> >> >> input PCollection.
> >>>> >> >> > > * ElasticsearchIO has an
> improved
> >>>> >> >> support
> >>>> >> >> of
> >>>> >> >> different
> >>>> >> >> > Elasticsearch
> >>>> >> >> > version
> >>>> >> >> > > (including Elasticsearch 5.x). It
> >>>> >> >> also
> >>>> >> >> now
> >>>> >> >> supports SSL/TLS.
> >>>> >> >> > > * HBaseIO is now able to do
> >>>> >> >> dynamic
> >>>> >> >> work
> >>>> >> >> rebalancing
> >>>> >> >> > > * KinesisIO uses a more
> accurate
> >>>> >> >> watermark
> >>>> >> >> (based on
> >>>> >> >> > approximateArrivalTimestamp)
> >>>> >> >> > > * TextIO now supports custom
> >>>> >> >> delimiter
> >>>> >> >> and like
> >>>> >> >> AvroIO,
> >>>> >> >> > supports the
> >>>> >> >> > readAll
> >>>> >> >> > > pattern,
> >>>> >> >> > > * Performance improvements on
> >>>> >> >> JdbcIO
> >>>> >> >> when
> >>>> >> >> it
> >>>> >> >> has to read lot
> >>>> >> >> > of rows
> >>>> >> >> > > * Kafka write supports
> >>>> >> >> Exactly-Once
> >>>> >> >> pattern
> >>>> >> >> (introduce in
> >>>> >> >> > Kafka 0.11.x)
> >>>> >> >> > >
> >>>> >> >> > > - A new DSL has been introduced:
> >>>> >> >> the SQL
> >>>> >> >> DSL !
> >>>> >> >> > >
> >>>> >> >> > > We are now focus on 2.3.0 release
> >>>> >> >> with
> >>>> >> >> new
> >>>> >> >> improvements and
> >>>> >> >> > features !
> >>>> >> >> > >
> >>>> >> >> > > Stay tuned !
> >>>> >> >> > >
> >>>> >> >> > > JB on behalf of the Apache Beam
> >>>> >> >> community.
> >>>> >> >> > >
> >>>> >> >> > > On 12/02/2017 11:40 PM, Reuven
> Lax
> >>>> >> >> wrote:
> >>>> >> >> > >> The Apache Beam community is
> >>>> >> >> pleased to
> >>>> >> >> announce the
> >>>> >> >> > availability of the
> >>>> >> >> > >> 2.2.0 release.
> >>>> >> >> > >>
> >>>> >> >> > >> This release adds support for
> >>>> >> >> generic
> >>>> >> >> file
> >>>> >> >> sources and sinks
> >>>> >> >> > (beyond TextIO
> >>>> >> >> > >> and AvroIO) using FileIO,
> >>>> >> >> including
> >>>> >> >> support for
> >>>> >> >> dynamic
> >>>> >> >> > filenames using
> >>>> >> >> > >> readAll; this allows streaming
> >>>> >> >> pipelines
> >>>> >> >> to now
> >>>> >> >> read from files by
> >>>> >> >> > >> continuously monitoring a
> >>>> >> >> directory for
> >>>> >> >> new
> >>>> >> >> filw. Many other
> >>>> >> >> > IOs are
> >>>> >> >> > improved,
> >>>> >> >> > >> notably including exactly-once
> >>>> >> >> support
> >>>> >> >> for
> >>>> >> >> the
> >>>> >> >> Kafka sink. Initial
> >>>> >> >> > support for
> >>>> >> >> > >> BEAM-SQL is also included in
> this
> >>>> >> >> release.
> >>>> >> >> For a
> >>>> >> >> more-complete
> >>>> >> >> > list of major
> >>>> >> >> > >> changes in the release, please
> >>>> >> >> refer to
> >>>> >> >> the
> >>>> >> >> release notes [2].
> >>>> >> >> > >>
> >>>> >> >> > >> The 2.2.0 release is now the
> >>>> >> >> recommended
> >>>> >> >> version; we encourage
> >>>> >> >> > everyone to
> >>>> >> >> > >> upgrade from any earlier
> releases.
> >>>> >> >> > >>
> >>>> >> >> > >> We’d like to invite everyone to
> >>>> >> >> try out
> >>>> >> >> Apache
> >>>> >> >> Beam today and
> >>>> >> >> > consider
> >>>> >> >> > >> joining our vibrant community.
> We
> >>>> >> >> welcome
> >>>> >> >> feedback,
> >>>> >> >> > contribution and
> >>>> >> >> > >> participation through our
> mailing
> >>>> >> >> lists,
> >>>> >> >> issue
> >>>> >> >> tracker, pull
> >>>> >> >> > requests, and
> >>>> >> >> > >> events.
> >>>> >> >> > >>
> >>>> >> >> > >> - Reuven Lax, on behalf of the
> >>>> >> >> Apache
> >>>> >> >> Beam
> >>>> >> >> community.
> >>>> >> >> > >>
> >>>> >> >> > >> [1]
> >>>> >> >> https://beam.apache.org/get-started/downloads/
> >>>> >> >> <https://beam.apache.org/get-started/downloads/>
> >>>> >> >> >
> >>>> >> >> <https://beam.apache.org/get-started/downloads/
> >>>> >> >> <https://beam.apache.org/get-started/downloads/>>
> >>>> >> >> > >> [2]
> >>>> >> >> > >>
> >>>> >> >> >
> >>>> >> >>
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527&version=12341044
> >>>> >> >>
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527&version=12341044>
> >>>> >> >> >
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527&version=12341044
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527&version=12341044>>
> >>>> >> >> > >>
> >>>> >> >> > >
> >>>> >> >> >
> >>>> >> >> > --
> >>>> >> >> > Jean-Baptiste Onofré
> >>>> >> >> > jbonofre@apache.org <ma...@apache.org>
> >>>> >> >> <mailto:jbonofre@apache.org
> >>>> >> >> <ma...@apache.org>>
> >>>> >> >> > <mailto:jbonofre@apache.org
> >>>> >> >> <ma...@apache.org>
> >>>> >> >> <mailto:jbonofre@apache.org
> >>>> >> >> <ma...@apache.org>>>
> >>>> >> >> > http://blog.nanthrax.net
> >>>> >> >> > Talend - http://www.talend.com
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> > --
> >>>> >> >> > Jean-Baptiste Onofré
> >>>> >> >> > jbonofre@apache.org <ma...@apache.org>
> >>>> >> >> <mailto:jbonofre@apache.org
> >>>> >> >> <ma...@apache.org>>
> >>>> >> >> > http://blog.nanthrax.net
> >>>> >> >> > Talend - http://www.talend.com
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >>
> >>>> >> >> --
> >>>> >> >> Jean-Baptiste Onofré
> >>>> >> >> jbonofre@apache.org <ma...@apache.org>
> >>>> >> >> http://blog.nanthrax.net
> >>>> >> >> Talend - http://www.talend.com
> >>>> >> >>
> >>>> >> >>
> >>>> >> >>
> >>>> >> >
> >>>> >> > --
> >>>> >> > Jean-Baptiste Onofré
> >>>> >> > jbonofre@apache.org
> >>>> >> > http://blog.nanthrax.net
> >>>> >> > Talend - http://www.talend.com
> >>>
> >>>
> >
>
Re: Apache Beam, version 2.2.0
Posted by Ismaël Mejía <ie...@gmail.com>.
Thanks Eugene for opening the poll (sorry if I didn't before I was
quite busy in the last two days but expected to do it today).
On Fri, Dec 8, 2017 at 1:27 AM, Ahmet Altay <al...@google.com> wrote:
>
> On Thu, Dec 7, 2017 at 3:51 PM, Eugene Kirpichov <ki...@google.com>
> wrote:
>>
>> I've sent the poll
>> https://lists.apache.org/thread.html/5bc2e184a24de9dbc8184ffd2720d1894010497d47d956b395e037df@%3Cuser.beam.apache.org%3E
>> Will figure out how to tweet from @ApacheBeam, and sent the Twitter poll
>> as well (or ask someone to).
>
>
> I tweeted the poll.
>
>>
>>
>> On Wed, Dec 6, 2017 at 1:47 PM Lukasz Cwik <lc...@google.com> wrote:
>>>
>>> +1 on moving forward with the plan suggested by kirpichov@
>>>
>>> On Wed, Dec 6, 2017 at 9:14 AM, Robert Bradshaw <ro...@google.com>
>>> wrote:
>>>>
>>>> +1 to moving forward with this plan.
>>>>
>>>> (FWIW, this seems *less* backwards incompatible than, say, moving from
>>>> Spark 1 to Spark 2, which was decided much quicker. I suppose the
>>>> Spark change has a lower bound on the number of users it could impact
>>>> though.)
>>>>
>>>> On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <ki...@google.com>
>>>> wrote:
>>>> > Okay, then let's go forward. Seems that we should:
>>>> > - Open a new poll on user@, in light of 2.2 having been released
>>>> > - Open a twitter poll
>>>> > - Tweet that there's also a poll going on on user@
>>>> > - Runner authors will reach out to respective runner user communities
>>>> > - 2 weeks later we gather results and decide
>>>> > ?
>>>> >
>>>> > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>> >>
>>>> >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far away,
>>>> >> and starting to improve Beam to offer a Java 8 friendly experience
>>>> >> seems like an excellent idea.
>>>> >>
>>>> >> I understand the backwards compatibility argument. We should do the
>>>> >> poll in twitter + try to reach more users for comments. If you
>>>> >> consider that it is worth, I can open a second poll at user@.
>>>> >>
>>>> >> In any case we should try to move forward, even if we have more than
>>>> >> 5% of users who want to stay on Java 7 we can consider to maintain
>>>> >> minor releases of a backwards compatible version where we can
>>>> >> backport
>>>> >> only critical fixes e.g. security/data related errors but nothing
>>>> >> new,
>>>> >> in case some user really needs to have them. Of course this can be
>>>> >> some extra work (to be discussed).
>>>> >>
>>>> >>
>>>> >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré
>>>> >> <jb...@nanthrax.net>
>>>> >> wrote:
>>>> >> > +1, and sorry again, I thought we got an consensus.
>>>> >> >
>>>> >> > Regards
>>>> >> > JB
>>>> >> >
>>>> >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote:
>>>> >> >>
>>>> >> >> +1 to the poll and also to Reuven's point.
>>>> >> >>
>>>> >> >> Those without a support contract would have been using JDK 7
>>>> >> >> without
>>>> >> >> security updates for years. IMO it seems harmful, as a netizen, to
>>>> >> >> encourage
>>>> >> >> its use/existence.
>>>> >> >>
>>>> >> >> If there's no noise from the prior thread, then I would assume no
>>>> >> >> one
>>>> >> >> on
>>>> >> >> user@ has any objection. Anyone else with customers should reach
>>>> >> >> out to
>>>> >> >> them.
>>>> >> >>
>>>> >> >> Kenn
>>>> >> >>
>>>> >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <relax@google.com
>>>> >> >> <ma...@google.com>> wrote:
>>>> >> >>
>>>> >> >> Technically it's a backwards-incompatible change, however if
>>>> >> >> we are
>>>> >> >> convinced the risk is low we could do it.
>>>> >> >>
>>>> >> >> As mentioned on the original thread, it's not clear that all
>>>> >> >> Beam
>>>> >> >> users read
>>>> >> >> user@ - e.g. most Dataflow users definitely do not. I think we
>>>> >> >> need
>>>> >> >> to
>>>> >> >> separately reach out to users of each runner through
>>>> >> >> runner-specific
>>>> >> >> channels.
>>>> >> >>
>>>> >> >> Reuven
>>>> >> >>
>>>> >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov
>>>> >> >> <kirpichov@google.com
>>>> >> >> <ma...@google.com>> wrote:
>>>> >> >>
>>>> >> >> On the original thread
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >> <https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E>
>>>> >> >> ,
>>>> >> >> Robert and Ismaël were in favor of no major version change
>>>> >> >> [Ismaël
>>>> >> >> said:/Also I am afraid that if we wait/
>>>> >> >> /until we have enough changes to switch Beam to a new
>>>> >> >> major
>>>> >> >> version the
>>>> >> >> switch to Java 8 will happen too late, probably after Java
>>>> >> >> 8's
>>>> >> >> end
>>>> >> >> of
>>>> >> >> life. And I am not exaggerating, Java 8 is planned to EOL
>>>> >> >> next
>>>> >> >> march
>>>> >> >> 2018!/]; JB and now Reuven are in favor of a major version
>>>> >> >> change;
>>>> >> >>
>>>> >> >> nobody so far argued against switching to Java8 in
>>>> >> >> general.
>>>> >> >>
>>>> >> >> I'm personally in favor of no major version change (i.e.
>>>> >> >> not
>>>> >> >> waiting
>>>> >> >> until all other large changes for Beam 3.0 converge, which
>>>> >> >> will
>>>> >> >> likely
>>>> >> >> be many months), because:
>>>> >> >> - Reasons Ismaël cited; plus the reason that most people
>>>> >> >> are
>>>> >> >> likely
>>>> >> >> already using Java 8.
>>>> >> >> - Going Java8-only earlier will make other Beam 3.0 APIs
>>>> >> >> better
>>>> >> >> for
>>>> >> >> Java8 users, because we (Beam contributors) will have
>>>> >> >> experience
>>>> >> >> working
>>>> >> >> with them within the SDK in Java8 (e.g. writing tests with
>>>> >> >> use
>>>> >> >> of
>>>> >> >> lambdas and noticing whether it's clunky, or whether some
>>>> >> >> other
>>>> >> >> Beam
>>>> >> >> APIs need better Java8 support).
>>>> >> >> - Going Java8 will make it more reasonable to include
>>>> >> >> (mostly
>>>> >> >> or
>>>> >> >> only)
>>>> >> >> Java8 snippets in Beam documentation, which will obviously
>>>> >> >> look
>>>> >> >> more
>>>> >> >> concise and attractive, addressing one of the common
>>>> >> >> concerns
>>>> >> >> of
>>>> >> >> Beam
>>>> >> >> users that it has a heavyweight API compared to
>>>> >> >> functional-style
>>>> >> >> APIs of
>>>> >> >> Spark etc.
>>>> >> >>
>>>> >> >> I think resolving this via a poll of users would be
>>>> >> >> reasonable.
>>>> >> >> I'd
>>>> >> >> suggest e.g. the following phrasing:
>>>> >> >>
>>>> >> >> Apache Beam is considering dropping support for Java 7,
>>>> >> >> and
>>>> >> >> supporting
>>>> >> >> only Java 8 and above in a subsequent release. How would
>>>> >> >> it
>>>> >> >> impact
>>>> >> >> your
>>>> >> >> usage of Beam?
>>>> >> >> - I am already using only Java 8+ for building my Beam
>>>> >> >> code
>>>> >> >> - I am using Java 7 for building my Beam code, but I would
>>>> >> >> have
>>>> >> >> no
>>>> >> >> trouble switching to Java 8
>>>> >> >> - I am using Java 7 for building my Beam code, and
>>>> >> >> dropping
>>>> >> >> Java 7
>>>> >> >> would
>>>> >> >> be a blocker or hindrance to adopting the new release for
>>>> >> >> me
>>>> >> >>
>>>> >> >> We could tweet this poll on Apache Beam twitter and
>>>> >> >> publish on
>>>> >> >> user@,
>>>> >> >> and, say, if we receive 5% or fewer votes for option 3
>>>> >> >> after
>>>> >> >> keeping it
>>>> >> >> open for 2 weeks, then adopt Java 8 without a major
>>>> >> >> version
>>>> >> >> change.
>>>> >> >>
>>>> >> >> WDYT?
>>>> >> >>
>>>> >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré
>>>> >> >> <jb@nanthrax.net
>>>> >> >> <ma...@nanthrax.net>> wrote:
>>>> >> >>
>>>> >> >> Good idea ! Definitely +1
>>>> >> >>
>>>> >> >> Regards
>>>> >> >> JB
>>>> >> >>
>>>> >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote:
>>>> >> >> > We should bring this up on the Beam 3.0 thread.
>>>> >> >> Since
>>>> >> >> it's
>>>> >> >> technically a
>>>> >> >> > backwards-incompatible change, it might make a good
>>>> >> >> item
>>>> >> >> for Beam
>>>> >> >> 3.0.
>>>> >> >> >
>>>> >> >> > Reuven
>>>> >> >> >
>>>> >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste
>>>> >> >> Onofré
>>>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
>>>> >> >> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>>
>>>> >> >> wrote:
>>>> >> >> >
>>>> >> >> > My apologizes, I thought we had a consensus
>>>> >> >> already.
>>>> >> >> >
>>>> >> >> > Regards
>>>> >> >> > JB
>>>> >> >> >
>>>> >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
>>>> >> >> >
>>>> >> >> > Thanks JB for sending the detailed notes
>>>> >> >> about
>>>> >> >> new
>>>> >> >> stuff
>>>> >> >> in 2.2.0! A lot
>>>> >> >> > of exciting things indeed.
>>>> >> >> >
>>>> >> >> > Regarding Java 8: I thought our consensus
>>>> >> >> was to
>>>> >> >> have the
>>>> >> >> release notes
>>>> >> >> > say that we're *considering* going
>>>> >> >> Java8-only,
>>>> >> >> and
>>>> >> >> use
>>>> >> >> that to get more
>>>> >> >> > opinions from the user community - but I
>>>> >> >> can't
>>>> >> >> find
>>>> >> >> the
>>>> >> >> emails that made
>>>> >> >> > me think so.
>>>> >> >> >
>>>> >> >> > +Ismaël Mejía <mailto:iemejia@gmail.com
>>>> >> >> <ma...@gmail.com> <mailto:iemejia@gmail.com
>>>> >> >> <ma...@gmail.com>>> - do
>>>> >> >> > you think we should formally conclude the
>>>> >> >> vote
>>>> >> >> on
>>>> >> >> the
>>>> >> >> thread [VOTE]
>>>> >> >> > [DISCUSSION] Remove support for Java 7?
>>>> >> >> > Or should we take more steps - e.g. perhaps
>>>> >> >> tweet a
>>>> >> >> link
>>>> >> >> to that thread
>>>> >> >> > from the Beam twitter account, ask people
>>>> >> >> to
>>>> >> >> chime
>>>> >> >> in,
>>>> >> >> and wait for say
>>>> >> >> > 2 weeks before declaring a conclusion?
>>>> >> >> >
>>>> >> >> > Let's also have a process JIRA for going
>>>> >> >> Java8.
>>>> >> >> I've
>>>> >> >> filed one:
>>>> >> >> > https://issues.apache.org/jira/browse/BEAM-3285
>>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>
>>>> >> >> >
>>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285
>>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>>
>>>> >> >> >
>>>> >> >> > On Mon, Dec 4, 2017 at 1:58 AM
>>>> >> >> Jean-Baptiste
>>>> >> >> Onofré
>>>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
>>>> >> >> > <mailto:jb@nanthrax.net
>>>> >> >> <ma...@nanthrax.net>>
>>>> >> >> <mailto:jb@nanthrax.net <ma...@nanthrax.net>
>>>> >> >> > <mailto:jb@nanthrax.net
>>>> >> >> <ma...@nanthrax.net>>>>
>>>> >> >> wrote:
>>>> >> >> >
>>>> >> >> > Just an important note that we forgot
>>>> >> >> to
>>>> >> >> mention.
>>>> >> >> >
>>>> >> >> > !! The 2.2.0 release will be the last
>>>> >> >> one
>>>> >> >> supporting
>>>> >> >> Spark 1.x and
>>>> >> >> > Java 7 !!
>>>> >> >> >
>>>> >> >> > Starting from Beam 2.3.0, the Spark
>>>> >> >> runner
>>>> >> >> will work
>>>> >> >> only with
>>>> >> >> > Spark 2.x and we
>>>> >> >> > will focus only Java 8.
>>>> >> >> >
>>>> >> >> > Regards
>>>> >> >> > JB
>>>> >> >> >
>>>> >> >> > On 12/04/2017 10:15 AM, Jean-Baptiste
>>>> >> >> Onofré
>>>> >> >> wrote:
>>>> >> >> > > Thanks Reuven !
>>>> >> >> > >
>>>> >> >> > > I would like to emphasize on some
>>>> >> >> highlights in
>>>> >> >> 2.2.0 release:
>>>> >> >> > >
>>>> >> >> > > - New IOs have been introduced:
>>>> >> >> > > * TikaIO leveraging Apache Tika,
>>>> >> >> allowing
>>>> >> >> the
>>>> >> >> deal with a lot
>>>> >> >> > of different
>>>> >> >> > > data formats
>>>> >> >> > > * RedisIO to read and write
>>>> >> >> key/value
>>>> >> >> pairs
>>>> >> >> from a Redis
>>>> >> >> > server. This
>>>> >> >> > IO will
>>>> >> >> > > be soon extended to Redis PubSub.
>>>> >> >> > > * FileIO provides transforms for
>>>> >> >> working
>>>> >> >> with
>>>> >> >> files (raw).
>>>> >> >> > Especially, it
>>>> >> >> > > provides matching file patterns and
>>>> >> >> read
>>>> >> >> on
>>>> >> >> patterns. It can be
>>>> >> >> > easily
>>>> >> >> > extended
>>>> >> >> > > for a specific format (like we do
>>>> >> >> in
>>>> >> >> AvroIO
>>>> >> >> or
>>>> >> >> TextIO now).
>>>> >> >> > > * SolrIO to interact with Apache
>>>> >> >> Solr
>>>> >> >> (Lucene)
>>>> >> >> > >
>>>> >> >> > > - On the other hand, improvements
>>>> >> >> have
>>>> >> >> been
>>>> >> >> performed on
>>>> >> >> > existing IOs:
>>>> >> >> > > * We started to introduce readAll
>>>> >> >> pattern
>>>> >> >> in
>>>> >> >> IOs (AvroIO,
>>>> >> >> > TextIO, JdbcIO,
>>>> >> >> > > ...), allowing to pass "request"
>>>> >> >> arguments
>>>> >> >> via an
>>>> >> >> input PCollection.
>>>> >> >> > > * ElasticsearchIO has an improved
>>>> >> >> support
>>>> >> >> of
>>>> >> >> different
>>>> >> >> > Elasticsearch
>>>> >> >> > version
>>>> >> >> > > (including Elasticsearch 5.x). It
>>>> >> >> also
>>>> >> >> now
>>>> >> >> supports SSL/TLS.
>>>> >> >> > > * HBaseIO is now able to do
>>>> >> >> dynamic
>>>> >> >> work
>>>> >> >> rebalancing
>>>> >> >> > > * KinesisIO uses a more accurate
>>>> >> >> watermark
>>>> >> >> (based on
>>>> >> >> > approximateArrivalTimestamp)
>>>> >> >> > > * TextIO now supports custom
>>>> >> >> delimiter
>>>> >> >> and like
>>>> >> >> AvroIO,
>>>> >> >> > supports the
>>>> >> >> > readAll
>>>> >> >> > > pattern,
>>>> >> >> > > * Performance improvements on
>>>> >> >> JdbcIO
>>>> >> >> when
>>>> >> >> it
>>>> >> >> has to read lot
>>>> >> >> > of rows
>>>> >> >> > > * Kafka write supports
>>>> >> >> Exactly-Once
>>>> >> >> pattern
>>>> >> >> (introduce in
>>>> >> >> > Kafka 0.11.x)
>>>> >> >> > >
>>>> >> >> > > - A new DSL has been introduced:
>>>> >> >> the SQL
>>>> >> >> DSL !
>>>> >> >> > >
>>>> >> >> > > We are now focus on 2.3.0 release
>>>> >> >> with
>>>> >> >> new
>>>> >> >> improvements and
>>>> >> >> > features !
>>>> >> >> > >
>>>> >> >> > > Stay tuned !
>>>> >> >> > >
>>>> >> >> > > JB on behalf of the Apache Beam
>>>> >> >> community.
>>>> >> >> > >
>>>> >> >> > > On 12/02/2017 11:40 PM, Reuven Lax
>>>> >> >> wrote:
>>>> >> >> > >> The Apache Beam community is
>>>> >> >> pleased to
>>>> >> >> announce the
>>>> >> >> > availability of the
>>>> >> >> > >> 2.2.0 release.
>>>> >> >> > >>
>>>> >> >> > >> This release adds support for
>>>> >> >> generic
>>>> >> >> file
>>>> >> >> sources and sinks
>>>> >> >> > (beyond TextIO
>>>> >> >> > >> and AvroIO) using FileIO,
>>>> >> >> including
>>>> >> >> support for
>>>> >> >> dynamic
>>>> >> >> > filenames using
>>>> >> >> > >> readAll; this allows streaming
>>>> >> >> pipelines
>>>> >> >> to now
>>>> >> >> read from files by
>>>> >> >> > >> continuously monitoring a
>>>> >> >> directory for
>>>> >> >> new
>>>> >> >> filw. Many other
>>>> >> >> > IOs are
>>>> >> >> > improved,
>>>> >> >> > >> notably including exactly-once
>>>> >> >> support
>>>> >> >> for
>>>> >> >> the
>>>> >> >> Kafka sink. Initial
>>>> >> >> > support for
>>>> >> >> > >> BEAM-SQL is also included in this
>>>> >> >> release.
>>>> >> >> For a
>>>> >> >> more-complete
>>>> >> >> > list of major
>>>> >> >> > >> changes in the release, please
>>>> >> >> refer to
>>>> >> >> the
>>>> >> >> release notes [2].
>>>> >> >> > >>
>>>> >> >> > >> The 2.2.0 release is now the
>>>> >> >> recommended
>>>> >> >> version; we encourage
>>>> >> >> > everyone to
>>>> >> >> > >> upgrade from any earlier releases.
>>>> >> >> > >>
>>>> >> >> > >> We’d like to invite everyone to
>>>> >> >> try out
>>>> >> >> Apache
>>>> >> >> Beam today and
>>>> >> >> > consider
>>>> >> >> > >> joining our vibrant community. We
>>>> >> >> welcome
>>>> >> >> feedback,
>>>> >> >> > contribution and
>>>> >> >> > >> participation through our mailing
>>>> >> >> lists,
>>>> >> >> issue
>>>> >> >> tracker, pull
>>>> >> >> > requests, and
>>>> >> >> > >> events.
>>>> >> >> > >>
>>>> >> >> > >> - Reuven Lax, on behalf of the
>>>> >> >> Apache
>>>> >> >> Beam
>>>> >> >> community.
>>>> >> >> > >>
>>>> >> >> > >> [1]
>>>> >> >> https://beam.apache.org/get-started/downloads/
>>>> >> >> <https://beam.apache.org/get-started/downloads/>
>>>> >> >> >
>>>> >> >> <https://beam.apache.org/get-started/downloads/
>>>> >> >> <https://beam.apache.org/get-started/downloads/>>
>>>> >> >> > >> [2]
>>>> >> >> > >>
>>>> >> >> >
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044>
>>>> >> >> >
>>>> >> >>
>>>> >> >>
>>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
>>>> >> >>
>>>> >> >>
>>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044>>
>>>> >> >> > >>
>>>> >> >> > >
>>>> >> >> >
>>>> >> >> > --
>>>> >> >> > Jean-Baptiste Onofré
>>>> >> >> > jbonofre@apache.org <ma...@apache.org>
>>>> >> >> <mailto:jbonofre@apache.org
>>>> >> >> <ma...@apache.org>>
>>>> >> >> > <mailto:jbonofre@apache.org
>>>> >> >> <ma...@apache.org>
>>>> >> >> <mailto:jbonofre@apache.org
>>>> >> >> <ma...@apache.org>>>
>>>> >> >> > http://blog.nanthrax.net
>>>> >> >> > Talend - http://www.talend.com
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > --
>>>> >> >> > Jean-Baptiste Onofré
>>>> >> >> > jbonofre@apache.org <ma...@apache.org>
>>>> >> >> <mailto:jbonofre@apache.org
>>>> >> >> <ma...@apache.org>>
>>>> >> >> > http://blog.nanthrax.net
>>>> >> >> > Talend - http://www.talend.com
>>>> >> >> >
>>>> >> >> >
>>>> >> >>
>>>> >> >> --
>>>> >> >> Jean-Baptiste Onofré
>>>> >> >> jbonofre@apache.org <ma...@apache.org>
>>>> >> >> http://blog.nanthrax.net
>>>> >> >> Talend - http://www.talend.com
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >
>>>> >> > --
>>>> >> > Jean-Baptiste Onofré
>>>> >> > jbonofre@apache.org
>>>> >> > http://blog.nanthrax.net
>>>> >> > Talend - http://www.talend.com
>>>
>>>
>
Re: Apache Beam, version 2.2.0
Posted by Ahmet Altay <al...@google.com>.
On Thu, Dec 7, 2017 at 3:51 PM, Eugene Kirpichov <ki...@google.com>
wrote:
> I've sent the poll https://lists.apache.org/thread.html/
> 5bc2e184a24de9dbc8184ffd2720d1894010497d47d956b395e037df@%
> 3Cuser.beam.apache.org%3E
> Will figure out how to tweet from @ApacheBeam, and sent the Twitter poll
> as well (or ask someone to).
>
I tweeted the poll.
>
> On Wed, Dec 6, 2017 at 1:47 PM Lukasz Cwik <lc...@google.com> wrote:
>
>> +1 on moving forward with the plan suggested by kirpichov@
>>
>> On Wed, Dec 6, 2017 at 9:14 AM, Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> +1 to moving forward with this plan.
>>>
>>> (FWIW, this seems *less* backwards incompatible than, say, moving from
>>> Spark 1 to Spark 2, which was decided much quicker. I suppose the
>>> Spark change has a lower bound on the number of users it could impact
>>> though.)
>>>
>>> On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <ki...@google.com>
>>> wrote:
>>> > Okay, then let's go forward. Seems that we should:
>>> > - Open a new poll on user@, in light of 2.2 having been released
>>> > - Open a twitter poll
>>> > - Tweet that there's also a poll going on on user@
>>> > - Runner authors will reach out to respective runner user communities
>>> > - 2 weeks later we gather results and decide
>>> > ?
>>> >
>>> > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>> >>
>>> >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far away,
>>> >> and starting to improve Beam to offer a Java 8 friendly experience
>>> >> seems like an excellent idea.
>>> >>
>>> >> I understand the backwards compatibility argument. We should do the
>>> >> poll in twitter + try to reach more users for comments. If you
>>> >> consider that it is worth, I can open a second poll at user@.
>>> >>
>>> >> In any case we should try to move forward, even if we have more than
>>> >> 5% of users who want to stay on Java 7 we can consider to maintain
>>> >> minor releases of a backwards compatible version where we can backport
>>> >> only critical fixes e.g. security/data related errors but nothing new,
>>> >> in case some user really needs to have them. Of course this can be
>>> >> some extra work (to be discussed).
>>> >>
>>> >>
>>> >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré <jb@nanthrax.net
>>> >
>>> >> wrote:
>>> >> > +1, and sorry again, I thought we got an consensus.
>>> >> >
>>> >> > Regards
>>> >> > JB
>>> >> >
>>> >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote:
>>> >> >>
>>> >> >> +1 to the poll and also to Reuven's point.
>>> >> >>
>>> >> >> Those without a support contract would have been using JDK 7
>>> without
>>> >> >> security updates for years. IMO it seems harmful, as a netizen, to
>>> >> >> encourage
>>> >> >> its use/existence.
>>> >> >>
>>> >> >> If there's no noise from the prior thread, then I would assume no
>>> one
>>> >> >> on
>>> >> >> user@ has any objection. Anyone else with customers should reach
>>> out to
>>> >> >> them.
>>> >> >>
>>> >> >> Kenn
>>> >> >>
>>> >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <relax@google.com
>>> >> >> <ma...@google.com>> wrote:
>>> >> >>
>>> >> >> Technically it's a backwards-incompatible change, however if
>>> we are
>>> >> >> convinced the risk is low we could do it.
>>> >> >>
>>> >> >> As mentioned on the original thread, it's not clear that all
>>> Beam
>>> >> >> users read
>>> >> >> user@ - e.g. most Dataflow users definitely do not. I think
>>> we need
>>> >> >> to
>>> >> >> separately reach out to users of each runner through
>>> >> >> runner-specific
>>> >> >> channels.
>>> >> >>
>>> >> >> Reuven
>>> >> >>
>>> >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov
>>> >> >> <kirpichov@google.com
>>> >> >> <ma...@google.com>> wrote:
>>> >> >>
>>> >> >> On the original thread
>>> >> >>
>>> >> >>
>>> >> >> https://lists.apache.org/thread.html/
>>> 2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%
>>> 3Cuser.beam.apache.org%3E
>>> >> >>
>>> >> >>
>>> >> >> <https://lists.apache.org/thread.html/
>>> 2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%
>>> 3Cuser.beam.apache.org%3E>
>>> >> >> ,
>>> >> >> Robert and Ismaël were in favor of no major version change
>>> >> >> [Ismaël
>>> >> >> said:/Also I am afraid that if we wait/
>>> >> >> /until we have enough changes to switch Beam to a new major
>>> >> >> version the
>>> >> >> switch to Java 8 will happen too late, probably after Java
>>> 8's
>>> >> >> end
>>> >> >> of
>>> >> >> life. And I am not exaggerating, Java 8 is planned to EOL
>>> next
>>> >> >> march
>>> >> >> 2018!/]; JB and now Reuven are in favor of a major version
>>> >> >> change;
>>> >> >>
>>> >> >> nobody so far argued against switching to Java8 in general.
>>> >> >>
>>> >> >> I'm personally in favor of no major version change (i.e.
>>> not
>>> >> >> waiting
>>> >> >> until all other large changes for Beam 3.0 converge, which
>>> will
>>> >> >> likely
>>> >> >> be many months), because:
>>> >> >> - Reasons Ismaël cited; plus the reason that most people
>>> are
>>> >> >> likely
>>> >> >> already using Java 8.
>>> >> >> - Going Java8-only earlier will make other Beam 3.0 APIs
>>> better
>>> >> >> for
>>> >> >> Java8 users, because we (Beam contributors) will have
>>> >> >> experience
>>> >> >> working
>>> >> >> with them within the SDK in Java8 (e.g. writing tests with
>>> use
>>> >> >> of
>>> >> >> lambdas and noticing whether it's clunky, or whether some
>>> other
>>> >> >> Beam
>>> >> >> APIs need better Java8 support).
>>> >> >> - Going Java8 will make it more reasonable to include
>>> (mostly
>>> >> >> or
>>> >> >> only)
>>> >> >> Java8 snippets in Beam documentation, which will obviously
>>> look
>>> >> >> more
>>> >> >> concise and attractive, addressing one of the common
>>> concerns
>>> >> >> of
>>> >> >> Beam
>>> >> >> users that it has a heavyweight API compared to
>>> >> >> functional-style
>>> >> >> APIs of
>>> >> >> Spark etc.
>>> >> >>
>>> >> >> I think resolving this via a poll of users would be
>>> reasonable.
>>> >> >> I'd
>>> >> >> suggest e.g. the following phrasing:
>>> >> >>
>>> >> >> Apache Beam is considering dropping support for Java 7, and
>>> >> >> supporting
>>> >> >> only Java 8 and above in a subsequent release. How would it
>>> >> >> impact
>>> >> >> your
>>> >> >> usage of Beam?
>>> >> >> - I am already using only Java 8+ for building my Beam code
>>> >> >> - I am using Java 7 for building my Beam code, but I would
>>> have
>>> >> >> no
>>> >> >> trouble switching to Java 8
>>> >> >> - I am using Java 7 for building my Beam code, and dropping
>>> >> >> Java 7
>>> >> >> would
>>> >> >> be a blocker or hindrance to adopting the new release for
>>> me
>>> >> >>
>>> >> >> We could tweet this poll on Apache Beam twitter and
>>> publish on
>>> >> >> user@,
>>> >> >> and, say, if we receive 5% or fewer votes for option 3
>>> after
>>> >> >> keeping it
>>> >> >> open for 2 weeks, then adopt Java 8 without a major version
>>> >> >> change.
>>> >> >>
>>> >> >> WDYT?
>>> >> >>
>>> >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré
>>> >> >> <jb@nanthrax.net
>>> >> >> <ma...@nanthrax.net>> wrote:
>>> >> >>
>>> >> >> Good idea ! Definitely +1
>>> >> >>
>>> >> >> Regards
>>> >> >> JB
>>> >> >>
>>> >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote:
>>> >> >> > We should bring this up on the Beam 3.0 thread.
>>> Since
>>> >> >> it's
>>> >> >> technically a
>>> >> >> > backwards-incompatible change, it might make a good
>>> item
>>> >> >> for Beam
>>> >> >> 3.0.
>>> >> >> >
>>> >> >> > Reuven
>>> >> >> >
>>> >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré
>>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
>>> >> >> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>>
>>> >> >> wrote:
>>> >> >> >
>>> >> >> > My apologizes, I thought we had a consensus
>>> already.
>>> >> >> >
>>> >> >> > Regards
>>> >> >> > JB
>>> >> >> >
>>> >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
>>> >> >> >
>>> >> >> > Thanks JB for sending the detailed notes
>>> about
>>> >> >> new
>>> >> >> stuff
>>> >> >> in 2.2.0! A lot
>>> >> >> > of exciting things indeed.
>>> >> >> >
>>> >> >> > Regarding Java 8: I thought our consensus
>>> was to
>>> >> >> have the
>>> >> >> release notes
>>> >> >> > say that we're *considering* going
>>> Java8-only,
>>> >> >> and
>>> >> >> use
>>> >> >> that to get more
>>> >> >> > opinions from the user community - but I
>>> can't
>>> >> >> find
>>> >> >> the
>>> >> >> emails that made
>>> >> >> > me think so.
>>> >> >> >
>>> >> >> > +Ismaël Mejía <mailto:iemejia@gmail.com
>>> >> >> <ma...@gmail.com> <mailto:iemejia@gmail.com
>>> >> >> <ma...@gmail.com>>> - do
>>> >> >> > you think we should formally conclude the
>>> vote
>>> >> >> on
>>> >> >> the
>>> >> >> thread [VOTE]
>>> >> >> > [DISCUSSION] Remove support for Java 7?
>>> >> >> > Or should we take more steps - e.g. perhaps
>>> >> >> tweet a
>>> >> >> link
>>> >> >> to that thread
>>> >> >> > from the Beam twitter account, ask people to
>>> >> >> chime
>>> >> >> in,
>>> >> >> and wait for say
>>> >> >> > 2 weeks before declaring a conclusion?
>>> >> >> >
>>> >> >> > Let's also have a process JIRA for going
>>> Java8.
>>> >> >> I've
>>> >> >> filed one:
>>> >> >> > https://issues.apache.org/jira/browse/BEAM-3285
>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>
>>> >> >> > <https://issues.apache.org/
>>> jira/browse/BEAM-3285
>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>>
>>> >> >> >
>>> >> >> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste
>>> >> >> Onofré
>>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
>>> >> >> > <mailto:jb@nanthrax.net
>>> >> >> <ma...@nanthrax.net>>
>>> >> >> <mailto:jb@nanthrax.net <ma...@nanthrax.net>
>>> >> >> > <mailto:jb@nanthrax.net
>>> >> >> <ma...@nanthrax.net>>>>
>>> >> >> wrote:
>>> >> >> >
>>> >> >> > Just an important note that we forgot
>>> to
>>> >> >> mention.
>>> >> >> >
>>> >> >> > !! The 2.2.0 release will be the last
>>> one
>>> >> >> supporting
>>> >> >> Spark 1.x and
>>> >> >> > Java 7 !!
>>> >> >> >
>>> >> >> > Starting from Beam 2.3.0, the Spark
>>> runner
>>> >> >> will work
>>> >> >> only with
>>> >> >> > Spark 2.x and we
>>> >> >> > will focus only Java 8.
>>> >> >> >
>>> >> >> > Regards
>>> >> >> > JB
>>> >> >> >
>>> >> >> > On 12/04/2017 10:15 AM, Jean-Baptiste
>>> >> >> Onofré
>>> >> >> wrote:
>>> >> >> > > Thanks Reuven !
>>> >> >> > >
>>> >> >> > > I would like to emphasize on some
>>> >> >> highlights in
>>> >> >> 2.2.0 release:
>>> >> >> > >
>>> >> >> > > - New IOs have been introduced:
>>> >> >> > > * TikaIO leveraging Apache Tika,
>>> >> >> allowing
>>> >> >> the
>>> >> >> deal with a lot
>>> >> >> > of different
>>> >> >> > > data formats
>>> >> >> > > * RedisIO to read and write
>>> key/value
>>> >> >> pairs
>>> >> >> from a Redis
>>> >> >> > server. This
>>> >> >> > IO will
>>> >> >> > > be soon extended to Redis PubSub.
>>> >> >> > > * FileIO provides transforms for
>>> >> >> working
>>> >> >> with
>>> >> >> files (raw).
>>> >> >> > Especially, it
>>> >> >> > > provides matching file patterns and
>>> read
>>> >> >> on
>>> >> >> patterns. It can be
>>> >> >> > easily
>>> >> >> > extended
>>> >> >> > > for a specific format (like we do in
>>> >> >> AvroIO
>>> >> >> or
>>> >> >> TextIO now).
>>> >> >> > > * SolrIO to interact with Apache
>>> Solr
>>> >> >> (Lucene)
>>> >> >> > >
>>> >> >> > > - On the other hand, improvements
>>> have
>>> >> >> been
>>> >> >> performed on
>>> >> >> > existing IOs:
>>> >> >> > > * We started to introduce readAll
>>> >> >> pattern
>>> >> >> in
>>> >> >> IOs (AvroIO,
>>> >> >> > TextIO, JdbcIO,
>>> >> >> > > ...), allowing to pass "request"
>>> >> >> arguments
>>> >> >> via an
>>> >> >> input PCollection.
>>> >> >> > > * ElasticsearchIO has an improved
>>> >> >> support
>>> >> >> of
>>> >> >> different
>>> >> >> > Elasticsearch
>>> >> >> > version
>>> >> >> > > (including Elasticsearch 5.x). It
>>> also
>>> >> >> now
>>> >> >> supports SSL/TLS.
>>> >> >> > > * HBaseIO is now able to do
>>> dynamic
>>> >> >> work
>>> >> >> rebalancing
>>> >> >> > > * KinesisIO uses a more accurate
>>> >> >> watermark
>>> >> >> (based on
>>> >> >> > approximateArrivalTimestamp)
>>> >> >> > > * TextIO now supports custom
>>> delimiter
>>> >> >> and like
>>> >> >> AvroIO,
>>> >> >> > supports the
>>> >> >> > readAll
>>> >> >> > > pattern,
>>> >> >> > > * Performance improvements on
>>> JdbcIO
>>> >> >> when
>>> >> >> it
>>> >> >> has to read lot
>>> >> >> > of rows
>>> >> >> > > * Kafka write supports
>>> Exactly-Once
>>> >> >> pattern
>>> >> >> (introduce in
>>> >> >> > Kafka 0.11.x)
>>> >> >> > >
>>> >> >> > > - A new DSL has been introduced:
>>> the SQL
>>> >> >> DSL !
>>> >> >> > >
>>> >> >> > > We are now focus on 2.3.0 release
>>> with
>>> >> >> new
>>> >> >> improvements and
>>> >> >> > features !
>>> >> >> > >
>>> >> >> > > Stay tuned !
>>> >> >> > >
>>> >> >> > > JB on behalf of the Apache Beam
>>> >> >> community.
>>> >> >> > >
>>> >> >> > > On 12/02/2017 11:40 PM, Reuven Lax
>>> >> >> wrote:
>>> >> >> > >> The Apache Beam community is
>>> pleased to
>>> >> >> announce the
>>> >> >> > availability of the
>>> >> >> > >> 2.2.0 release.
>>> >> >> > >>
>>> >> >> > >> This release adds support for
>>> generic
>>> >> >> file
>>> >> >> sources and sinks
>>> >> >> > (beyond TextIO
>>> >> >> > >> and AvroIO) using FileIO, including
>>> >> >> support for
>>> >> >> dynamic
>>> >> >> > filenames using
>>> >> >> > >> readAll; this allows streaming
>>> >> >> pipelines
>>> >> >> to now
>>> >> >> read from files by
>>> >> >> > >> continuously monitoring a
>>> directory for
>>> >> >> new
>>> >> >> filw. Many other
>>> >> >> > IOs are
>>> >> >> > improved,
>>> >> >> > >> notably including exactly-once
>>> support
>>> >> >> for
>>> >> >> the
>>> >> >> Kafka sink. Initial
>>> >> >> > support for
>>> >> >> > >> BEAM-SQL is also included in this
>>> >> >> release.
>>> >> >> For a
>>> >> >> more-complete
>>> >> >> > list of major
>>> >> >> > >> changes in the release, please
>>> refer to
>>> >> >> the
>>> >> >> release notes [2].
>>> >> >> > >>
>>> >> >> > >> The 2.2.0 release is now the
>>> >> >> recommended
>>> >> >> version; we encourage
>>> >> >> > everyone to
>>> >> >> > >> upgrade from any earlier releases.
>>> >> >> > >>
>>> >> >> > >> We’d like to invite everyone to
>>> try out
>>> >> >> Apache
>>> >> >> Beam today and
>>> >> >> > consider
>>> >> >> > >> joining our vibrant community. We
>>> >> >> welcome
>>> >> >> feedback,
>>> >> >> > contribution and
>>> >> >> > >> participation through our mailing
>>> >> >> lists,
>>> >> >> issue
>>> >> >> tracker, pull
>>> >> >> > requests, and
>>> >> >> > >> events.
>>> >> >> > >>
>>> >> >> > >> - Reuven Lax, on behalf of the
>>> Apache
>>> >> >> Beam
>>> >> >> community.
>>> >> >> > >>
>>> >> >> > >> [1]
>>> >> >> https://beam.apache.org/get-started/downloads/
>>> >> >> <https://beam.apache.org/get-started/downloads/>
>>> >> >> > <https://beam.apache.org/get-
>>> started/downloads/
>>> >> >> <https://beam.apache.org/get-started/downloads/>>
>>> >> >> > >> [2]
>>> >> >> > >>
>>> >> >> >
>>> >> >>
>>> >> >>
>>> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>> projectId=12319527&version=12341044
>>> >> >>
>>> >> >>
>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>> projectId=12319527&version=12341044>
>>> >> >> >
>>> >> >>
>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>> projectId=12319527&version=12341044
>>> >> >>
>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>> projectId=12319527&version=12341044>>
>>> >> >> > >>
>>> >> >> > >
>>> >> >> >
>>> >> >> > --
>>> >> >> > Jean-Baptiste Onofré
>>> >> >> > jbonofre@apache.org <ma...@apache.org>
>>> >> >> <mailto:jbonofre@apache.org <mailto:
>>> jbonofre@apache.org>>
>>> >> >> > <mailto:jbonofre@apache.org
>>> >> >> <ma...@apache.org>
>>> >> >> <mailto:jbonofre@apache.org <mailto:
>>> jbonofre@apache.org>>>
>>> >> >> > http://blog.nanthrax.net
>>> >> >> > Talend - http://www.talend.com
>>> >> >> >
>>> >> >> >
>>> >> >> > --
>>> >> >> > Jean-Baptiste Onofré
>>> >> >> > jbonofre@apache.org <ma...@apache.org>
>>> >> >> <mailto:jbonofre@apache.org <mailto:
>>> jbonofre@apache.org>>
>>> >> >> > http://blog.nanthrax.net
>>> >> >> > Talend - http://www.talend.com
>>> >> >> >
>>> >> >> >
>>> >> >>
>>> >> >> --
>>> >> >> Jean-Baptiste Onofré
>>> >> >> jbonofre@apache.org <ma...@apache.org>
>>> >> >> http://blog.nanthrax.net
>>> >> >> Talend - http://www.talend.com
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >
>>> >> > --
>>> >> > Jean-Baptiste Onofré
>>> >> > jbonofre@apache.org
>>> >> > http://blog.nanthrax.net
>>> >> > Talend - http://www.talend.com
>>>
>>
>>
Re: Apache Beam, version 2.2.0
Posted by Eugene Kirpichov <ki...@google.com>.
I've sent the poll
https://lists.apache.org/thread.html/5bc2e184a24de9dbc8184ffd2720d1894010497d47d956b395e037df@%3Cuser.beam.apache.org%3E
Will figure out how to tweet from @ApacheBeam, and sent the Twitter poll as
well (or ask someone to).
On Wed, Dec 6, 2017 at 1:47 PM Lukasz Cwik <lc...@google.com> wrote:
> +1 on moving forward with the plan suggested by kirpichov@
>
> On Wed, Dec 6, 2017 at 9:14 AM, Robert Bradshaw <ro...@google.com>
> wrote:
>
>> +1 to moving forward with this plan.
>>
>> (FWIW, this seems *less* backwards incompatible than, say, moving from
>> Spark 1 to Spark 2, which was decided much quicker. I suppose the
>> Spark change has a lower bound on the number of users it could impact
>> though.)
>>
>> On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <ki...@google.com>
>> wrote:
>> > Okay, then let's go forward. Seems that we should:
>> > - Open a new poll on user@, in light of 2.2 having been released
>> > - Open a twitter poll
>> > - Tweet that there's also a poll going on on user@
>> > - Runner authors will reach out to respective runner user communities
>> > - 2 weeks later we gather results and decide
>> > ?
>> >
>> > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <ie...@gmail.com> wrote:
>> >>
>> >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far away,
>> >> and starting to improve Beam to offer a Java 8 friendly experience
>> >> seems like an excellent idea.
>> >>
>> >> I understand the backwards compatibility argument. We should do the
>> >> poll in twitter + try to reach more users for comments. If you
>> >> consider that it is worth, I can open a second poll at user@.
>> >>
>> >> In any case we should try to move forward, even if we have more than
>> >> 5% of users who want to stay on Java 7 we can consider to maintain
>> >> minor releases of a backwards compatible version where we can backport
>> >> only critical fixes e.g. security/data related errors but nothing new,
>> >> in case some user really needs to have them. Of course this can be
>> >> some extra work (to be discussed).
>> >>
>> >>
>> >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>> >> wrote:
>> >> > +1, and sorry again, I thought we got an consensus.
>> >> >
>> >> > Regards
>> >> > JB
>> >> >
>> >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote:
>> >> >>
>> >> >> +1 to the poll and also to Reuven's point.
>> >> >>
>> >> >> Those without a support contract would have been using JDK 7 without
>> >> >> security updates for years. IMO it seems harmful, as a netizen, to
>> >> >> encourage
>> >> >> its use/existence.
>> >> >>
>> >> >> If there's no noise from the prior thread, then I would assume no
>> one
>> >> >> on
>> >> >> user@ has any objection. Anyone else with customers should reach
>> out to
>> >> >> them.
>> >> >>
>> >> >> Kenn
>> >> >>
>> >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <relax@google.com
>> >> >> <ma...@google.com>> wrote:
>> >> >>
>> >> >> Technically it's a backwards-incompatible change, however if we
>> are
>> >> >> convinced the risk is low we could do it.
>> >> >>
>> >> >> As mentioned on the original thread, it's not clear that all
>> Beam
>> >> >> users read
>> >> >> user@ - e.g. most Dataflow users definitely do not. I think we
>> need
>> >> >> to
>> >> >> separately reach out to users of each runner through
>> >> >> runner-specific
>> >> >> channels.
>> >> >>
>> >> >> Reuven
>> >> >>
>> >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov
>> >> >> <kirpichov@google.com
>> >> >> <ma...@google.com>> wrote:
>> >> >>
>> >> >> On the original thread
>> >> >>
>> >> >>
>> >> >>
>> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
>> >> >>
>> >> >>
>> >> >> <
>> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
>> >
>> >> >> ,
>> >> >> Robert and Ismaël were in favor of no major version change
>> >> >> [Ismaël
>> >> >> said:/Also I am afraid that if we wait/
>> >> >> /until we have enough changes to switch Beam to a new major
>> >> >> version the
>> >> >> switch to Java 8 will happen too late, probably after Java
>> 8's
>> >> >> end
>> >> >> of
>> >> >> life. And I am not exaggerating, Java 8 is planned to EOL
>> next
>> >> >> march
>> >> >> 2018!/]; JB and now Reuven are in favor of a major version
>> >> >> change;
>> >> >>
>> >> >> nobody so far argued against switching to Java8 in general.
>> >> >>
>> >> >> I'm personally in favor of no major version change (i.e. not
>> >> >> waiting
>> >> >> until all other large changes for Beam 3.0 converge, which
>> will
>> >> >> likely
>> >> >> be many months), because:
>> >> >> - Reasons Ismaël cited; plus the reason that most people are
>> >> >> likely
>> >> >> already using Java 8.
>> >> >> - Going Java8-only earlier will make other Beam 3.0 APIs
>> better
>> >> >> for
>> >> >> Java8 users, because we (Beam contributors) will have
>> >> >> experience
>> >> >> working
>> >> >> with them within the SDK in Java8 (e.g. writing tests with
>> use
>> >> >> of
>> >> >> lambdas and noticing whether it's clunky, or whether some
>> other
>> >> >> Beam
>> >> >> APIs need better Java8 support).
>> >> >> - Going Java8 will make it more reasonable to include
>> (mostly
>> >> >> or
>> >> >> only)
>> >> >> Java8 snippets in Beam documentation, which will obviously
>> look
>> >> >> more
>> >> >> concise and attractive, addressing one of the common
>> concerns
>> >> >> of
>> >> >> Beam
>> >> >> users that it has a heavyweight API compared to
>> >> >> functional-style
>> >> >> APIs of
>> >> >> Spark etc.
>> >> >>
>> >> >> I think resolving this via a poll of users would be
>> reasonable.
>> >> >> I'd
>> >> >> suggest e.g. the following phrasing:
>> >> >>
>> >> >> Apache Beam is considering dropping support for Java 7, and
>> >> >> supporting
>> >> >> only Java 8 and above in a subsequent release. How would it
>> >> >> impact
>> >> >> your
>> >> >> usage of Beam?
>> >> >> - I am already using only Java 8+ for building my Beam code
>> >> >> - I am using Java 7 for building my Beam code, but I would
>> have
>> >> >> no
>> >> >> trouble switching to Java 8
>> >> >> - I am using Java 7 for building my Beam code, and dropping
>> >> >> Java 7
>> >> >> would
>> >> >> be a blocker or hindrance to adopting the new release for me
>> >> >>
>> >> >> We could tweet this poll on Apache Beam twitter and publish
>> on
>> >> >> user@,
>> >> >> and, say, if we receive 5% or fewer votes for option 3 after
>> >> >> keeping it
>> >> >> open for 2 weeks, then adopt Java 8 without a major version
>> >> >> change.
>> >> >>
>> >> >> WDYT?
>> >> >>
>> >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré
>> >> >> <jb@nanthrax.net
>> >> >> <ma...@nanthrax.net>> wrote:
>> >> >>
>> >> >> Good idea ! Definitely +1
>> >> >>
>> >> >> Regards
>> >> >> JB
>> >> >>
>> >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote:
>> >> >> > We should bring this up on the Beam 3.0 thread. Since
>> >> >> it's
>> >> >> technically a
>> >> >> > backwards-incompatible change, it might make a good
>> item
>> >> >> for Beam
>> >> >> 3.0.
>> >> >> >
>> >> >> > Reuven
>> >> >> >
>> >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré
>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
>> >> >> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>>
>> >> >> wrote:
>> >> >> >
>> >> >> > My apologizes, I thought we had a consensus
>> already.
>> >> >> >
>> >> >> > Regards
>> >> >> > JB
>> >> >> >
>> >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
>> >> >> >
>> >> >> > Thanks JB for sending the detailed notes
>> about
>> >> >> new
>> >> >> stuff
>> >> >> in 2.2.0! A lot
>> >> >> > of exciting things indeed.
>> >> >> >
>> >> >> > Regarding Java 8: I thought our consensus
>> was to
>> >> >> have the
>> >> >> release notes
>> >> >> > say that we're *considering* going
>> Java8-only,
>> >> >> and
>> >> >> use
>> >> >> that to get more
>> >> >> > opinions from the user community - but I
>> can't
>> >> >> find
>> >> >> the
>> >> >> emails that made
>> >> >> > me think so.
>> >> >> >
>> >> >> > +Ismaël Mejía <mailto:iemejia@gmail.com
>> >> >> <ma...@gmail.com> <mailto:iemejia@gmail.com
>> >> >> <ma...@gmail.com>>> - do
>> >> >> > you think we should formally conclude the
>> vote
>> >> >> on
>> >> >> the
>> >> >> thread [VOTE]
>> >> >> > [DISCUSSION] Remove support for Java 7?
>> >> >> > Or should we take more steps - e.g. perhaps
>> >> >> tweet a
>> >> >> link
>> >> >> to that thread
>> >> >> > from the Beam twitter account, ask people to
>> >> >> chime
>> >> >> in,
>> >> >> and wait for say
>> >> >> > 2 weeks before declaring a conclusion?
>> >> >> >
>> >> >> > Let's also have a process JIRA for going
>> Java8.
>> >> >> I've
>> >> >> filed one:
>> >> >> > https://issues.apache.org/jira/browse/BEAM-3285
>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>
>> >> >> > <
>> https://issues.apache.org/jira/browse/BEAM-3285
>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>>
>> >> >> >
>> >> >> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste
>> >> >> Onofré
>> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
>> >> >> > <mailto:jb@nanthrax.net
>> >> >> <ma...@nanthrax.net>>
>> >> >> <mailto:jb@nanthrax.net <ma...@nanthrax.net>
>> >> >> > <mailto:jb@nanthrax.net
>> >> >> <ma...@nanthrax.net>>>>
>> >> >> wrote:
>> >> >> >
>> >> >> > Just an important note that we forgot to
>> >> >> mention.
>> >> >> >
>> >> >> > !! The 2.2.0 release will be the last
>> one
>> >> >> supporting
>> >> >> Spark 1.x and
>> >> >> > Java 7 !!
>> >> >> >
>> >> >> > Starting from Beam 2.3.0, the Spark
>> runner
>> >> >> will work
>> >> >> only with
>> >> >> > Spark 2.x and we
>> >> >> > will focus only Java 8.
>> >> >> >
>> >> >> > Regards
>> >> >> > JB
>> >> >> >
>> >> >> > On 12/04/2017 10:15 AM, Jean-Baptiste
>> >> >> Onofré
>> >> >> wrote:
>> >> >> > > Thanks Reuven !
>> >> >> > >
>> >> >> > > I would like to emphasize on some
>> >> >> highlights in
>> >> >> 2.2.0 release:
>> >> >> > >
>> >> >> > > - New IOs have been introduced:
>> >> >> > > * TikaIO leveraging Apache Tika,
>> >> >> allowing
>> >> >> the
>> >> >> deal with a lot
>> >> >> > of different
>> >> >> > > data formats
>> >> >> > > * RedisIO to read and write
>> key/value
>> >> >> pairs
>> >> >> from a Redis
>> >> >> > server. This
>> >> >> > IO will
>> >> >> > > be soon extended to Redis PubSub.
>> >> >> > > * FileIO provides transforms for
>> >> >> working
>> >> >> with
>> >> >> files (raw).
>> >> >> > Especially, it
>> >> >> > > provides matching file patterns and
>> read
>> >> >> on
>> >> >> patterns. It can be
>> >> >> > easily
>> >> >> > extended
>> >> >> > > for a specific format (like we do in
>> >> >> AvroIO
>> >> >> or
>> >> >> TextIO now).
>> >> >> > > * SolrIO to interact with Apache
>> Solr
>> >> >> (Lucene)
>> >> >> > >
>> >> >> > > - On the other hand, improvements
>> have
>> >> >> been
>> >> >> performed on
>> >> >> > existing IOs:
>> >> >> > > * We started to introduce readAll
>> >> >> pattern
>> >> >> in
>> >> >> IOs (AvroIO,
>> >> >> > TextIO, JdbcIO,
>> >> >> > > ...), allowing to pass "request"
>> >> >> arguments
>> >> >> via an
>> >> >> input PCollection.
>> >> >> > > * ElasticsearchIO has an improved
>> >> >> support
>> >> >> of
>> >> >> different
>> >> >> > Elasticsearch
>> >> >> > version
>> >> >> > > (including Elasticsearch 5.x). It
>> also
>> >> >> now
>> >> >> supports SSL/TLS.
>> >> >> > > * HBaseIO is now able to do dynamic
>> >> >> work
>> >> >> rebalancing
>> >> >> > > * KinesisIO uses a more accurate
>> >> >> watermark
>> >> >> (based on
>> >> >> > approximateArrivalTimestamp)
>> >> >> > > * TextIO now supports custom
>> delimiter
>> >> >> and like
>> >> >> AvroIO,
>> >> >> > supports the
>> >> >> > readAll
>> >> >> > > pattern,
>> >> >> > > * Performance improvements on
>> JdbcIO
>> >> >> when
>> >> >> it
>> >> >> has to read lot
>> >> >> > of rows
>> >> >> > > * Kafka write supports Exactly-Once
>> >> >> pattern
>> >> >> (introduce in
>> >> >> > Kafka 0.11.x)
>> >> >> > >
>> >> >> > > - A new DSL has been introduced: the
>> SQL
>> >> >> DSL !
>> >> >> > >
>> >> >> > > We are now focus on 2.3.0 release
>> with
>> >> >> new
>> >> >> improvements and
>> >> >> > features !
>> >> >> > >
>> >> >> > > Stay tuned !
>> >> >> > >
>> >> >> > > JB on behalf of the Apache Beam
>> >> >> community.
>> >> >> > >
>> >> >> > > On 12/02/2017 11:40 PM, Reuven Lax
>> >> >> wrote:
>> >> >> > >> The Apache Beam community is
>> pleased to
>> >> >> announce the
>> >> >> > availability of the
>> >> >> > >> 2.2.0 release.
>> >> >> > >>
>> >> >> > >> This release adds support for
>> generic
>> >> >> file
>> >> >> sources and sinks
>> >> >> > (beyond TextIO
>> >> >> > >> and AvroIO) using FileIO, including
>> >> >> support for
>> >> >> dynamic
>> >> >> > filenames using
>> >> >> > >> readAll; this allows streaming
>> >> >> pipelines
>> >> >> to now
>> >> >> read from files by
>> >> >> > >> continuously monitoring a directory
>> for
>> >> >> new
>> >> >> filw. Many other
>> >> >> > IOs are
>> >> >> > improved,
>> >> >> > >> notably including exactly-once
>> support
>> >> >> for
>> >> >> the
>> >> >> Kafka sink. Initial
>> >> >> > support for
>> >> >> > >> BEAM-SQL is also included in this
>> >> >> release.
>> >> >> For a
>> >> >> more-complete
>> >> >> > list of major
>> >> >> > >> changes in the release, please
>> refer to
>> >> >> the
>> >> >> release notes [2].
>> >> >> > >>
>> >> >> > >> The 2.2.0 release is now the
>> >> >> recommended
>> >> >> version; we encourage
>> >> >> > everyone to
>> >> >> > >> upgrade from any earlier releases.
>> >> >> > >>
>> >> >> > >> We’d like to invite everyone to try
>> out
>> >> >> Apache
>> >> >> Beam today and
>> >> >> > consider
>> >> >> > >> joining our vibrant community. We
>> >> >> welcome
>> >> >> feedback,
>> >> >> > contribution and
>> >> >> > >> participation through our mailing
>> >> >> lists,
>> >> >> issue
>> >> >> tracker, pull
>> >> >> > requests, and
>> >> >> > >> events.
>> >> >> > >>
>> >> >> > >> - Reuven Lax, on behalf of the
>> Apache
>> >> >> Beam
>> >> >> community.
>> >> >> > >>
>> >> >> > >> [1]
>> >> >> https://beam.apache.org/get-started/downloads/
>> >> >> <https://beam.apache.org/get-started/downloads/>
>> >> >> > <
>> https://beam.apache.org/get-started/downloads/
>> >> >> <https://beam.apache.org/get-started/downloads/>>
>> >> >> > >> [2]
>> >> >> > >>
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
>> >> >>
>> >> >>
>> >> >> <
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
>> >
>> >> >> >
>> >> >>
>> >> >> <
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
>> >> >>
>> >> >> <
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
>> >>
>> >> >> > >>
>> >> >> > >
>> >> >> >
>> >> >> > --
>> >> >> > Jean-Baptiste Onofré
>> >> >> > jbonofre@apache.org <ma...@apache.org>
>> >> >> <mailto:jbonofre@apache.org <mailto:jbonofre@apache.org
>> >>
>> >> >> > <mailto:jbonofre@apache.org
>> >> >> <ma...@apache.org>
>> >> >> <mailto:jbonofre@apache.org <mailto:jbonofre@apache.org
>> >>>
>> >> >> > http://blog.nanthrax.net
>> >> >> > Talend - http://www.talend.com
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Jean-Baptiste Onofré
>> >> >> > jbonofre@apache.org <ma...@apache.org>
>> >> >> <mailto:jbonofre@apache.org <mailto:jbonofre@apache.org
>> >>
>> >> >> > http://blog.nanthrax.net
>> >> >> > Talend - http://www.talend.com
>> >> >> >
>> >> >> >
>> >> >>
>> >> >> --
>> >> >> Jean-Baptiste Onofré
>> >> >> jbonofre@apache.org <ma...@apache.org>
>> >> >> http://blog.nanthrax.net
>> >> >> Talend - http://www.talend.com
>> >> >>
>> >> >>
>> >> >>
>> >> >
>> >> > --
>> >> > Jean-Baptiste Onofré
>> >> > jbonofre@apache.org
>> >> > http://blog.nanthrax.net
>> >> > Talend - http://www.talend.com
>>
>
>
Re: Apache Beam, version 2.2.0
Posted by Lukasz Cwik <lc...@google.com>.
+1 on moving forward with the plan suggested by kirpichov@
On Wed, Dec 6, 2017 at 9:14 AM, Robert Bradshaw <ro...@google.com> wrote:
> +1 to moving forward with this plan.
>
> (FWIW, this seems *less* backwards incompatible than, say, moving from
> Spark 1 to Spark 2, which was decided much quicker. I suppose the
> Spark change has a lower bound on the number of users it could impact
> though.)
>
> On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <ki...@google.com>
> wrote:
> > Okay, then let's go forward. Seems that we should:
> > - Open a new poll on user@, in light of 2.2 having been released
> > - Open a twitter poll
> > - Tweet that there's also a poll going on on user@
> > - Runner authors will reach out to respective runner user communities
> > - 2 weeks later we gather results and decide
> > ?
> >
> > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <ie...@gmail.com> wrote:
> >>
> >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far away,
> >> and starting to improve Beam to offer a Java 8 friendly experience
> >> seems like an excellent idea.
> >>
> >> I understand the backwards compatibility argument. We should do the
> >> poll in twitter + try to reach more users for comments. If you
> >> consider that it is worth, I can open a second poll at user@.
> >>
> >> In any case we should try to move forward, even if we have more than
> >> 5% of users who want to stay on Java 7 we can consider to maintain
> >> minor releases of a backwards compatible version where we can backport
> >> only critical fixes e.g. security/data related errors but nothing new,
> >> in case some user really needs to have them. Of course this can be
> >> some extra work (to be discussed).
> >>
> >>
> >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> >> wrote:
> >> > +1, and sorry again, I thought we got an consensus.
> >> >
> >> > Regards
> >> > JB
> >> >
> >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote:
> >> >>
> >> >> +1 to the poll and also to Reuven's point.
> >> >>
> >> >> Those without a support contract would have been using JDK 7 without
> >> >> security updates for years. IMO it seems harmful, as a netizen, to
> >> >> encourage
> >> >> its use/existence.
> >> >>
> >> >> If there's no noise from the prior thread, then I would assume no one
> >> >> on
> >> >> user@ has any objection. Anyone else with customers should reach
> out to
> >> >> them.
> >> >>
> >> >> Kenn
> >> >>
> >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <relax@google.com
> >> >> <ma...@google.com>> wrote:
> >> >>
> >> >> Technically it's a backwards-incompatible change, however if we
> are
> >> >> convinced the risk is low we could do it.
> >> >>
> >> >> As mentioned on the original thread, it's not clear that all Beam
> >> >> users read
> >> >> user@ - e.g. most Dataflow users definitely do not. I think we
> need
> >> >> to
> >> >> separately reach out to users of each runner through
> >> >> runner-specific
> >> >> channels.
> >> >>
> >> >> Reuven
> >> >>
> >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov
> >> >> <kirpichov@google.com
> >> >> <ma...@google.com>> wrote:
> >> >>
> >> >> On the original thread
> >> >>
> >> >>
> >> >> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130
> fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
> >> >>
> >> >>
> >> >> <https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130
> fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E>
> >> >> ,
> >> >> Robert and Ismaël were in favor of no major version change
> >> >> [Ismaël
> >> >> said:/Also I am afraid that if we wait/
> >> >> /until we have enough changes to switch Beam to a new major
> >> >> version the
> >> >> switch to Java 8 will happen too late, probably after Java
> 8's
> >> >> end
> >> >> of
> >> >> life. And I am not exaggerating, Java 8 is planned to EOL
> next
> >> >> march
> >> >> 2018!/]; JB and now Reuven are in favor of a major version
> >> >> change;
> >> >>
> >> >> nobody so far argued against switching to Java8 in general.
> >> >>
> >> >> I'm personally in favor of no major version change (i.e. not
> >> >> waiting
> >> >> until all other large changes for Beam 3.0 converge, which
> will
> >> >> likely
> >> >> be many months), because:
> >> >> - Reasons Ismaël cited; plus the reason that most people are
> >> >> likely
> >> >> already using Java 8.
> >> >> - Going Java8-only earlier will make other Beam 3.0 APIs
> better
> >> >> for
> >> >> Java8 users, because we (Beam contributors) will have
> >> >> experience
> >> >> working
> >> >> with them within the SDK in Java8 (e.g. writing tests with
> use
> >> >> of
> >> >> lambdas and noticing whether it's clunky, or whether some
> other
> >> >> Beam
> >> >> APIs need better Java8 support).
> >> >> - Going Java8 will make it more reasonable to include (mostly
> >> >> or
> >> >> only)
> >> >> Java8 snippets in Beam documentation, which will obviously
> look
> >> >> more
> >> >> concise and attractive, addressing one of the common concerns
> >> >> of
> >> >> Beam
> >> >> users that it has a heavyweight API compared to
> >> >> functional-style
> >> >> APIs of
> >> >> Spark etc.
> >> >>
> >> >> I think resolving this via a poll of users would be
> reasonable.
> >> >> I'd
> >> >> suggest e.g. the following phrasing:
> >> >>
> >> >> Apache Beam is considering dropping support for Java 7, and
> >> >> supporting
> >> >> only Java 8 and above in a subsequent release. How would it
> >> >> impact
> >> >> your
> >> >> usage of Beam?
> >> >> - I am already using only Java 8+ for building my Beam code
> >> >> - I am using Java 7 for building my Beam code, but I would
> have
> >> >> no
> >> >> trouble switching to Java 8
> >> >> - I am using Java 7 for building my Beam code, and dropping
> >> >> Java 7
> >> >> would
> >> >> be a blocker or hindrance to adopting the new release for me
> >> >>
> >> >> We could tweet this poll on Apache Beam twitter and publish
> on
> >> >> user@,
> >> >> and, say, if we receive 5% or fewer votes for option 3 after
> >> >> keeping it
> >> >> open for 2 weeks, then adopt Java 8 without a major version
> >> >> change.
> >> >>
> >> >> WDYT?
> >> >>
> >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré
> >> >> <jb@nanthrax.net
> >> >> <ma...@nanthrax.net>> wrote:
> >> >>
> >> >> Good idea ! Definitely +1
> >> >>
> >> >> Regards
> >> >> JB
> >> >>
> >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote:
> >> >> > We should bring this up on the Beam 3.0 thread. Since
> >> >> it's
> >> >> technically a
> >> >> > backwards-incompatible change, it might make a good
> item
> >> >> for Beam
> >> >> 3.0.
> >> >> >
> >> >> > Reuven
> >> >> >
> >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré
> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
> >> >> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>>
> >> >> wrote:
> >> >> >
> >> >> > My apologizes, I thought we had a consensus
> already.
> >> >> >
> >> >> > Regards
> >> >> > JB
> >> >> >
> >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
> >> >> >
> >> >> > Thanks JB for sending the detailed notes about
> >> >> new
> >> >> stuff
> >> >> in 2.2.0! A lot
> >> >> > of exciting things indeed.
> >> >> >
> >> >> > Regarding Java 8: I thought our consensus was
> to
> >> >> have the
> >> >> release notes
> >> >> > say that we're *considering* going Java8-only,
> >> >> and
> >> >> use
> >> >> that to get more
> >> >> > opinions from the user community - but I can't
> >> >> find
> >> >> the
> >> >> emails that made
> >> >> > me think so.
> >> >> >
> >> >> > +Ismaël Mejía <mailto:iemejia@gmail.com
> >> >> <ma...@gmail.com> <mailto:iemejia@gmail.com
> >> >> <ma...@gmail.com>>> - do
> >> >> > you think we should formally conclude the vote
> >> >> on
> >> >> the
> >> >> thread [VOTE]
> >> >> > [DISCUSSION] Remove support for Java 7?
> >> >> > Or should we take more steps - e.g. perhaps
> >> >> tweet a
> >> >> link
> >> >> to that thread
> >> >> > from the Beam twitter account, ask people to
> >> >> chime
> >> >> in,
> >> >> and wait for say
> >> >> > 2 weeks before declaring a conclusion?
> >> >> >
> >> >> > Let's also have a process JIRA for going
> Java8.
> >> >> I've
> >> >> filed one:
> >> >> > https://issues.apache.org/jira/browse/BEAM-3285
> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>
> >> >> > <https://issues.apache.org/
> jira/browse/BEAM-3285
> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>>
> >> >> >
> >> >> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste
> >> >> Onofré
> >> >> <jb@nanthrax.net <ma...@nanthrax.net>
> >> >> > <mailto:jb@nanthrax.net
> >> >> <ma...@nanthrax.net>>
> >> >> <mailto:jb@nanthrax.net <ma...@nanthrax.net>
> >> >> > <mailto:jb@nanthrax.net
> >> >> <ma...@nanthrax.net>>>>
> >> >> wrote:
> >> >> >
> >> >> > Just an important note that we forgot to
> >> >> mention.
> >> >> >
> >> >> > !! The 2.2.0 release will be the last one
> >> >> supporting
> >> >> Spark 1.x and
> >> >> > Java 7 !!
> >> >> >
> >> >> > Starting from Beam 2.3.0, the Spark
> runner
> >> >> will work
> >> >> only with
> >> >> > Spark 2.x and we
> >> >> > will focus only Java 8.
> >> >> >
> >> >> > Regards
> >> >> > JB
> >> >> >
> >> >> > On 12/04/2017 10:15 AM, Jean-Baptiste
> >> >> Onofré
> >> >> wrote:
> >> >> > > Thanks Reuven !
> >> >> > >
> >> >> > > I would like to emphasize on some
> >> >> highlights in
> >> >> 2.2.0 release:
> >> >> > >
> >> >> > > - New IOs have been introduced:
> >> >> > > * TikaIO leveraging Apache Tika,
> >> >> allowing
> >> >> the
> >> >> deal with a lot
> >> >> > of different
> >> >> > > data formats
> >> >> > > * RedisIO to read and write
> key/value
> >> >> pairs
> >> >> from a Redis
> >> >> > server. This
> >> >> > IO will
> >> >> > > be soon extended to Redis PubSub.
> >> >> > > * FileIO provides transforms for
> >> >> working
> >> >> with
> >> >> files (raw).
> >> >> > Especially, it
> >> >> > > provides matching file patterns and
> read
> >> >> on
> >> >> patterns. It can be
> >> >> > easily
> >> >> > extended
> >> >> > > for a specific format (like we do in
> >> >> AvroIO
> >> >> or
> >> >> TextIO now).
> >> >> > > * SolrIO to interact with Apache
> Solr
> >> >> (Lucene)
> >> >> > >
> >> >> > > - On the other hand, improvements have
> >> >> been
> >> >> performed on
> >> >> > existing IOs:
> >> >> > > * We started to introduce readAll
> >> >> pattern
> >> >> in
> >> >> IOs (AvroIO,
> >> >> > TextIO, JdbcIO,
> >> >> > > ...), allowing to pass "request"
> >> >> arguments
> >> >> via an
> >> >> input PCollection.
> >> >> > > * ElasticsearchIO has an improved
> >> >> support
> >> >> of
> >> >> different
> >> >> > Elasticsearch
> >> >> > version
> >> >> > > (including Elasticsearch 5.x). It also
> >> >> now
> >> >> supports SSL/TLS.
> >> >> > > * HBaseIO is now able to do dynamic
> >> >> work
> >> >> rebalancing
> >> >> > > * KinesisIO uses a more accurate
> >> >> watermark
> >> >> (based on
> >> >> > approximateArrivalTimestamp)
> >> >> > > * TextIO now supports custom
> delimiter
> >> >> and like
> >> >> AvroIO,
> >> >> > supports the
> >> >> > readAll
> >> >> > > pattern,
> >> >> > > * Performance improvements on JdbcIO
> >> >> when
> >> >> it
> >> >> has to read lot
> >> >> > of rows
> >> >> > > * Kafka write supports Exactly-Once
> >> >> pattern
> >> >> (introduce in
> >> >> > Kafka 0.11.x)
> >> >> > >
> >> >> > > - A new DSL has been introduced: the
> SQL
> >> >> DSL !
> >> >> > >
> >> >> > > We are now focus on 2.3.0 release with
> >> >> new
> >> >> improvements and
> >> >> > features !
> >> >> > >
> >> >> > > Stay tuned !
> >> >> > >
> >> >> > > JB on behalf of the Apache Beam
> >> >> community.
> >> >> > >
> >> >> > > On 12/02/2017 11:40 PM, Reuven Lax
> >> >> wrote:
> >> >> > >> The Apache Beam community is pleased
> to
> >> >> announce the
> >> >> > availability of the
> >> >> > >> 2.2.0 release.
> >> >> > >>
> >> >> > >> This release adds support for generic
> >> >> file
> >> >> sources and sinks
> >> >> > (beyond TextIO
> >> >> > >> and AvroIO) using FileIO, including
> >> >> support for
> >> >> dynamic
> >> >> > filenames using
> >> >> > >> readAll; this allows streaming
> >> >> pipelines
> >> >> to now
> >> >> read from files by
> >> >> > >> continuously monitoring a directory
> for
> >> >> new
> >> >> filw. Many other
> >> >> > IOs are
> >> >> > improved,
> >> >> > >> notably including exactly-once
> support
> >> >> for
> >> >> the
> >> >> Kafka sink. Initial
> >> >> > support for
> >> >> > >> BEAM-SQL is also included in this
> >> >> release.
> >> >> For a
> >> >> more-complete
> >> >> > list of major
> >> >> > >> changes in the release, please refer
> to
> >> >> the
> >> >> release notes [2].
> >> >> > >>
> >> >> > >> The 2.2.0 release is now the
> >> >> recommended
> >> >> version; we encourage
> >> >> > everyone to
> >> >> > >> upgrade from any earlier releases.
> >> >> > >>
> >> >> > >> We’d like to invite everyone to try
> out
> >> >> Apache
> >> >> Beam today and
> >> >> > consider
> >> >> > >> joining our vibrant community. We
> >> >> welcome
> >> >> feedback,
> >> >> > contribution and
> >> >> > >> participation through our mailing
> >> >> lists,
> >> >> issue
> >> >> tracker, pull
> >> >> > requests, and
> >> >> > >> events.
> >> >> > >>
> >> >> > >> - Reuven Lax, on behalf of the Apache
> >> >> Beam
> >> >> community.
> >> >> > >>
> >> >> > >> [1]
> >> >> https://beam.apache.org/get-started/downloads/
> >> >> <https://beam.apache.org/get-started/downloads/>
> >> >> > <https://beam.apache.org/get-
> started/downloads/
> >> >> <https://beam.apache.org/get-started/downloads/>>
> >> >> > >> [2]
> >> >> > >>
> >> >> >
> >> >>
> >> >>
> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527&version=12341044
> >> >>
> >> >>
> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527&version=12341044>
> >> >> >
> >> >>
> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527&version=12341044
> >> >>
> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527&version=12341044>>
> >> >> > >>
> >> >> > >
> >> >> >
> >> >> > --
> >> >> > Jean-Baptiste Onofré
> >> >> > jbonofre@apache.org <ma...@apache.org>
> >> >> <mailto:jbonofre@apache.org <mailto:jbonofre@apache.org
> >>
> >> >> > <mailto:jbonofre@apache.org
> >> >> <ma...@apache.org>
> >> >> <mailto:jbonofre@apache.org <mailto:jbonofre@apache.org
> >>>
> >> >> > http://blog.nanthrax.net
> >> >> > Talend - http://www.talend.com
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Jean-Baptiste Onofré
> >> >> > jbonofre@apache.org <ma...@apache.org>
> >> >> <mailto:jbonofre@apache.org <mailto:jbonofre@apache.org
> >>
> >> >> > http://blog.nanthrax.net
> >> >> > Talend - http://www.talend.com
> >> >> >
> >> >> >
> >> >>
> >> >> --
> >> >> Jean-Baptiste Onofré
> >> >> jbonofre@apache.org <ma...@apache.org>
> >> >> http://blog.nanthrax.net
> >> >> Talend - http://www.talend.com
> >> >>
> >> >>
> >> >>
> >> >
> >> > --
> >> > Jean-Baptiste Onofré
> >> > jbonofre@apache.org
> >> > http://blog.nanthrax.net
> >> > Talend - http://www.talend.com
>
Re: Apache Beam, version 2.2.0
Posted by Robert Bradshaw <ro...@google.com>.
+1 to moving forward with this plan.
(FWIW, this seems *less* backwards incompatible than, say, moving from
Spark 1 to Spark 2, which was decided much quicker. I suppose the
Spark change has a lower bound on the number of users it could impact
though.)
On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <ki...@google.com> wrote:
> Okay, then let's go forward. Seems that we should:
> - Open a new poll on user@, in light of 2.2 having been released
> - Open a twitter poll
> - Tweet that there's also a poll going on on user@
> - Runner authors will reach out to respective runner user communities
> - 2 weeks later we gather results and decide
> ?
>
> On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far away,
>> and starting to improve Beam to offer a Java 8 friendly experience
>> seems like an excellent idea.
>>
>> I understand the backwards compatibility argument. We should do the
>> poll in twitter + try to reach more users for comments. If you
>> consider that it is worth, I can open a second poll at user@.
>>
>> In any case we should try to move forward, even if we have more than
>> 5% of users who want to stay on Java 7 we can consider to maintain
>> minor releases of a backwards compatible version where we can backport
>> only critical fixes e.g. security/data related errors but nothing new,
>> in case some user really needs to have them. Of course this can be
>> some extra work (to be discussed).
>>
>>
>> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>> > +1, and sorry again, I thought we got an consensus.
>> >
>> > Regards
>> > JB
>> >
>> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote:
>> >>
>> >> +1 to the poll and also to Reuven's point.
>> >>
>> >> Those without a support contract would have been using JDK 7 without
>> >> security updates for years. IMO it seems harmful, as a netizen, to
>> >> encourage
>> >> its use/existence.
>> >>
>> >> If there's no noise from the prior thread, then I would assume no one
>> >> on
>> >> user@ has any objection. Anyone else with customers should reach out to
>> >> them.
>> >>
>> >> Kenn
>> >>
>> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <relax@google.com
>> >> <ma...@google.com>> wrote:
>> >>
>> >> Technically it's a backwards-incompatible change, however if we are
>> >> convinced the risk is low we could do it.
>> >>
>> >> As mentioned on the original thread, it's not clear that all Beam
>> >> users read
>> >> user@ - e.g. most Dataflow users definitely do not. I think we need
>> >> to
>> >> separately reach out to users of each runner through
>> >> runner-specific
>> >> channels.
>> >>
>> >> Reuven
>> >>
>> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov
>> >> <kirpichov@google.com
>> >> <ma...@google.com>> wrote:
>> >>
>> >> On the original thread
>> >>
>> >>
>> >> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
>> >>
>> >>
>> >> <https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E>
>> >> ,
>> >> Robert and Ismaël were in favor of no major version change
>> >> [Ismaël
>> >> said:/Also I am afraid that if we wait/
>> >> /until we have enough changes to switch Beam to a new major
>> >> version the
>> >> switch to Java 8 will happen too late, probably after Java 8's
>> >> end
>> >> of
>> >> life. And I am not exaggerating, Java 8 is planned to EOL next
>> >> march
>> >> 2018!/]; JB and now Reuven are in favor of a major version
>> >> change;
>> >>
>> >> nobody so far argued against switching to Java8 in general.
>> >>
>> >> I'm personally in favor of no major version change (i.e. not
>> >> waiting
>> >> until all other large changes for Beam 3.0 converge, which will
>> >> likely
>> >> be many months), because:
>> >> - Reasons Ismaël cited; plus the reason that most people are
>> >> likely
>> >> already using Java 8.
>> >> - Going Java8-only earlier will make other Beam 3.0 APIs better
>> >> for
>> >> Java8 users, because we (Beam contributors) will have
>> >> experience
>> >> working
>> >> with them within the SDK in Java8 (e.g. writing tests with use
>> >> of
>> >> lambdas and noticing whether it's clunky, or whether some other
>> >> Beam
>> >> APIs need better Java8 support).
>> >> - Going Java8 will make it more reasonable to include (mostly
>> >> or
>> >> only)
>> >> Java8 snippets in Beam documentation, which will obviously look
>> >> more
>> >> concise and attractive, addressing one of the common concerns
>> >> of
>> >> Beam
>> >> users that it has a heavyweight API compared to
>> >> functional-style
>> >> APIs of
>> >> Spark etc.
>> >>
>> >> I think resolving this via a poll of users would be reasonable.
>> >> I'd
>> >> suggest e.g. the following phrasing:
>> >>
>> >> Apache Beam is considering dropping support for Java 7, and
>> >> supporting
>> >> only Java 8 and above in a subsequent release. How would it
>> >> impact
>> >> your
>> >> usage of Beam?
>> >> - I am already using only Java 8+ for building my Beam code
>> >> - I am using Java 7 for building my Beam code, but I would have
>> >> no
>> >> trouble switching to Java 8
>> >> - I am using Java 7 for building my Beam code, and dropping
>> >> Java 7
>> >> would
>> >> be a blocker or hindrance to adopting the new release for me
>> >>
>> >> We could tweet this poll on Apache Beam twitter and publish on
>> >> user@,
>> >> and, say, if we receive 5% or fewer votes for option 3 after
>> >> keeping it
>> >> open for 2 weeks, then adopt Java 8 without a major version
>> >> change.
>> >>
>> >> WDYT?
>> >>
>> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré
>> >> <jb@nanthrax.net
>> >> <ma...@nanthrax.net>> wrote:
>> >>
>> >> Good idea ! Definitely +1
>> >>
>> >> Regards
>> >> JB
>> >>
>> >> On 12/05/2017 05:25 AM, Reuven Lax wrote:
>> >> > We should bring this up on the Beam 3.0 thread. Since
>> >> it's
>> >> technically a
>> >> > backwards-incompatible change, it might make a good item
>> >> for Beam
>> >> 3.0.
>> >> >
>> >> > Reuven
>> >> >
>> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré
>> >> <jb@nanthrax.net <ma...@nanthrax.net>
>> >> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>>
>> >> wrote:
>> >> >
>> >> > My apologizes, I thought we had a consensus already.
>> >> >
>> >> > Regards
>> >> > JB
>> >> >
>> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
>> >> >
>> >> > Thanks JB for sending the detailed notes about
>> >> new
>> >> stuff
>> >> in 2.2.0! A lot
>> >> > of exciting things indeed.
>> >> >
>> >> > Regarding Java 8: I thought our consensus was to
>> >> have the
>> >> release notes
>> >> > say that we're *considering* going Java8-only,
>> >> and
>> >> use
>> >> that to get more
>> >> > opinions from the user community - but I can't
>> >> find
>> >> the
>> >> emails that made
>> >> > me think so.
>> >> >
>> >> > +Ismaël Mejía <mailto:iemejia@gmail.com
>> >> <ma...@gmail.com> <mailto:iemejia@gmail.com
>> >> <ma...@gmail.com>>> - do
>> >> > you think we should formally conclude the vote
>> >> on
>> >> the
>> >> thread [VOTE]
>> >> > [DISCUSSION] Remove support for Java 7?
>> >> > Or should we take more steps - e.g. perhaps
>> >> tweet a
>> >> link
>> >> to that thread
>> >> > from the Beam twitter account, ask people to
>> >> chime
>> >> in,
>> >> and wait for say
>> >> > 2 weeks before declaring a conclusion?
>> >> >
>> >> > Let's also have a process JIRA for going Java8.
>> >> I've
>> >> filed one:
>> >> > https://issues.apache.org/jira/browse/BEAM-3285
>> >> <https://issues.apache.org/jira/browse/BEAM-3285>
>> >> > <https://issues.apache.org/jira/browse/BEAM-3285
>> >> <https://issues.apache.org/jira/browse/BEAM-3285>>
>> >> >
>> >> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste
>> >> Onofré
>> >> <jb@nanthrax.net <ma...@nanthrax.net>
>> >> > <mailto:jb@nanthrax.net
>> >> <ma...@nanthrax.net>>
>> >> <mailto:jb@nanthrax.net <ma...@nanthrax.net>
>> >> > <mailto:jb@nanthrax.net
>> >> <ma...@nanthrax.net>>>>
>> >> wrote:
>> >> >
>> >> > Just an important note that we forgot to
>> >> mention.
>> >> >
>> >> > !! The 2.2.0 release will be the last one
>> >> supporting
>> >> Spark 1.x and
>> >> > Java 7 !!
>> >> >
>> >> > Starting from Beam 2.3.0, the Spark runner
>> >> will work
>> >> only with
>> >> > Spark 2.x and we
>> >> > will focus only Java 8.
>> >> >
>> >> > Regards
>> >> > JB
>> >> >
>> >> > On 12/04/2017 10:15 AM, Jean-Baptiste
>> >> Onofré
>> >> wrote:
>> >> > > Thanks Reuven !
>> >> > >
>> >> > > I would like to emphasize on some
>> >> highlights in
>> >> 2.2.0 release:
>> >> > >
>> >> > > - New IOs have been introduced:
>> >> > > * TikaIO leveraging Apache Tika,
>> >> allowing
>> >> the
>> >> deal with a lot
>> >> > of different
>> >> > > data formats
>> >> > > * RedisIO to read and write key/value
>> >> pairs
>> >> from a Redis
>> >> > server. This
>> >> > IO will
>> >> > > be soon extended to Redis PubSub.
>> >> > > * FileIO provides transforms for
>> >> working
>> >> with
>> >> files (raw).
>> >> > Especially, it
>> >> > > provides matching file patterns and read
>> >> on
>> >> patterns. It can be
>> >> > easily
>> >> > extended
>> >> > > for a specific format (like we do in
>> >> AvroIO
>> >> or
>> >> TextIO now).
>> >> > > * SolrIO to interact with Apache Solr
>> >> (Lucene)
>> >> > >
>> >> > > - On the other hand, improvements have
>> >> been
>> >> performed on
>> >> > existing IOs:
>> >> > > * We started to introduce readAll
>> >> pattern
>> >> in
>> >> IOs (AvroIO,
>> >> > TextIO, JdbcIO,
>> >> > > ...), allowing to pass "request"
>> >> arguments
>> >> via an
>> >> input PCollection.
>> >> > > * ElasticsearchIO has an improved
>> >> support
>> >> of
>> >> different
>> >> > Elasticsearch
>> >> > version
>> >> > > (including Elasticsearch 5.x). It also
>> >> now
>> >> supports SSL/TLS.
>> >> > > * HBaseIO is now able to do dynamic
>> >> work
>> >> rebalancing
>> >> > > * KinesisIO uses a more accurate
>> >> watermark
>> >> (based on
>> >> > approximateArrivalTimestamp)
>> >> > > * TextIO now supports custom delimiter
>> >> and like
>> >> AvroIO,
>> >> > supports the
>> >> > readAll
>> >> > > pattern,
>> >> > > * Performance improvements on JdbcIO
>> >> when
>> >> it
>> >> has to read lot
>> >> > of rows
>> >> > > * Kafka write supports Exactly-Once
>> >> pattern
>> >> (introduce in
>> >> > Kafka 0.11.x)
>> >> > >
>> >> > > - A new DSL has been introduced: the SQL
>> >> DSL !
>> >> > >
>> >> > > We are now focus on 2.3.0 release with
>> >> new
>> >> improvements and
>> >> > features !
>> >> > >
>> >> > > Stay tuned !
>> >> > >
>> >> > > JB on behalf of the Apache Beam
>> >> community.
>> >> > >
>> >> > > On 12/02/2017 11:40 PM, Reuven Lax
>> >> wrote:
>> >> > >> The Apache Beam community is pleased to
>> >> announce the
>> >> > availability of the
>> >> > >> 2.2.0 release.
>> >> > >>
>> >> > >> This release adds support for generic
>> >> file
>> >> sources and sinks
>> >> > (beyond TextIO
>> >> > >> and AvroIO) using FileIO, including
>> >> support for
>> >> dynamic
>> >> > filenames using
>> >> > >> readAll; this allows streaming
>> >> pipelines
>> >> to now
>> >> read from files by
>> >> > >> continuously monitoring a directory for
>> >> new
>> >> filw. Many other
>> >> > IOs are
>> >> > improved,
>> >> > >> notably including exactly-once support
>> >> for
>> >> the
>> >> Kafka sink. Initial
>> >> > support for
>> >> > >> BEAM-SQL is also included in this
>> >> release.
>> >> For a
>> >> more-complete
>> >> > list of major
>> >> > >> changes in the release, please refer to
>> >> the
>> >> release notes [2].
>> >> > >>
>> >> > >> The 2.2.0 release is now the
>> >> recommended
>> >> version; we encourage
>> >> > everyone to
>> >> > >> upgrade from any earlier releases.
>> >> > >>
>> >> > >> We’d like to invite everyone to try out
>> >> Apache
>> >> Beam today and
>> >> > consider
>> >> > >> joining our vibrant community. We
>> >> welcome
>> >> feedback,
>> >> > contribution and
>> >> > >> participation through our mailing
>> >> lists,
>> >> issue
>> >> tracker, pull
>> >> > requests, and
>> >> > >> events.
>> >> > >>
>> >> > >> - Reuven Lax, on behalf of the Apache
>> >> Beam
>> >> community.
>> >> > >>
>> >> > >> [1]
>> >> https://beam.apache.org/get-started/downloads/
>> >> <https://beam.apache.org/get-started/downloads/>
>> >> > <https://beam.apache.org/get-started/downloads/
>> >> <https://beam.apache.org/get-started/downloads/>>
>> >> > >> [2]
>> >> > >>
>> >> >
>> >>
>> >>
>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
>> >>
>> >>
>> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044>
>> >> >
>> >>
>> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
>> >>
>> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044>>
>> >> > >>
>> >> > >
>> >> >
>> >> > --
>> >> > Jean-Baptiste Onofré
>> >> > jbonofre@apache.org <ma...@apache.org>
>> >> <mailto:jbonofre@apache.org <ma...@apache.org>>
>> >> > <mailto:jbonofre@apache.org
>> >> <ma...@apache.org>
>> >> <mailto:jbonofre@apache.org <ma...@apache.org>>>
>> >> > http://blog.nanthrax.net
>> >> > Talend - http://www.talend.com
>> >> >
>> >> >
>> >> > --
>> >> > Jean-Baptiste Onofré
>> >> > jbonofre@apache.org <ma...@apache.org>
>> >> <mailto:jbonofre@apache.org <ma...@apache.org>>
>> >> > http://blog.nanthrax.net
>> >> > Talend - http://www.talend.com
>> >> >
>> >> >
>> >>
>> >> --
>> >> Jean-Baptiste Onofré
>> >> jbonofre@apache.org <ma...@apache.org>
>> >> http://blog.nanthrax.net
>> >> Talend - http://www.talend.com
>> >>
>> >>
>> >>
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
Re: Apache Beam, version 2.2.0
Posted by Eugene Kirpichov <ki...@google.com>.
Okay, then let's go forward. Seems that we should:
- Open a new poll on user@, in light of 2.2 having been released
- Open a twitter poll
- Tweet that there's also a poll going on on user@
- Runner authors will reach out to respective runner user communities
- 2 weeks later we gather results and decide
?
On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <ie...@gmail.com> wrote:
> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far away,
> and starting to improve Beam to offer a Java 8 friendly experience
> seems like an excellent idea.
>
> I understand the backwards compatibility argument. We should do the
> poll in twitter + try to reach more users for comments. If you
> consider that it is worth, I can open a second poll at user@.
>
> In any case we should try to move forward, even if we have more than
> 5% of users who want to stay on Java 7 we can consider to maintain
> minor releases of a backwards compatible version where we can backport
> only critical fixes e.g. security/data related errors but nothing new,
> in case some user really needs to have them. Of course this can be
> some extra work (to be discussed).
>
>
> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> > +1, and sorry again, I thought we got an consensus.
> >
> > Regards
> > JB
> >
> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote:
> >>
> >> +1 to the poll and also to Reuven's point.
> >>
> >> Those without a support contract would have been using JDK 7 without
> >> security updates for years. IMO it seems harmful, as a netizen, to
> encourage
> >> its use/existence.
> >>
> >> If there's no noise from the prior thread, then I would assume no one on
> >> user@ has any objection. Anyone else with customers should reach out to
> >> them.
> >>
> >> Kenn
> >>
> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <relax@google.com
> >> <ma...@google.com>> wrote:
> >>
> >> Technically it's a backwards-incompatible change, however if we are
> >> convinced the risk is low we could do it.
> >>
> >> As mentioned on the original thread, it's not clear that all Beam
> >> users read
> >> user@ - e.g. most Dataflow users definitely do not. I think we
> need to
> >> separately reach out to users of each runner through runner-specific
> >> channels.
> >>
> >> Reuven
> >>
> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov <
> kirpichov@google.com
> >> <ma...@google.com>> wrote:
> >>
> >> On the original thread
> >>
> >>
> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
> >>
> >> <
> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
> >
> >> ,
> >> Robert and Ismaël were in favor of no major version change
> [Ismaël
> >> said:/Also I am afraid that if we wait/
> >> /until we have enough changes to switch Beam to a new major
> >> version the
> >> switch to Java 8 will happen too late, probably after Java 8's
> end
> >> of
> >> life. And I am not exaggerating, Java 8 is planned to EOL next
> >> march
> >> 2018!/]; JB and now Reuven are in favor of a major version
> change;
> >>
> >> nobody so far argued against switching to Java8 in general.
> >>
> >> I'm personally in favor of no major version change (i.e. not
> >> waiting
> >> until all other large changes for Beam 3.0 converge, which will
> >> likely
> >> be many months), because:
> >> - Reasons Ismaël cited; plus the reason that most people are
> >> likely
> >> already using Java 8.
> >> - Going Java8-only earlier will make other Beam 3.0 APIs better
> >> for
> >> Java8 users, because we (Beam contributors) will have experience
> >> working
> >> with them within the SDK in Java8 (e.g. writing tests with use
> of
> >> lambdas and noticing whether it's clunky, or whether some other
> >> Beam
> >> APIs need better Java8 support).
> >> - Going Java8 will make it more reasonable to include (mostly or
> >> only)
> >> Java8 snippets in Beam documentation, which will obviously look
> >> more
> >> concise and attractive, addressing one of the common concerns of
> >> Beam
> >> users that it has a heavyweight API compared to functional-style
> >> APIs of
> >> Spark etc.
> >>
> >> I think resolving this via a poll of users would be reasonable.
> >> I'd
> >> suggest e.g. the following phrasing:
> >>
> >> Apache Beam is considering dropping support for Java 7, and
> >> supporting
> >> only Java 8 and above in a subsequent release. How would it
> impact
> >> your
> >> usage of Beam?
> >> - I am already using only Java 8+ for building my Beam code
> >> - I am using Java 7 for building my Beam code, but I would have
> no
> >> trouble switching to Java 8
> >> - I am using Java 7 for building my Beam code, and dropping
> Java 7
> >> would
> >> be a blocker or hindrance to adopting the new release for me
> >>
> >> We could tweet this poll on Apache Beam twitter and publish on
> >> user@,
> >> and, say, if we receive 5% or fewer votes for option 3 after
> >> keeping it
> >> open for 2 weeks, then adopt Java 8 without a major version
> >> change.
> >>
> >> WDYT?
> >>
> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré
> >> <jb@nanthrax.net
> >> <ma...@nanthrax.net>> wrote:
> >>
> >> Good idea ! Definitely +1
> >>
> >> Regards
> >> JB
> >>
> >> On 12/05/2017 05:25 AM, Reuven Lax wrote:
> >> > We should bring this up on the Beam 3.0 thread. Since
> it's
> >> technically a
> >> > backwards-incompatible change, it might make a good item
> >> for Beam
> >> 3.0.
> >> >
> >> > Reuven
> >> >
> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré
> >> <jb@nanthrax.net <ma...@nanthrax.net>
> >> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>>
> wrote:
> >> >
> >> > My apologizes, I thought we had a consensus already.
> >> >
> >> > Regards
> >> > JB
> >> >
> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
> >> >
> >> > Thanks JB for sending the detailed notes about
> new
> >> stuff
> >> in 2.2.0! A lot
> >> > of exciting things indeed.
> >> >
> >> > Regarding Java 8: I thought our consensus was to
> >> have the
> >> release notes
> >> > say that we're *considering* going Java8-only,
> and
> >> use
> >> that to get more
> >> > opinions from the user community - but I can't
> find
> >> the
> >> emails that made
> >> > me think so.
> >> >
> >> > +Ismaël Mejía <mailto:iemejia@gmail.com
> >> <ma...@gmail.com> <mailto:iemejia@gmail.com
> >> <ma...@gmail.com>>> - do
> >> > you think we should formally conclude the vote on
> >> the
> >> thread [VOTE]
> >> > [DISCUSSION] Remove support for Java 7?
> >> > Or should we take more steps - e.g. perhaps
> tweet a
> >> link
> >> to that thread
> >> > from the Beam twitter account, ask people to
> chime
> >> in,
> >> and wait for say
> >> > 2 weeks before declaring a conclusion?
> >> >
> >> > Let's also have a process JIRA for going Java8.
> >> I've
> >> filed one:
> >> > https://issues.apache.org/jira/browse/BEAM-3285
> >> <https://issues.apache.org/jira/browse/BEAM-3285>
> >> > <https://issues.apache.org/jira/browse/BEAM-3285
> >> <https://issues.apache.org/jira/browse/BEAM-3285>>
> >> >
> >> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste
> Onofré
> >> <jb@nanthrax.net <ma...@nanthrax.net>
> >> > <mailto:jb@nanthrax.net <mailto:jb@nanthrax.net
> >>
> >> <mailto:jb@nanthrax.net <ma...@nanthrax.net>
> >> > <mailto:jb@nanthrax.net <mailto:jb@nanthrax.net
> >>>>
> >> wrote:
> >> >
> >> > Just an important note that we forgot to
> >> mention.
> >> >
> >> > !! The 2.2.0 release will be the last one
> >> supporting
> >> Spark 1.x and
> >> > Java 7 !!
> >> >
> >> > Starting from Beam 2.3.0, the Spark runner
> >> will work
> >> only with
> >> > Spark 2.x and we
> >> > will focus only Java 8.
> >> >
> >> > Regards
> >> > JB
> >> >
> >> > On 12/04/2017 10:15 AM, Jean-Baptiste Onofré
> >> wrote:
> >> > > Thanks Reuven !
> >> > >
> >> > > I would like to emphasize on some
> >> highlights in
> >> 2.2.0 release:
> >> > >
> >> > > - New IOs have been introduced:
> >> > > * TikaIO leveraging Apache Tika,
> allowing
> >> the
> >> deal with a lot
> >> > of different
> >> > > data formats
> >> > > * RedisIO to read and write key/value
> >> pairs
> >> from a Redis
> >> > server. This
> >> > IO will
> >> > > be soon extended to Redis PubSub.
> >> > > * FileIO provides transforms for
> working
> >> with
> >> files (raw).
> >> > Especially, it
> >> > > provides matching file patterns and read
> on
> >> patterns. It can be
> >> > easily
> >> > extended
> >> > > for a specific format (like we do in
> AvroIO
> >> or
> >> TextIO now).
> >> > > * SolrIO to interact with Apache Solr
> >> (Lucene)
> >> > >
> >> > > - On the other hand, improvements have
> been
> >> performed on
> >> > existing IOs:
> >> > > * We started to introduce readAll
> pattern
> >> in
> >> IOs (AvroIO,
> >> > TextIO, JdbcIO,
> >> > > ...), allowing to pass "request"
> arguments
> >> via an
> >> input PCollection.
> >> > > * ElasticsearchIO has an improved
> support
> >> of
> >> different
> >> > Elasticsearch
> >> > version
> >> > > (including Elasticsearch 5.x). It also
> now
> >> supports SSL/TLS.
> >> > > * HBaseIO is now able to do dynamic
> work
> >> rebalancing
> >> > > * KinesisIO uses a more accurate
> >> watermark
> >> (based on
> >> > approximateArrivalTimestamp)
> >> > > * TextIO now supports custom delimiter
> >> and like
> >> AvroIO,
> >> > supports the
> >> > readAll
> >> > > pattern,
> >> > > * Performance improvements on JdbcIO
> when
> >> it
> >> has to read lot
> >> > of rows
> >> > > * Kafka write supports Exactly-Once
> >> pattern
> >> (introduce in
> >> > Kafka 0.11.x)
> >> > >
> >> > > - A new DSL has been introduced: the SQL
> >> DSL !
> >> > >
> >> > > We are now focus on 2.3.0 release with
> new
> >> improvements and
> >> > features !
> >> > >
> >> > > Stay tuned !
> >> > >
> >> > > JB on behalf of the Apache Beam
> community.
> >> > >
> >> > > On 12/02/2017 11:40 PM, Reuven Lax wrote:
> >> > >> The Apache Beam community is pleased to
> >> announce the
> >> > availability of the
> >> > >> 2.2.0 release.
> >> > >>
> >> > >> This release adds support for generic
> file
> >> sources and sinks
> >> > (beyond TextIO
> >> > >> and AvroIO) using FileIO, including
> >> support for
> >> dynamic
> >> > filenames using
> >> > >> readAll; this allows streaming pipelines
> >> to now
> >> read from files by
> >> > >> continuously monitoring a directory for
> >> new
> >> filw. Many other
> >> > IOs are
> >> > improved,
> >> > >> notably including exactly-once support
> for
> >> the
> >> Kafka sink. Initial
> >> > support for
> >> > >> BEAM-SQL is also included in this
> release.
> >> For a
> >> more-complete
> >> > list of major
> >> > >> changes in the release, please refer to
> >> the
> >> release notes [2].
> >> > >>
> >> > >> The 2.2.0 release is now the recommended
> >> version; we encourage
> >> > everyone to
> >> > >> upgrade from any earlier releases.
> >> > >>
> >> > >> We’d like to invite everyone to try out
> >> Apache
> >> Beam today and
> >> > consider
> >> > >> joining our vibrant community. We
> welcome
> >> feedback,
> >> > contribution and
> >> > >> participation through our mailing lists,
> >> issue
> >> tracker, pull
> >> > requests, and
> >> > >> events.
> >> > >>
> >> > >> - Reuven Lax, on behalf of the Apache
> Beam
> >> community.
> >> > >>
> >> > >> [1]
> >> https://beam.apache.org/get-started/downloads/
> >> <https://beam.apache.org/get-started/downloads/>
> >> > <https://beam.apache.org/get-started/downloads/
> >> <https://beam.apache.org/get-started/downloads/>>
> >> > >> [2]
> >> > >>
> >> >
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
> >>
> >> <
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
> >
> >> >
> >> <
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
> >> <
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
> >>
> >> > >>
> >> > >
> >> >
> >> > --
> >> > Jean-Baptiste Onofré
> >> > jbonofre@apache.org <ma...@apache.org>
> >> <mailto:jbonofre@apache.org <ma...@apache.org>>
> >> > <mailto:jbonofre@apache.org
> >> <ma...@apache.org>
> >> <mailto:jbonofre@apache.org <ma...@apache.org>>>
> >> > http://blog.nanthrax.net
> >> > Talend - http://www.talend.com
> >> >
> >> >
> >> > --
> >> > Jean-Baptiste Onofré
> >> > jbonofre@apache.org <ma...@apache.org>
> >> <mailto:jbonofre@apache.org <ma...@apache.org>>
> >> > http://blog.nanthrax.net
> >> > Talend - http://www.talend.com
> >> >
> >> >
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org <ma...@apache.org>
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >>
> >>
> >>
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
Re: Apache Beam, version 2.2.0
Posted by Ismaël Mejía <ie...@gmail.com>.
+1 For Eugene’s arguments waiting for Beam 3.0 seems still far away,
and starting to improve Beam to offer a Java 8 friendly experience
seems like an excellent idea.
I understand the backwards compatibility argument. We should do the
poll in twitter + try to reach more users for comments. If you
consider that it is worth, I can open a second poll at user@.
In any case we should try to move forward, even if we have more than
5% of users who want to stay on Java 7 we can consider to maintain
minor releases of a backwards compatible version where we can backport
only critical fixes e.g. security/data related errors but nothing new,
in case some user really needs to have them. Of course this can be
some extra work (to be discussed).
On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> +1, and sorry again, I thought we got an consensus.
>
> Regards
> JB
>
> On 12/05/2017 07:10 AM, Kenneth Knowles wrote:
>>
>> +1 to the poll and also to Reuven's point.
>>
>> Those without a support contract would have been using JDK 7 without
>> security updates for years. IMO it seems harmful, as a netizen, to encourage
>> its use/existence.
>>
>> If there's no noise from the prior thread, then I would assume no one on
>> user@ has any objection. Anyone else with customers should reach out to
>> them.
>>
>> Kenn
>>
>> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <relax@google.com
>> <ma...@google.com>> wrote:
>>
>> Technically it's a backwards-incompatible change, however if we are
>> convinced the risk is low we could do it.
>>
>> As mentioned on the original thread, it's not clear that all Beam
>> users read
>> user@ - e.g. most Dataflow users definitely do not. I think we need to
>> separately reach out to users of each runner through runner-specific
>> channels.
>>
>> Reuven
>>
>> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov <kirpichov@google.com
>> <ma...@google.com>> wrote:
>>
>> On the original thread
>>
>> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
>>
>> <https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E>
>> ,
>> Robert and Ismaël were in favor of no major version change [Ismaël
>> said:/Also I am afraid that if we wait/
>> /until we have enough changes to switch Beam to a new major
>> version the
>> switch to Java 8 will happen too late, probably after Java 8's end
>> of
>> life. And I am not exaggerating, Java 8 is planned to EOL next
>> march
>> 2018!/]; JB and now Reuven are in favor of a major version change;
>>
>> nobody so far argued against switching to Java8 in general.
>>
>> I'm personally in favor of no major version change (i.e. not
>> waiting
>> until all other large changes for Beam 3.0 converge, which will
>> likely
>> be many months), because:
>> - Reasons Ismaël cited; plus the reason that most people are
>> likely
>> already using Java 8.
>> - Going Java8-only earlier will make other Beam 3.0 APIs better
>> for
>> Java8 users, because we (Beam contributors) will have experience
>> working
>> with them within the SDK in Java8 (e.g. writing tests with use of
>> lambdas and noticing whether it's clunky, or whether some other
>> Beam
>> APIs need better Java8 support).
>> - Going Java8 will make it more reasonable to include (mostly or
>> only)
>> Java8 snippets in Beam documentation, which will obviously look
>> more
>> concise and attractive, addressing one of the common concerns of
>> Beam
>> users that it has a heavyweight API compared to functional-style
>> APIs of
>> Spark etc.
>>
>> I think resolving this via a poll of users would be reasonable.
>> I'd
>> suggest e.g. the following phrasing:
>>
>> Apache Beam is considering dropping support for Java 7, and
>> supporting
>> only Java 8 and above in a subsequent release. How would it impact
>> your
>> usage of Beam?
>> - I am already using only Java 8+ for building my Beam code
>> - I am using Java 7 for building my Beam code, but I would have no
>> trouble switching to Java 8
>> - I am using Java 7 for building my Beam code, and dropping Java 7
>> would
>> be a blocker or hindrance to adopting the new release for me
>>
>> We could tweet this poll on Apache Beam twitter and publish on
>> user@,
>> and, say, if we receive 5% or fewer votes for option 3 after
>> keeping it
>> open for 2 weeks, then adopt Java 8 without a major version
>> change.
>>
>> WDYT?
>>
>> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré
>> <jb@nanthrax.net
>> <ma...@nanthrax.net>> wrote:
>>
>> Good idea ! Definitely +1
>>
>> Regards
>> JB
>>
>> On 12/05/2017 05:25 AM, Reuven Lax wrote:
>> > We should bring this up on the Beam 3.0 thread. Since it's
>> technically a
>> > backwards-incompatible change, it might make a good item
>> for Beam
>> 3.0.
>> >
>> > Reuven
>> >
>> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré
>> <jb@nanthrax.net <ma...@nanthrax.net>
>> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
>> >
>> > My apologizes, I thought we had a consensus already.
>> >
>> > Regards
>> > JB
>> >
>> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
>> >
>> > Thanks JB for sending the detailed notes about new
>> stuff
>> in 2.2.0! A lot
>> > of exciting things indeed.
>> >
>> > Regarding Java 8: I thought our consensus was to
>> have the
>> release notes
>> > say that we're *considering* going Java8-only, and
>> use
>> that to get more
>> > opinions from the user community - but I can't find
>> the
>> emails that made
>> > me think so.
>> >
>> > +Ismaël Mejía <mailto:iemejia@gmail.com
>> <ma...@gmail.com> <mailto:iemejia@gmail.com
>> <ma...@gmail.com>>> - do
>> > you think we should formally conclude the vote on
>> the
>> thread [VOTE]
>> > [DISCUSSION] Remove support for Java 7?
>> > Or should we take more steps - e.g. perhaps tweet a
>> link
>> to that thread
>> > from the Beam twitter account, ask people to chime
>> in,
>> and wait for say
>> > 2 weeks before declaring a conclusion?
>> >
>> > Let's also have a process JIRA for going Java8.
>> I've
>> filed one:
>> > https://issues.apache.org/jira/browse/BEAM-3285
>> <https://issues.apache.org/jira/browse/BEAM-3285>
>> > <https://issues.apache.org/jira/browse/BEAM-3285
>> <https://issues.apache.org/jira/browse/BEAM-3285>>
>> >
>> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré
>> <jb@nanthrax.net <ma...@nanthrax.net>
>> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>
>> <mailto:jb@nanthrax.net <ma...@nanthrax.net>
>> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>>>
>> wrote:
>> >
>> > Just an important note that we forgot to
>> mention.
>> >
>> > !! The 2.2.0 release will be the last one
>> supporting
>> Spark 1.x and
>> > Java 7 !!
>> >
>> > Starting from Beam 2.3.0, the Spark runner
>> will work
>> only with
>> > Spark 2.x and we
>> > will focus only Java 8.
>> >
>> > Regards
>> > JB
>> >
>> > On 12/04/2017 10:15 AM, Jean-Baptiste Onofré
>> wrote:
>> > > Thanks Reuven !
>> > >
>> > > I would like to emphasize on some
>> highlights in
>> 2.2.0 release:
>> > >
>> > > - New IOs have been introduced:
>> > > * TikaIO leveraging Apache Tika, allowing
>> the
>> deal with a lot
>> > of different
>> > > data formats
>> > > * RedisIO to read and write key/value
>> pairs
>> from a Redis
>> > server. This
>> > IO will
>> > > be soon extended to Redis PubSub.
>> > > * FileIO provides transforms for working
>> with
>> files (raw).
>> > Especially, it
>> > > provides matching file patterns and read on
>> patterns. It can be
>> > easily
>> > extended
>> > > for a specific format (like we do in AvroIO
>> or
>> TextIO now).
>> > > * SolrIO to interact with Apache Solr
>> (Lucene)
>> > >
>> > > - On the other hand, improvements have been
>> performed on
>> > existing IOs:
>> > > * We started to introduce readAll pattern
>> in
>> IOs (AvroIO,
>> > TextIO, JdbcIO,
>> > > ...), allowing to pass "request" arguments
>> via an
>> input PCollection.
>> > > * ElasticsearchIO has an improved support
>> of
>> different
>> > Elasticsearch
>> > version
>> > > (including Elasticsearch 5.x). It also now
>> supports SSL/TLS.
>> > > * HBaseIO is now able to do dynamic work
>> rebalancing
>> > > * KinesisIO uses a more accurate
>> watermark
>> (based on
>> > approximateArrivalTimestamp)
>> > > * TextIO now supports custom delimiter
>> and like
>> AvroIO,
>> > supports the
>> > readAll
>> > > pattern,
>> > > * Performance improvements on JdbcIO when
>> it
>> has to read lot
>> > of rows
>> > > * Kafka write supports Exactly-Once
>> pattern
>> (introduce in
>> > Kafka 0.11.x)
>> > >
>> > > - A new DSL has been introduced: the SQL
>> DSL !
>> > >
>> > > We are now focus on 2.3.0 release with new
>> improvements and
>> > features !
>> > >
>> > > Stay tuned !
>> > >
>> > > JB on behalf of the Apache Beam community.
>> > >
>> > > On 12/02/2017 11:40 PM, Reuven Lax wrote:
>> > >> The Apache Beam community is pleased to
>> announce the
>> > availability of the
>> > >> 2.2.0 release.
>> > >>
>> > >> This release adds support for generic file
>> sources and sinks
>> > (beyond TextIO
>> > >> and AvroIO) using FileIO, including
>> support for
>> dynamic
>> > filenames using
>> > >> readAll; this allows streaming pipelines
>> to now
>> read from files by
>> > >> continuously monitoring a directory for
>> new
>> filw. Many other
>> > IOs are
>> > improved,
>> > >> notably including exactly-once support for
>> the
>> Kafka sink. Initial
>> > support for
>> > >> BEAM-SQL is also included in this release.
>> For a
>> more-complete
>> > list of major
>> > >> changes in the release, please refer to
>> the
>> release notes [2].
>> > >>
>> > >> The 2.2.0 release is now the recommended
>> version; we encourage
>> > everyone to
>> > >> upgrade from any earlier releases.
>> > >>
>> > >> We’d like to invite everyone to try out
>> Apache
>> Beam today and
>> > consider
>> > >> joining our vibrant community. We welcome
>> feedback,
>> > contribution and
>> > >> participation through our mailing lists,
>> issue
>> tracker, pull
>> > requests, and
>> > >> events.
>> > >>
>> > >> - Reuven Lax, on behalf of the Apache Beam
>> community.
>> > >>
>> > >> [1]
>> https://beam.apache.org/get-started/downloads/
>> <https://beam.apache.org/get-started/downloads/>
>> > <https://beam.apache.org/get-started/downloads/
>> <https://beam.apache.org/get-started/downloads/>>
>> > >> [2]
>> > >>
>> >
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
>>
>> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044>
>> >
>> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
>> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044>>
>> > >>
>> > >
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org <ma...@apache.org>
>> <mailto:jbonofre@apache.org <ma...@apache.org>>
>> > <mailto:jbonofre@apache.org
>> <ma...@apache.org>
>> <mailto:jbonofre@apache.org <ma...@apache.org>>>
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>> >
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org <ma...@apache.org>
>> <mailto:jbonofre@apache.org <ma...@apache.org>>
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>> >
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org <ma...@apache.org>
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>>
>>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
Re: Apache Beam, version 2.2.0
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1, and sorry again, I thought we got an consensus.
Regards
JB
On 12/05/2017 07:10 AM, Kenneth Knowles wrote:
> +1 to the poll and also to Reuven's point.
>
> Those without a support contract would have been using JDK 7 without security
> updates for years. IMO it seems harmful, as a netizen, to encourage its
> use/existence.
>
> If there's no noise from the prior thread, then I would assume no one on user@
> has any objection. Anyone else with customers should reach out to them.
>
> Kenn
>
> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <relax@google.com
> <ma...@google.com>> wrote:
>
> Technically it's a backwards-incompatible change, however if we are
> convinced the risk is low we could do it.
>
> As mentioned on the original thread, it's not clear that all Beam users read
> user@ - e.g. most Dataflow users definitely do not. I think we need to
> separately reach out to users of each runner through runner-specific channels.
>
> Reuven
>
> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov <kirpichov@google.com
> <ma...@google.com>> wrote:
>
> On the original thread
> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
> <https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E> ,
> Robert and Ismaël were in favor of no major version change [Ismaël
> said:/Also I am afraid that if we wait/
> /until we have enough changes to switch Beam to a new major version the
> switch to Java 8 will happen too late, probably after Java 8's end of
> life. And I am not exaggerating, Java 8 is planned to EOL next march
> 2018!/]; JB and now Reuven are in favor of a major version change;
> nobody so far argued against switching to Java8 in general.
>
> I'm personally in favor of no major version change (i.e. not waiting
> until all other large changes for Beam 3.0 converge, which will likely
> be many months), because:
> - Reasons Ismaël cited; plus the reason that most people are likely
> already using Java 8.
> - Going Java8-only earlier will make other Beam 3.0 APIs better for
> Java8 users, because we (Beam contributors) will have experience working
> with them within the SDK in Java8 (e.g. writing tests with use of
> lambdas and noticing whether it's clunky, or whether some other Beam
> APIs need better Java8 support).
> - Going Java8 will make it more reasonable to include (mostly or only)
> Java8 snippets in Beam documentation, which will obviously look more
> concise and attractive, addressing one of the common concerns of Beam
> users that it has a heavyweight API compared to functional-style APIs of
> Spark etc.
>
> I think resolving this via a poll of users would be reasonable. I'd
> suggest e.g. the following phrasing:
>
> Apache Beam is considering dropping support for Java 7, and supporting
> only Java 8 and above in a subsequent release. How would it impact your
> usage of Beam?
> - I am already using only Java 8+ for building my Beam code
> - I am using Java 7 for building my Beam code, but I would have no
> trouble switching to Java 8
> - I am using Java 7 for building my Beam code, and dropping Java 7 would
> be a blocker or hindrance to adopting the new release for me
>
> We could tweet this poll on Apache Beam twitter and publish on user@,
> and, say, if we receive 5% or fewer votes for option 3 after keeping it
> open for 2 weeks, then adopt Java 8 without a major version change.
>
> WDYT?
>
> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> Good idea ! Definitely +1
>
> Regards
> JB
>
> On 12/05/2017 05:25 AM, Reuven Lax wrote:
> > We should bring this up on the Beam 3.0 thread. Since it's
> technically a
> > backwards-incompatible change, it might make a good item for Beam
> 3.0.
> >
> > Reuven
> >
> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré
> <jb@nanthrax.net <ma...@nanthrax.net>
> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
> >
> > My apologizes, I thought we had a consensus already.
> >
> > Regards
> > JB
> >
> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
> >
> > Thanks JB for sending the detailed notes about new stuff
> in 2.2.0! A lot
> > of exciting things indeed.
> >
> > Regarding Java 8: I thought our consensus was to have the
> release notes
> > say that we're *considering* going Java8-only, and use
> that to get more
> > opinions from the user community - but I can't find the
> emails that made
> > me think so.
> >
> > +Ismaël Mejía <mailto:iemejia@gmail.com
> <ma...@gmail.com> <mailto:iemejia@gmail.com
> <ma...@gmail.com>>> - do
> > you think we should formally conclude the vote on the
> thread [VOTE]
> > [DISCUSSION] Remove support for Java 7?
> > Or should we take more steps - e.g. perhaps tweet a link
> to that thread
> > from the Beam twitter account, ask people to chime in,
> and wait for say
> > 2 weeks before declaring a conclusion?
> >
> > Let's also have a process JIRA for going Java8. I've
> filed one:
> > https://issues.apache.org/jira/browse/BEAM-3285
> <https://issues.apache.org/jira/browse/BEAM-3285>
> > <https://issues.apache.org/jira/browse/BEAM-3285
> <https://issues.apache.org/jira/browse/BEAM-3285>>
> >
> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré
> <jb@nanthrax.net <ma...@nanthrax.net>
> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>
> <mailto:jb@nanthrax.net <ma...@nanthrax.net>
> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>>> wrote:
> >
> > Just an important note that we forgot to mention.
> >
> > !! The 2.2.0 release will be the last one supporting
> Spark 1.x and
> > Java 7 !!
> >
> > Starting from Beam 2.3.0, the Spark runner will work
> only with
> > Spark 2.x and we
> > will focus only Java 8.
> >
> > Regards
> > JB
> >
> > On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
> > > Thanks Reuven !
> > >
> > > I would like to emphasize on some highlights in
> 2.2.0 release:
> > >
> > > - New IOs have been introduced:
> > > * TikaIO leveraging Apache Tika, allowing the
> deal with a lot
> > of different
> > > data formats
> > > * RedisIO to read and write key/value pairs
> from a Redis
> > server. This
> > IO will
> > > be soon extended to Redis PubSub.
> > > * FileIO provides transforms for working with
> files (raw).
> > Especially, it
> > > provides matching file patterns and read on
> patterns. It can be
> > easily
> > extended
> > > for a specific format (like we do in AvroIO or
> TextIO now).
> > > * SolrIO to interact with Apache Solr (Lucene)
> > >
> > > - On the other hand, improvements have been
> performed on
> > existing IOs:
> > > * We started to introduce readAll pattern in
> IOs (AvroIO,
> > TextIO, JdbcIO,
> > > ...), allowing to pass "request" arguments via an
> input PCollection.
> > > * ElasticsearchIO has an improved support of
> different
> > Elasticsearch
> > version
> > > (including Elasticsearch 5.x). It also now
> supports SSL/TLS.
> > > * HBaseIO is now able to do dynamic work
> rebalancing
> > > * KinesisIO uses a more accurate watermark
> (based on
> > approximateArrivalTimestamp)
> > > * TextIO now supports custom delimiter and like
> AvroIO,
> > supports the
> > readAll
> > > pattern,
> > > * Performance improvements on JdbcIO when it
> has to read lot
> > of rows
> > > * Kafka write supports Exactly-Once pattern
> (introduce in
> > Kafka 0.11.x)
> > >
> > > - A new DSL has been introduced: the SQL DSL !
> > >
> > > We are now focus on 2.3.0 release with new
> improvements and
> > features !
> > >
> > > Stay tuned !
> > >
> > > JB on behalf of the Apache Beam community.
> > >
> > > On 12/02/2017 11:40 PM, Reuven Lax wrote:
> > >> The Apache Beam community is pleased to announce the
> > availability of the
> > >> 2.2.0 release.
> > >>
> > >> This release adds support for generic file
> sources and sinks
> > (beyond TextIO
> > >> and AvroIO) using FileIO, including support for
> dynamic
> > filenames using
> > >> readAll; this allows streaming pipelines to now
> read from files by
> > >> continuously monitoring a directory for new
> filw. Many other
> > IOs are
> > improved,
> > >> notably including exactly-once support for the
> Kafka sink. Initial
> > support for
> > >> BEAM-SQL is also included in this release. For a
> more-complete
> > list of major
> > >> changes in the release, please refer to the
> release notes [2].
> > >>
> > >> The 2.2.0 release is now the recommended
> version; we encourage
> > everyone to
> > >> upgrade from any earlier releases.
> > >>
> > >> We’d like to invite everyone to try out Apache
> Beam today and
> > consider
> > >> joining our vibrant community. We welcome feedback,
> > contribution and
> > >> participation through our mailing lists, issue
> tracker, pull
> > requests, and
> > >> events.
> > >>
> > >> - Reuven Lax, on behalf of the Apache Beam
> community.
> > >>
> > >> [1]
> https://beam.apache.org/get-started/downloads/
> <https://beam.apache.org/get-started/downloads/>
> > <https://beam.apache.org/get-started/downloads/
> <https://beam.apache.org/get-started/downloads/>>
> > >> [2]
> > >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044>
> >
> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044 <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044>>
> > >>
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> <mailto:jbonofre@apache.org <ma...@apache.org>>
> > <mailto:jbonofre@apache.org <ma...@apache.org>
> <mailto:jbonofre@apache.org <ma...@apache.org>>>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> <mailto:jbonofre@apache.org <ma...@apache.org>>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Apache Beam, version 2.2.0
Posted by Kenneth Knowles <ke...@apache.org>.
+1 to the poll and also to Reuven's point.
Those without a support contract would have been using JDK 7 without
security updates for years. IMO it seems harmful, as a netizen, to
encourage its use/existence.
If there's no noise from the prior thread, then I would assume no one on
user@ has any objection. Anyone else with customers should reach out to
them.
Kenn
On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <re...@google.com> wrote:
> Technically it's a backwards-incompatible change, however if we are
> convinced the risk is low we could do it.
>
> As mentioned on the original thread, it's not clear that all Beam users
> read user@ - e.g. most Dataflow users definitely do not. I think we need
> to separately reach out to users of each runner through runner-specific
> channels.
>
> Reuven
>
> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov <ki...@google.com>
> wrote:
>
>> On the original thread https://lists.apache.or
>> g/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e3919790
>> 87f725d2e0@%3Cuser.beam.apache.org%3E , Robert and Ismaël were in favor
>> of no major version change [Ismaël said:* Also I am afraid that if we
>> wait*
>> *until we have enough changes to switch Beam to a new major version the
>> switch to Java 8 will happen too late, probably after Java 8's end of life.
>> And I am not exaggerating, Java 8 is planned to EOL next march 2018!*];
>> JB and now Reuven are in favor of a major version change; nobody so far
>> argued against switching to Java8 in general.
>>
>> I'm personally in favor of no major version change (i.e. not waiting
>> until all other large changes for Beam 3.0 converge, which will likely be
>> many months), because:
>> - Reasons Ismaël cited; plus the reason that most people are likely
>> already using Java 8.
>> - Going Java8-only earlier will make other Beam 3.0 APIs better for Java8
>> users, because we (Beam contributors) will have experience working with
>> them within the SDK in Java8 (e.g. writing tests with use of lambdas and
>> noticing whether it's clunky, or whether some other Beam APIs need better
>> Java8 support).
>> - Going Java8 will make it more reasonable to include (mostly or only)
>> Java8 snippets in Beam documentation, which will obviously look more
>> concise and attractive, addressing one of the common concerns of Beam users
>> that it has a heavyweight API compared to functional-style APIs of Spark
>> etc.
>>
>> I think resolving this via a poll of users would be reasonable. I'd
>> suggest e.g. the following phrasing:
>>
>> Apache Beam is considering dropping support for Java 7, and supporting
>> only Java 8 and above in a subsequent release. How would it impact your
>> usage of Beam?
>> - I am already using only Java 8+ for building my Beam code
>> - I am using Java 7 for building my Beam code, but I would have no
>> trouble switching to Java 8
>> - I am using Java 7 for building my Beam code, and dropping Java 7 would
>> be a blocker or hindrance to adopting the new release for me
>>
>> We could tweet this poll on Apache Beam twitter and publish on user@,
>> and, say, if we receive 5% or fewer votes for option 3 after keeping it
>> open for 2 weeks, then adopt Java 8 without a major version change.
>>
>> WDYT?
>>
>> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Good idea ! Definitely +1
>>>
>>> Regards
>>> JB
>>>
>>> On 12/05/2017 05:25 AM, Reuven Lax wrote:
>>> > We should bring this up on the Beam 3.0 thread. Since it's technically
>>> a
>>> > backwards-incompatible change, it might make a good item for Beam 3.0.
>>> >
>>> > Reuven
>>> >
>>> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré <jb@nanthrax.net
>>> > <ma...@nanthrax.net>> wrote:
>>> >
>>> > My apologizes, I thought we had a consensus already.
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
>>> >
>>> > Thanks JB for sending the detailed notes about new stuff in
>>> 2.2.0! A lot
>>> > of exciting things indeed.
>>> >
>>> > Regarding Java 8: I thought our consensus was to have the
>>> release notes
>>> > say that we're *considering* going Java8-only, and use that to
>>> get more
>>> > opinions from the user community - but I can't find the emails
>>> that made
>>> > me think so.
>>> >
>>> > +Ismaël Mejía <mailto:iemejia@gmail.com <mailto:
>>> iemejia@gmail.com>> - do
>>> > you think we should formally conclude the vote on the
>>> thread [VOTE]
>>> > [DISCUSSION] Remove support for Java 7?
>>> > Or should we take more steps - e.g. perhaps tweet a link to
>>> that thread
>>> > from the Beam twitter account, ask people to chime in, and
>>> wait for say
>>> > 2 weeks before declaring a conclusion?
>>> >
>>> > Let's also have a process JIRA for going Java8. I've filed one:
>>> > https://issues.apache.org/jira/browse/BEAM-3285
>>> > <https://issues.apache.org/jira/browse/BEAM-3285>
>>> >
>>> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré <
>>> jb@nanthrax.net
>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>> > <ma...@nanthrax.net>>> wrote:
>>> >
>>> > Just an important note that we forgot to mention.
>>> >
>>> > !! The 2.2.0 release will be the last one supporting
>>> Spark 1.x and
>>> > Java 7 !!
>>> >
>>> > Starting from Beam 2.3.0, the Spark runner will work only
>>> with
>>> > Spark 2.x and we
>>> > will focus only Java 8.
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
>>> > > Thanks Reuven !
>>> > >
>>> > > I would like to emphasize on some highlights in 2.2.0
>>> release:
>>> > >
>>> > > - New IOs have been introduced:
>>> > > * TikaIO leveraging Apache Tika, allowing the deal
>>> with a lot
>>> > of different
>>> > > data formats
>>> > > * RedisIO to read and write key/value pairs from a
>>> Redis
>>> > server. This
>>> > IO will
>>> > > be soon extended to Redis PubSub.
>>> > > * FileIO provides transforms for working with files
>>> (raw).
>>> > Especially, it
>>> > > provides matching file patterns and read on patterns.
>>> It can be
>>> > easily
>>> > extended
>>> > > for a specific format (like we do in AvroIO or TextIO
>>> now).
>>> > > * SolrIO to interact with Apache Solr (Lucene)
>>> > >
>>> > > - On the other hand, improvements have been performed
>>> on
>>> > existing IOs:
>>> > > * We started to introduce readAll pattern in IOs
>>> (AvroIO,
>>> > TextIO, JdbcIO,
>>> > > ...), allowing to pass "request" arguments via an
>>> input PCollection.
>>> > > * ElasticsearchIO has an improved support of
>>> different
>>> > Elasticsearch
>>> > version
>>> > > (including Elasticsearch 5.x). It also now supports
>>> SSL/TLS.
>>> > > * HBaseIO is now able to do dynamic work rebalancing
>>> > > * KinesisIO uses a more accurate watermark (based on
>>> > approximateArrivalTimestamp)
>>> > > * TextIO now supports custom delimiter and like
>>> AvroIO,
>>> > supports the
>>> > readAll
>>> > > pattern,
>>> > > * Performance improvements on JdbcIO when it has to
>>> read lot
>>> > of rows
>>> > > * Kafka write supports Exactly-Once pattern
>>> (introduce in
>>> > Kafka 0.11.x)
>>> > >
>>> > > - A new DSL has been introduced: the SQL DSL !
>>> > >
>>> > > We are now focus on 2.3.0 release with new
>>> improvements and
>>> > features !
>>> > >
>>> > > Stay tuned !
>>> > >
>>> > > JB on behalf of the Apache Beam community.
>>> > >
>>> > > On 12/02/2017 11:40 PM, Reuven Lax wrote:
>>> > >> The Apache Beam community is pleased to announce the
>>> > availability of the
>>> > >> 2.2.0 release.
>>> > >>
>>> > >> This release adds support for generic file sources
>>> and sinks
>>> > (beyond TextIO
>>> > >> and AvroIO) using FileIO, including support for
>>> dynamic
>>> > filenames using
>>> > >> readAll; this allows streaming pipelines to now read
>>> from files by
>>> > >> continuously monitoring a directory for new filw.
>>> Many other
>>> > IOs are
>>> > improved,
>>> > >> notably including exactly-once support for the Kafka
>>> sink. Initial
>>> > support for
>>> > >> BEAM-SQL is also included in this release. For a
>>> more-complete
>>> > list of major
>>> > >> changes in the release, please refer to the release
>>> notes [2].
>>> > >>
>>> > >> The 2.2.0 release is now the recommended version; we
>>> encourage
>>> > everyone to
>>> > >> upgrade from any earlier releases.
>>> > >>
>>> > >> We’d like to invite everyone to try out Apache Beam
>>> today and
>>> > consider
>>> > >> joining our vibrant community. We welcome feedback,
>>> > contribution and
>>> > >> participation through our mailing lists, issue
>>> tracker, pull
>>> > requests, and
>>> > >> events.
>>> > >>
>>> > >> - Reuven Lax, on behalf of the Apache Beam community.
>>> > >>
>>> > >> [1] https://beam.apache.org/get-started/downloads/
>>> > <https://beam.apache.org/get-started/downloads/>
>>> > >> [2]
>>> > >>
>>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?proj
>>> ectId=12319527&version=12341044
>>> > <https://issues.apache.org/jira/secure/ReleaseNote.jspa?pro
>>> jectId=12319527&version=12341044>
>>> > >>
>>> > >
>>> >
>>> > --
>>> > Jean-Baptiste Onofré
>>> > jbonofre@apache.org <ma...@apache.org>
>>> > <mailto:jbonofre@apache.org <ma...@apache.org>>
>>> > http://blog.nanthrax.net
>>> > Talend - http://www.talend.com
>>> >
>>> >
>>> > --
>>> > Jean-Baptiste Onofré
>>> > jbonofre@apache.org <ma...@apache.org>
>>> > http://blog.nanthrax.net
>>> > Talend - http://www.talend.com
>>> >
>>> >
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
>
Re: Apache Beam, version 2.2.0
Posted by Reuven Lax <re...@google.com>.
Technically it's a backwards-incompatible change, however if we are
convinced the risk is low we could do it.
As mentioned on the original thread, it's not clear that all Beam users
read user@ - e.g. most Dataflow users definitely do not. I think we need to
separately reach out to users of each runner through runner-specific
channels.
Reuven
On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov <ki...@google.com>
wrote:
> On the original thread https://lists.apache.org/thread.html/
> 2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%
> 3Cuser.beam.apache.org%3E , Robert and Ismaël were in favor of no major
> version change [Ismaël said:* Also I am afraid that if we wait*
> *until we have enough changes to switch Beam to a new major version the
> switch to Java 8 will happen too late, probably after Java 8's end of life.
> And I am not exaggerating, Java 8 is planned to EOL next march 2018!*];
> JB and now Reuven are in favor of a major version change; nobody so far
> argued against switching to Java8 in general.
>
> I'm personally in favor of no major version change (i.e. not waiting until
> all other large changes for Beam 3.0 converge, which will likely be many
> months), because:
> - Reasons Ismaël cited; plus the reason that most people are likely
> already using Java 8.
> - Going Java8-only earlier will make other Beam 3.0 APIs better for Java8
> users, because we (Beam contributors) will have experience working with
> them within the SDK in Java8 (e.g. writing tests with use of lambdas and
> noticing whether it's clunky, or whether some other Beam APIs need better
> Java8 support).
> - Going Java8 will make it more reasonable to include (mostly or only)
> Java8 snippets in Beam documentation, which will obviously look more
> concise and attractive, addressing one of the common concerns of Beam users
> that it has a heavyweight API compared to functional-style APIs of Spark
> etc.
>
> I think resolving this via a poll of users would be reasonable. I'd
> suggest e.g. the following phrasing:
>
> Apache Beam is considering dropping support for Java 7, and supporting
> only Java 8 and above in a subsequent release. How would it impact your
> usage of Beam?
> - I am already using only Java 8+ for building my Beam code
> - I am using Java 7 for building my Beam code, but I would have no trouble
> switching to Java 8
> - I am using Java 7 for building my Beam code, and dropping Java 7 would
> be a blocker or hindrance to adopting the new release for me
>
> We could tweet this poll on Apache Beam twitter and publish on user@,
> and, say, if we receive 5% or fewer votes for option 3 after keeping it
> open for 2 weeks, then adopt Java 8 without a major version change.
>
> WDYT?
>
> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Good idea ! Definitely +1
>>
>> Regards
>> JB
>>
>> On 12/05/2017 05:25 AM, Reuven Lax wrote:
>> > We should bring this up on the Beam 3.0 thread. Since it's technically a
>> > backwards-incompatible change, it might make a good item for Beam 3.0.
>> >
>> > Reuven
>> >
>> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré <jb@nanthrax.net
>> > <ma...@nanthrax.net>> wrote:
>> >
>> > My apologizes, I thought we had a consensus already.
>> >
>> > Regards
>> > JB
>> >
>> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
>> >
>> > Thanks JB for sending the detailed notes about new stuff in
>> 2.2.0! A lot
>> > of exciting things indeed.
>> >
>> > Regarding Java 8: I thought our consensus was to have the
>> release notes
>> > say that we're *considering* going Java8-only, and use that to
>> get more
>> > opinions from the user community - but I can't find the emails
>> that made
>> > me think so.
>> >
>> > +Ismaël Mejía <mailto:iemejia@gmail.com <mailto:
>> iemejia@gmail.com>> - do
>> > you think we should formally conclude the vote on the
>> thread [VOTE]
>> > [DISCUSSION] Remove support for Java 7?
>> > Or should we take more steps - e.g. perhaps tweet a link to
>> that thread
>> > from the Beam twitter account, ask people to chime in, and wait
>> for say
>> > 2 weeks before declaring a conclusion?
>> >
>> > Let's also have a process JIRA for going Java8. I've filed one:
>> > https://issues.apache.org/jira/browse/BEAM-3285
>> > <https://issues.apache.org/jira/browse/BEAM-3285>
>> >
>> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré <
>> jb@nanthrax.net
>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>> > <ma...@nanthrax.net>>> wrote:
>> >
>> > Just an important note that we forgot to mention.
>> >
>> > !! The 2.2.0 release will be the last one supporting Spark
>> 1.x and
>> > Java 7 !!
>> >
>> > Starting from Beam 2.3.0, the Spark runner will work only
>> with
>> > Spark 2.x and we
>> > will focus only Java 8.
>> >
>> > Regards
>> > JB
>> >
>> > On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
>> > > Thanks Reuven !
>> > >
>> > > I would like to emphasize on some highlights in 2.2.0
>> release:
>> > >
>> > > - New IOs have been introduced:
>> > > * TikaIO leveraging Apache Tika, allowing the deal
>> with a lot
>> > of different
>> > > data formats
>> > > * RedisIO to read and write key/value pairs from a
>> Redis
>> > server. This
>> > IO will
>> > > be soon extended to Redis PubSub.
>> > > * FileIO provides transforms for working with files
>> (raw).
>> > Especially, it
>> > > provides matching file patterns and read on patterns.
>> It can be
>> > easily
>> > extended
>> > > for a specific format (like we do in AvroIO or TextIO
>> now).
>> > > * SolrIO to interact with Apache Solr (Lucene)
>> > >
>> > > - On the other hand, improvements have been performed on
>> > existing IOs:
>> > > * We started to introduce readAll pattern in IOs
>> (AvroIO,
>> > TextIO, JdbcIO,
>> > > ...), allowing to pass "request" arguments via an input
>> PCollection.
>> > > * ElasticsearchIO has an improved support of different
>> > Elasticsearch
>> > version
>> > > (including Elasticsearch 5.x). It also now supports
>> SSL/TLS.
>> > > * HBaseIO is now able to do dynamic work rebalancing
>> > > * KinesisIO uses a more accurate watermark (based on
>> > approximateArrivalTimestamp)
>> > > * TextIO now supports custom delimiter and like
>> AvroIO,
>> > supports the
>> > readAll
>> > > pattern,
>> > > * Performance improvements on JdbcIO when it has to
>> read lot
>> > of rows
>> > > * Kafka write supports Exactly-Once pattern
>> (introduce in
>> > Kafka 0.11.x)
>> > >
>> > > - A new DSL has been introduced: the SQL DSL !
>> > >
>> > > We are now focus on 2.3.0 release with new improvements
>> and
>> > features !
>> > >
>> > > Stay tuned !
>> > >
>> > > JB on behalf of the Apache Beam community.
>> > >
>> > > On 12/02/2017 11:40 PM, Reuven Lax wrote:
>> > >> The Apache Beam community is pleased to announce the
>> > availability of the
>> > >> 2.2.0 release.
>> > >>
>> > >> This release adds support for generic file sources and
>> sinks
>> > (beyond TextIO
>> > >> and AvroIO) using FileIO, including support for dynamic
>> > filenames using
>> > >> readAll; this allows streaming pipelines to now read
>> from files by
>> > >> continuously monitoring a directory for new filw. Many
>> other
>> > IOs are
>> > improved,
>> > >> notably including exactly-once support for the Kafka
>> sink. Initial
>> > support for
>> > >> BEAM-SQL is also included in this release. For a
>> more-complete
>> > list of major
>> > >> changes in the release, please refer to the release
>> notes [2].
>> > >>
>> > >> The 2.2.0 release is now the recommended version; we
>> encourage
>> > everyone to
>> > >> upgrade from any earlier releases.
>> > >>
>> > >> We’d like to invite everyone to try out Apache Beam
>> today and
>> > consider
>> > >> joining our vibrant community. We welcome feedback,
>> > contribution and
>> > >> participation through our mailing lists, issue
>> tracker, pull
>> > requests, and
>> > >> events.
>> > >>
>> > >> - Reuven Lax, on behalf of the Apache Beam community.
>> > >>
>> > >> [1] https://beam.apache.org/get-started/downloads/
>> > <https://beam.apache.org/get-started/downloads/>
>> > >> [2]
>> > >>
>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12319527&version=12341044
>> > <https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12319527&version=12341044>
>> > >>
>> > >
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org <ma...@apache.org>
>> > <mailto:jbonofre@apache.org <ma...@apache.org>>
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>> >
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org <ma...@apache.org>
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>> >
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
Re: Apache Beam, version 2.2.0
Posted by Eugene Kirpichov <ki...@google.com>.
On the original thread
https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
,
Robert and Ismaël were in favor of no major version change [Ismaël said:*
Also I am afraid that if we wait*
*until we have enough changes to switch Beam to a new major version the
switch to Java 8 will happen too late, probably after Java 8's end of life.
And I am not exaggerating, Java 8 is planned to EOL next march 2018!*]; JB
and now Reuven are in favor of a major version change; nobody so far argued
against switching to Java8 in general.
I'm personally in favor of no major version change (i.e. not waiting until
all other large changes for Beam 3.0 converge, which will likely be many
months), because:
- Reasons Ismaël cited; plus the reason that most people are likely already
using Java 8.
- Going Java8-only earlier will make other Beam 3.0 APIs better for Java8
users, because we (Beam contributors) will have experience working with
them within the SDK in Java8 (e.g. writing tests with use of lambdas and
noticing whether it's clunky, or whether some other Beam APIs need better
Java8 support).
- Going Java8 will make it more reasonable to include (mostly or only)
Java8 snippets in Beam documentation, which will obviously look more
concise and attractive, addressing one of the common concerns of Beam users
that it has a heavyweight API compared to functional-style APIs of Spark
etc.
I think resolving this via a poll of users would be reasonable. I'd suggest
e.g. the following phrasing:
Apache Beam is considering dropping support for Java 7, and supporting only
Java 8 and above in a subsequent release. How would it impact your usage of
Beam?
- I am already using only Java 8+ for building my Beam code
- I am using Java 7 for building my Beam code, but I would have no trouble
switching to Java 8
- I am using Java 7 for building my Beam code, and dropping Java 7 would be
a blocker or hindrance to adopting the new release for me
We could tweet this poll on Apache Beam twitter and publish on user@, and,
say, if we receive 5% or fewer votes for option 3 after keeping it open for
2 weeks, then adopt Java 8 without a major version change.
WDYT?
On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Good idea ! Definitely +1
>
> Regards
> JB
>
> On 12/05/2017 05:25 AM, Reuven Lax wrote:
> > We should bring this up on the Beam 3.0 thread. Since it's technically a
> > backwards-incompatible change, it might make a good item for Beam 3.0.
> >
> > Reuven
> >
> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré <jb@nanthrax.net
> > <ma...@nanthrax.net>> wrote:
> >
> > My apologizes, I thought we had a consensus already.
> >
> > Regards
> > JB
> >
> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
> >
> > Thanks JB for sending the detailed notes about new stuff in
> 2.2.0! A lot
> > of exciting things indeed.
> >
> > Regarding Java 8: I thought our consensus was to have the
> release notes
> > say that we're *considering* going Java8-only, and use that to
> get more
> > opinions from the user community - but I can't find the emails
> that made
> > me think so.
> >
> > +Ismaël Mejía <mailto:iemejia@gmail.com <mailto:
> iemejia@gmail.com>> - do
> > you think we should formally conclude the vote on the
> thread [VOTE]
> > [DISCUSSION] Remove support for Java 7?
> > Or should we take more steps - e.g. perhaps tweet a link to that
> thread
> > from the Beam twitter account, ask people to chime in, and wait
> for say
> > 2 weeks before declaring a conclusion?
> >
> > Let's also have a process JIRA for going Java8. I've filed one:
> > https://issues.apache.org/jira/browse/BEAM-3285
> > <https://issues.apache.org/jira/browse/BEAM-3285>
> >
> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré <
> jb@nanthrax.net
> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
> > <ma...@nanthrax.net>>> wrote:
> >
> > Just an important note that we forgot to mention.
> >
> > !! The 2.2.0 release will be the last one supporting Spark
> 1.x and
> > Java 7 !!
> >
> > Starting from Beam 2.3.0, the Spark runner will work only
> with
> > Spark 2.x and we
> > will focus only Java 8.
> >
> > Regards
> > JB
> >
> > On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
> > > Thanks Reuven !
> > >
> > > I would like to emphasize on some highlights in 2.2.0
> release:
> > >
> > > - New IOs have been introduced:
> > > * TikaIO leveraging Apache Tika, allowing the deal
> with a lot
> > of different
> > > data formats
> > > * RedisIO to read and write key/value pairs from a
> Redis
> > server. This
> > IO will
> > > be soon extended to Redis PubSub.
> > > * FileIO provides transforms for working with files
> (raw).
> > Especially, it
> > > provides matching file patterns and read on patterns. It
> can be
> > easily
> > extended
> > > for a specific format (like we do in AvroIO or TextIO
> now).
> > > * SolrIO to interact with Apache Solr (Lucene)
> > >
> > > - On the other hand, improvements have been performed on
> > existing IOs:
> > > * We started to introduce readAll pattern in IOs
> (AvroIO,
> > TextIO, JdbcIO,
> > > ...), allowing to pass "request" arguments via an input
> PCollection.
> > > * ElasticsearchIO has an improved support of different
> > Elasticsearch
> > version
> > > (including Elasticsearch 5.x). It also now supports
> SSL/TLS.
> > > * HBaseIO is now able to do dynamic work rebalancing
> > > * KinesisIO uses a more accurate watermark (based on
> > approximateArrivalTimestamp)
> > > * TextIO now supports custom delimiter and like AvroIO,
> > supports the
> > readAll
> > > pattern,
> > > * Performance improvements on JdbcIO when it has to
> read lot
> > of rows
> > > * Kafka write supports Exactly-Once pattern (introduce
> in
> > Kafka 0.11.x)
> > >
> > > - A new DSL has been introduced: the SQL DSL !
> > >
> > > We are now focus on 2.3.0 release with new improvements
> and
> > features !
> > >
> > > Stay tuned !
> > >
> > > JB on behalf of the Apache Beam community.
> > >
> > > On 12/02/2017 11:40 PM, Reuven Lax wrote:
> > >> The Apache Beam community is pleased to announce the
> > availability of the
> > >> 2.2.0 release.
> > >>
> > >> This release adds support for generic file sources and
> sinks
> > (beyond TextIO
> > >> and AvroIO) using FileIO, including support for dynamic
> > filenames using
> > >> readAll; this allows streaming pipelines to now read
> from files by
> > >> continuously monitoring a directory for new filw. Many
> other
> > IOs are
> > improved,
> > >> notably including exactly-once support for the Kafka
> sink. Initial
> > support for
> > >> BEAM-SQL is also included in this release. For a
> more-complete
> > list of major
> > >> changes in the release, please refer to the release
> notes [2].
> > >>
> > >> The 2.2.0 release is now the recommended version; we
> encourage
> > everyone to
> > >> upgrade from any earlier releases.
> > >>
> > >> We’d like to invite everyone to try out Apache Beam
> today and
> > consider
> > >> joining our vibrant community. We welcome feedback,
> > contribution and
> > >> participation through our mailing lists, issue tracker,
> pull
> > requests, and
> > >> events.
> > >>
> > >> - Reuven Lax, on behalf of the Apache Beam community.
> > >>
> > >> [1] https://beam.apache.org/get-started/downloads/
> > <https://beam.apache.org/get-started/downloads/>
> > >> [2]
> > >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
> > <
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
> >
> > >>
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > <mailto:jbonofre@apache.org <ma...@apache.org>>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Apache Beam, version 2.2.0
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Good idea ! Definitely +1
Regards
JB
On 12/05/2017 05:25 AM, Reuven Lax wrote:
> We should bring this up on the Beam 3.0 thread. Since it's technically a
> backwards-incompatible change, it might make a good item for Beam 3.0.
>
> Reuven
>
> On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> My apologizes, I thought we had a consensus already.
>
> Regards
> JB
>
> On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
>
> Thanks JB for sending the detailed notes about new stuff in 2.2.0! A lot
> of exciting things indeed.
>
> Regarding Java 8: I thought our consensus was to have the release notes
> say that we're *considering* going Java8-only, and use that to get more
> opinions from the user community - but I can't find the emails that made
> me think so.
>
> +Ismaël Mejía <mailto:iemejia@gmail.com <ma...@gmail.com>> - do
> you think we should formally conclude the vote on the thread [VOTE]
> [DISCUSSION] Remove support for Java 7?
> Or should we take more steps - e.g. perhaps tweet a link to that thread
> from the Beam twitter account, ask people to chime in, and wait for say
> 2 weeks before declaring a conclusion?
>
> Let's also have a process JIRA for going Java8. I've filed one:
> https://issues.apache.org/jira/browse/BEAM-3285
> <https://issues.apache.org/jira/browse/BEAM-3285>
>
> On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net> <mailto:jb@nanthrax.net
> <ma...@nanthrax.net>>> wrote:
>
> Just an important note that we forgot to mention.
>
> !! The 2.2.0 release will be the last one supporting Spark 1.x and
> Java 7 !!
>
> Starting from Beam 2.3.0, the Spark runner will work only with
> Spark 2.x and we
> will focus only Java 8.
>
> Regards
> JB
>
> On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
> > Thanks Reuven !
> >
> > I would like to emphasize on some highlights in 2.2.0 release:
> >
> > - New IOs have been introduced:
> > * TikaIO leveraging Apache Tika, allowing the deal with a lot
> of different
> > data formats
> > * RedisIO to read and write key/value pairs from a Redis
> server. This
> IO will
> > be soon extended to Redis PubSub.
> > * FileIO provides transforms for working with files (raw).
> Especially, it
> > provides matching file patterns and read on patterns. It can be
> easily
> extended
> > for a specific format (like we do in AvroIO or TextIO now).
> > * SolrIO to interact with Apache Solr (Lucene)
> >
> > - On the other hand, improvements have been performed on
> existing IOs:
> > * We started to introduce readAll pattern in IOs (AvroIO,
> TextIO, JdbcIO,
> > ...), allowing to pass "request" arguments via an input PCollection.
> > * ElasticsearchIO has an improved support of different
> Elasticsearch
> version
> > (including Elasticsearch 5.x). It also now supports SSL/TLS.
> > * HBaseIO is now able to do dynamic work rebalancing
> > * KinesisIO uses a more accurate watermark (based on
> approximateArrivalTimestamp)
> > * TextIO now supports custom delimiter and like AvroIO,
> supports the
> readAll
> > pattern,
> > * Performance improvements on JdbcIO when it has to read lot
> of rows
> > * Kafka write supports Exactly-Once pattern (introduce in
> Kafka 0.11.x)
> >
> > - A new DSL has been introduced: the SQL DSL !
> >
> > We are now focus on 2.3.0 release with new improvements and
> features !
> >
> > Stay tuned !
> >
> > JB on behalf of the Apache Beam community.
> >
> > On 12/02/2017 11:40 PM, Reuven Lax wrote:
> >> The Apache Beam community is pleased to announce the
> availability of the
> >> 2.2.0 release.
> >>
> >> This release adds support for generic file sources and sinks
> (beyond TextIO
> >> and AvroIO) using FileIO, including support for dynamic
> filenames using
> >> readAll; this allows streaming pipelines to now read from files by
> >> continuously monitoring a directory for new filw. Many other
> IOs are
> improved,
> >> notably including exactly-once support for the Kafka sink. Initial
> support for
> >> BEAM-SQL is also included in this release. For a more-complete
> list of major
> >> changes in the release, please refer to the release notes [2].
> >>
> >> The 2.2.0 release is now the recommended version; we encourage
> everyone to
> >> upgrade from any earlier releases.
> >>
> >> We’d like to invite everyone to try out Apache Beam today and
> consider
> >> joining our vibrant community. We welcome feedback,
> contribution and
> >> participation through our mailing lists, issue tracker, pull
> requests, and
> >> events.
> >>
> >> - Reuven Lax, on behalf of the Apache Beam community.
> >>
> >> [1] https://beam.apache.org/get-started/downloads/
> <https://beam.apache.org/get-started/downloads/>
> >> [2]
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044>
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> <mailto:jbonofre@apache.org <ma...@apache.org>>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Apache Beam, version 2.2.0
Posted by Reuven Lax <re...@google.com>.
We should bring this up on the Beam 3.0 thread. Since it's technically a
backwards-incompatible change, it might make a good item for Beam 3.0.
Reuven
On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> My apologizes, I thought we had a consensus already.
>
> Regards
> JB
>
> On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
>
>> Thanks JB for sending the detailed notes about new stuff in 2.2.0! A lot
>> of exciting things indeed.
>>
>> Regarding Java 8: I thought our consensus was to have the release notes
>> say that we're *considering* going Java8-only, and use that to get more
>> opinions from the user community - but I can't find the emails that made me
>> think so.
>>
>> +Ismaël Mejía <ma...@gmail.com> - do you think we should
>> formally conclude the vote on the thread [VOTE] [DISCUSSION] Remove support
>> for Java 7?
>> Or should we take more steps - e.g. perhaps tweet a link to that thread
>> from the Beam twitter account, ask people to chime in, and wait for say 2
>> weeks before declaring a conclusion?
>>
>> Let's also have a process JIRA for going Java8. I've filed one:
>> https://issues.apache.org/jira/browse/BEAM-3285
>>
>> On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré <jb@nanthrax.net
>> <ma...@nanthrax.net>> wrote:
>>
>> Just an important note that we forgot to mention.
>>
>> !! The 2.2.0 release will be the last one supporting Spark 1.x and
>> Java 7 !!
>>
>> Starting from Beam 2.3.0, the Spark runner will work only with Spark
>> 2.x and we
>> will focus only Java 8.
>>
>> Regards
>> JB
>>
>> On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
>> > Thanks Reuven !
>> >
>> > I would like to emphasize on some highlights in 2.2.0 release:
>> >
>> > - New IOs have been introduced:
>> > * TikaIO leveraging Apache Tika, allowing the deal with a lot of
>> different
>> > data formats
>> > * RedisIO to read and write key/value pairs from a Redis server.
>> This
>> IO will
>> > be soon extended to Redis PubSub.
>> > * FileIO provides transforms for working with files (raw).
>> Especially, it
>> > provides matching file patterns and read on patterns. It can be
>> easily
>> extended
>> > for a specific format (like we do in AvroIO or TextIO now).
>> > * SolrIO to interact with Apache Solr (Lucene)
>> >
>> > - On the other hand, improvements have been performed on existing
>> IOs:
>> > * We started to introduce readAll pattern in IOs (AvroIO,
>> TextIO, JdbcIO,
>> > ...), allowing to pass "request" arguments via an input
>> PCollection.
>> > * ElasticsearchIO has an improved support of different
>> Elasticsearch
>> version
>> > (including Elasticsearch 5.x). It also now supports SSL/TLS.
>> > * HBaseIO is now able to do dynamic work rebalancing
>> > * KinesisIO uses a more accurate watermark (based on
>> approximateArrivalTimestamp)
>> > * TextIO now supports custom delimiter and like AvroIO, supports
>> the
>> readAll
>> > pattern,
>> > * Performance improvements on JdbcIO when it has to read lot of
>> rows
>> > * Kafka write supports Exactly-Once pattern (introduce in Kafka
>> 0.11.x)
>> >
>> > - A new DSL has been introduced: the SQL DSL !
>> >
>> > We are now focus on 2.3.0 release with new improvements and
>> features !
>> >
>> > Stay tuned !
>> >
>> > JB on behalf of the Apache Beam community.
>> >
>> > On 12/02/2017 11:40 PM, Reuven Lax wrote:
>> >> The Apache Beam community is pleased to announce the availability
>> of the
>> >> 2.2.0 release.
>> >>
>> >> This release adds support for generic file sources and sinks
>> (beyond TextIO
>> >> and AvroIO) using FileIO, including support for dynamic filenames
>> using
>> >> readAll; this allows streaming pipelines to now read from files by
>> >> continuously monitoring a directory for new filw. Many other IOs
>> are
>> improved,
>> >> notably including exactly-once support for the Kafka sink. Initial
>> support for
>> >> BEAM-SQL is also included in this release. For a more-complete
>> list of major
>> >> changes in the release, please refer to the release notes [2].
>> >>
>> >> The 2.2.0 release is now the recommended version; we encourage
>> everyone to
>> >> upgrade from any earlier releases.
>> >>
>> >> We’d like to invite everyone to try out Apache Beam today and
>> consider
>> >> joining our vibrant community. We welcome feedback, contribution
>> and
>> >> participation through our mailing lists, issue tracker, pull
>> requests, and
>> >> events.
>> >>
>> >> - Reuven Lax, on behalf of the Apache Beam community.
>> >>
>> >> [1] https://beam.apache.org/get-started/downloads/
>> >> [2]
>> >>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>> ctId=12319527&version=12341044
>> >>
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org <ma...@apache.org>
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Apache Beam, version 2.2.0
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
My apologizes, I thought we had a consensus already.
Regards
JB
On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
> Thanks JB for sending the detailed notes about new stuff in 2.2.0! A lot of
> exciting things indeed.
>
> Regarding Java 8: I thought our consensus was to have the release notes say that
> we're *considering* going Java8-only, and use that to get more opinions from the
> user community - but I can't find the emails that made me think so.
>
> +Ismaël Mejía <ma...@gmail.com> - do you think we should formally
> conclude the vote on the thread [VOTE] [DISCUSSION] Remove support for Java 7?
> Or should we take more steps - e.g. perhaps tweet a link to that thread from the
> Beam twitter account, ask people to chime in, and wait for say 2 weeks before
> declaring a conclusion?
>
> Let's also have a process JIRA for going Java8. I've filed one:
> https://issues.apache.org/jira/browse/BEAM-3285
>
> On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> Just an important note that we forgot to mention.
>
> !! The 2.2.0 release will be the last one supporting Spark 1.x and Java 7 !!
>
> Starting from Beam 2.3.0, the Spark runner will work only with Spark 2.x and we
> will focus only Java 8.
>
> Regards
> JB
>
> On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
> > Thanks Reuven !
> >
> > I would like to emphasize on some highlights in 2.2.0 release:
> >
> > - New IOs have been introduced:
> > * TikaIO leveraging Apache Tika, allowing the deal with a lot of different
> > data formats
> > * RedisIO to read and write key/value pairs from a Redis server. This
> IO will
> > be soon extended to Redis PubSub.
> > * FileIO provides transforms for working with files (raw). Especially, it
> > provides matching file patterns and read on patterns. It can be easily
> extended
> > for a specific format (like we do in AvroIO or TextIO now).
> > * SolrIO to interact with Apache Solr (Lucene)
> >
> > - On the other hand, improvements have been performed on existing IOs:
> > * We started to introduce readAll pattern in IOs (AvroIO, TextIO, JdbcIO,
> > ...), allowing to pass "request" arguments via an input PCollection.
> > * ElasticsearchIO has an improved support of different Elasticsearch
> version
> > (including Elasticsearch 5.x). It also now supports SSL/TLS.
> > * HBaseIO is now able to do dynamic work rebalancing
> > * KinesisIO uses a more accurate watermark (based on
> approximateArrivalTimestamp)
> > * TextIO now supports custom delimiter and like AvroIO, supports the
> readAll
> > pattern,
> > * Performance improvements on JdbcIO when it has to read lot of rows
> > * Kafka write supports Exactly-Once pattern (introduce in Kafka 0.11.x)
> >
> > - A new DSL has been introduced: the SQL DSL !
> >
> > We are now focus on 2.3.0 release with new improvements and features !
> >
> > Stay tuned !
> >
> > JB on behalf of the Apache Beam community.
> >
> > On 12/02/2017 11:40 PM, Reuven Lax wrote:
> >> The Apache Beam community is pleased to announce the availability of the
> >> 2.2.0 release.
> >>
> >> This release adds support for generic file sources and sinks (beyond TextIO
> >> and AvroIO) using FileIO, including support for dynamic filenames using
> >> readAll; this allows streaming pipelines to now read from files by
> >> continuously monitoring a directory for new filw. Many other IOs are
> improved,
> >> notably including exactly-once support for the Kafka sink. Initial
> support for
> >> BEAM-SQL is also included in this release. For a more-complete list of major
> >> changes in the release, please refer to the release notes [2].
> >>
> >> The 2.2.0 release is now the recommended version; we encourage everyone to
> >> upgrade from any earlier releases.
> >>
> >> We’d like to invite everyone to try out Apache Beam today and consider
> >> joining our vibrant community. We welcome feedback, contribution and
> >> participation through our mailing lists, issue tracker, pull requests, and
> >> events.
> >>
> >> - Reuven Lax, on behalf of the Apache Beam community.
> >>
> >> [1] https://beam.apache.org/get-started/downloads/
> >> [2]
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Apache Beam, version 2.2.0
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
My apologizes, I thought we had a consensus already.
Regards
JB
On 12/04/2017 11:22 PM, Eugene Kirpichov wrote:
> Thanks JB for sending the detailed notes about new stuff in 2.2.0! A lot of
> exciting things indeed.
>
> Regarding Java 8: I thought our consensus was to have the release notes say that
> we're *considering* going Java8-only, and use that to get more opinions from the
> user community - but I can't find the emails that made me think so.
>
> +Ismaël Mejía <ma...@gmail.com> - do you think we should formally
> conclude the vote on the thread [VOTE] [DISCUSSION] Remove support for Java 7?
> Or should we take more steps - e.g. perhaps tweet a link to that thread from the
> Beam twitter account, ask people to chime in, and wait for say 2 weeks before
> declaring a conclusion?
>
> Let's also have a process JIRA for going Java8. I've filed one:
> https://issues.apache.org/jira/browse/BEAM-3285
>
> On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> Just an important note that we forgot to mention.
>
> !! The 2.2.0 release will be the last one supporting Spark 1.x and Java 7 !!
>
> Starting from Beam 2.3.0, the Spark runner will work only with Spark 2.x and we
> will focus only Java 8.
>
> Regards
> JB
>
> On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
> > Thanks Reuven !
> >
> > I would like to emphasize on some highlights in 2.2.0 release:
> >
> > - New IOs have been introduced:
> > * TikaIO leveraging Apache Tika, allowing the deal with a lot of different
> > data formats
> > * RedisIO to read and write key/value pairs from a Redis server. This
> IO will
> > be soon extended to Redis PubSub.
> > * FileIO provides transforms for working with files (raw). Especially, it
> > provides matching file patterns and read on patterns. It can be easily
> extended
> > for a specific format (like we do in AvroIO or TextIO now).
> > * SolrIO to interact with Apache Solr (Lucene)
> >
> > - On the other hand, improvements have been performed on existing IOs:
> > * We started to introduce readAll pattern in IOs (AvroIO, TextIO, JdbcIO,
> > ...), allowing to pass "request" arguments via an input PCollection.
> > * ElasticsearchIO has an improved support of different Elasticsearch
> version
> > (including Elasticsearch 5.x). It also now supports SSL/TLS.
> > * HBaseIO is now able to do dynamic work rebalancing
> > * KinesisIO uses a more accurate watermark (based on
> approximateArrivalTimestamp)
> > * TextIO now supports custom delimiter and like AvroIO, supports the
> readAll
> > pattern,
> > * Performance improvements on JdbcIO when it has to read lot of rows
> > * Kafka write supports Exactly-Once pattern (introduce in Kafka 0.11.x)
> >
> > - A new DSL has been introduced: the SQL DSL !
> >
> > We are now focus on 2.3.0 release with new improvements and features !
> >
> > Stay tuned !
> >
> > JB on behalf of the Apache Beam community.
> >
> > On 12/02/2017 11:40 PM, Reuven Lax wrote:
> >> The Apache Beam community is pleased to announce the availability of the
> >> 2.2.0 release.
> >>
> >> This release adds support for generic file sources and sinks (beyond TextIO
> >> and AvroIO) using FileIO, including support for dynamic filenames using
> >> readAll; this allows streaming pipelines to now read from files by
> >> continuously monitoring a directory for new filw. Many other IOs are
> improved,
> >> notably including exactly-once support for the Kafka sink. Initial
> support for
> >> BEAM-SQL is also included in this release. For a more-complete list of major
> >> changes in the release, please refer to the release notes [2].
> >>
> >> The 2.2.0 release is now the recommended version; we encourage everyone to
> >> upgrade from any earlier releases.
> >>
> >> We’d like to invite everyone to try out Apache Beam today and consider
> >> joining our vibrant community. We welcome feedback, contribution and
> >> participation through our mailing lists, issue tracker, pull requests, and
> >> events.
> >>
> >> - Reuven Lax, on behalf of the Apache Beam community.
> >>
> >> [1] https://beam.apache.org/get-started/downloads/
> >> [2]
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Apache Beam, version 2.2.0
Posted by Lukasz Cwik <lc...@google.com>.
I also believe we were still in the investigatory phase for dropping
support for Java 7.
On Mon, Dec 4, 2017 at 2:22 PM, Eugene Kirpichov <ki...@google.com>
wrote:
> Thanks JB for sending the detailed notes about new stuff in 2.2.0! A lot
> of exciting things indeed.
>
> Regarding Java 8: I thought our consensus was to have the release notes
> say that we're *considering* going Java8-only, and use that to get more
> opinions from the user community - but I can't find the emails that made me
> think so.
>
> +Ismaël Mejía <ie...@gmail.com> - do you think we should formally
> conclude the vote on the thread [VOTE] [DISCUSSION] Remove support for Java
> 7?
> Or should we take more steps - e.g. perhaps tweet a link to that thread
> from the Beam twitter account, ask people to chime in, and wait for say 2
> weeks before declaring a conclusion?
>
> Let's also have a process JIRA for going Java8. I've filed one:
> https://issues.apache.org/jira/browse/BEAM-3285
>
> On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Just an important note that we forgot to mention.
>>
>> !! The 2.2.0 release will be the last one supporting Spark 1.x and Java 7
>> !!
>>
>> Starting from Beam 2.3.0, the Spark runner will work only with Spark 2.x
>> and we
>> will focus only Java 8.
>>
>> Regards
>> JB
>>
>> On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
>> > Thanks Reuven !
>> >
>> > I would like to emphasize on some highlights in 2.2.0 release:
>> >
>> > - New IOs have been introduced:
>> > * TikaIO leveraging Apache Tika, allowing the deal with a lot of
>> different
>> > data formats
>> > * RedisIO to read and write key/value pairs from a Redis server. This
>> IO will
>> > be soon extended to Redis PubSub.
>> > * FileIO provides transforms for working with files (raw).
>> Especially, it
>> > provides matching file patterns and read on patterns. It can be easily
>> extended
>> > for a specific format (like we do in AvroIO or TextIO now).
>> > * SolrIO to interact with Apache Solr (Lucene)
>> >
>> > - On the other hand, improvements have been performed on existing IOs:
>> > * We started to introduce readAll pattern in IOs (AvroIO, TextIO,
>> JdbcIO,
>> > ...), allowing to pass "request" arguments via an input PCollection.
>> > * ElasticsearchIO has an improved support of different Elasticsearch
>> version
>> > (including Elasticsearch 5.x). It also now supports SSL/TLS.
>> > * HBaseIO is now able to do dynamic work rebalancing
>> > * KinesisIO uses a more accurate watermark (based on
>> approximateArrivalTimestamp)
>> > * TextIO now supports custom delimiter and like AvroIO, supports the
>> readAll
>> > pattern,
>> > * Performance improvements on JdbcIO when it has to read lot of rows
>> > * Kafka write supports Exactly-Once pattern (introduce in Kafka
>> 0.11.x)
>> >
>> > - A new DSL has been introduced: the SQL DSL !
>> >
>> > We are now focus on 2.3.0 release with new improvements and features !
>> >
>> > Stay tuned !
>> >
>> > JB on behalf of the Apache Beam community.
>> >
>> > On 12/02/2017 11:40 PM, Reuven Lax wrote:
>> >> The Apache Beam community is pleased to announce the availability of
>> the
>> >> 2.2.0 release.
>> >>
>> >> This release adds support for generic file sources and sinks (beyond
>> TextIO
>> >> and AvroIO) using FileIO, including support for dynamic filenames using
>> >> readAll; this allows streaming pipelines to now read from files by
>> >> continuously monitoring a directory for new filw. Many other IOs are
>> improved,
>> >> notably including exactly-once support for the Kafka sink. Initial
>> support for
>> >> BEAM-SQL is also included in this release. For a more-complete list of
>> major
>> >> changes in the release, please refer to the release notes [2].
>> >>
>> >> The 2.2.0 release is now the recommended version; we encourage
>> everyone to
>> >> upgrade from any earlier releases.
>> >>
>> >> We’d like to invite everyone to try out Apache Beam today and consider
>> >> joining our vibrant community. We welcome feedback, contribution and
>> >> participation through our mailing lists, issue tracker, pull requests,
>> and
>> >> events.
>> >>
>> >> - Reuven Lax, on behalf of the Apache Beam community.
>> >>
>> >> [1] https://beam.apache.org/get-started/downloads/
>> >> [2]
>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12319527&version=12341044
>> >>
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
Re: Apache Beam, version 2.2.0
Posted by Eugene Kirpichov <ki...@google.com>.
Thanks JB for sending the detailed notes about new stuff in 2.2.0! A lot of
exciting things indeed.
Regarding Java 8: I thought our consensus was to have the release notes say
that we're *considering* going Java8-only, and use that to get more
opinions from the user community - but I can't find the emails that made me
think so.
+Ismaël Mejía <ie...@gmail.com> - do you think we should formally
conclude the vote on the thread [VOTE] [DISCUSSION] Remove support for Java
7?
Or should we take more steps - e.g. perhaps tweet a link to that thread
from the Beam twitter account, ask people to chime in, and wait for say 2
weeks before declaring a conclusion?
Let's also have a process JIRA for going Java8. I've filed one:
https://issues.apache.org/jira/browse/BEAM-3285
On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Just an important note that we forgot to mention.
>
> !! The 2.2.0 release will be the last one supporting Spark 1.x and Java 7
> !!
>
> Starting from Beam 2.3.0, the Spark runner will work only with Spark 2.x
> and we
> will focus only Java 8.
>
> Regards
> JB
>
> On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
> > Thanks Reuven !
> >
> > I would like to emphasize on some highlights in 2.2.0 release:
> >
> > - New IOs have been introduced:
> > * TikaIO leveraging Apache Tika, allowing the deal with a lot of
> different
> > data formats
> > * RedisIO to read and write key/value pairs from a Redis server. This
> IO will
> > be soon extended to Redis PubSub.
> > * FileIO provides transforms for working with files (raw). Especially,
> it
> > provides matching file patterns and read on patterns. It can be easily
> extended
> > for a specific format (like we do in AvroIO or TextIO now).
> > * SolrIO to interact with Apache Solr (Lucene)
> >
> > - On the other hand, improvements have been performed on existing IOs:
> > * We started to introduce readAll pattern in IOs (AvroIO, TextIO,
> JdbcIO,
> > ...), allowing to pass "request" arguments via an input PCollection.
> > * ElasticsearchIO has an improved support of different Elasticsearch
> version
> > (including Elasticsearch 5.x). It also now supports SSL/TLS.
> > * HBaseIO is now able to do dynamic work rebalancing
> > * KinesisIO uses a more accurate watermark (based on
> approximateArrivalTimestamp)
> > * TextIO now supports custom delimiter and like AvroIO, supports the
> readAll
> > pattern,
> > * Performance improvements on JdbcIO when it has to read lot of rows
> > * Kafka write supports Exactly-Once pattern (introduce in Kafka 0.11.x)
> >
> > - A new DSL has been introduced: the SQL DSL !
> >
> > We are now focus on 2.3.0 release with new improvements and features !
> >
> > Stay tuned !
> >
> > JB on behalf of the Apache Beam community.
> >
> > On 12/02/2017 11:40 PM, Reuven Lax wrote:
> >> The Apache Beam community is pleased to announce the availability of the
> >> 2.2.0 release.
> >>
> >> This release adds support for generic file sources and sinks (beyond
> TextIO
> >> and AvroIO) using FileIO, including support for dynamic filenames using
> >> readAll; this allows streaming pipelines to now read from files by
> >> continuously monitoring a directory for new filw. Many other IOs are
> improved,
> >> notably including exactly-once support for the Kafka sink. Initial
> support for
> >> BEAM-SQL is also included in this release. For a more-complete list of
> major
> >> changes in the release, please refer to the release notes [2].
> >>
> >> The 2.2.0 release is now the recommended version; we encourage everyone
> to
> >> upgrade from any earlier releases.
> >>
> >> We’d like to invite everyone to try out Apache Beam today and consider
> >> joining our vibrant community. We welcome feedback, contribution and
> >> participation through our mailing lists, issue tracker, pull requests,
> and
> >> events.
> >>
> >> - Reuven Lax, on behalf of the Apache Beam community.
> >>
> >> [1] https://beam.apache.org/get-started/downloads/
> >> [2]
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Apache Beam, version 2.2.0
Posted by Vilhelm von Ehrenheim <vo...@gmail.com>.
I'm super excited about this release! Great work everyone involved!
On Mon, Dec 4, 2017 at 10:58 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Just an important note that we forgot to mention.
>
> !! The 2.2.0 release will be the last one supporting Spark 1.x and Java 7
> !!
>
> Starting from Beam 2.3.0, the Spark runner will work only with Spark 2.x
> and we will focus only Java 8.
>
> Regards
> JB
>
>
> On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
>
>> Thanks Reuven !
>>
>> I would like to emphasize on some highlights in 2.2.0 release:
>>
>> - New IOs have been introduced:
>> * TikaIO leveraging Apache Tika, allowing the deal with a lot of
>> different data formats
>> * RedisIO to read and write key/value pairs from a Redis server. This
>> IO will be soon extended to Redis PubSub.
>> * FileIO provides transforms for working with files (raw). Especially,
>> it provides matching file patterns and read on patterns. It can be easily
>> extended for a specific format (like we do in AvroIO or TextIO now).
>> * SolrIO to interact with Apache Solr (Lucene)
>>
>> - On the other hand, improvements have been performed on existing IOs:
>> * We started to introduce readAll pattern in IOs (AvroIO, TextIO,
>> JdbcIO, ...), allowing to pass "request" arguments via an input PCollection.
>> * ElasticsearchIO has an improved support of different Elasticsearch
>> version (including Elasticsearch 5.x). It also now supports SSL/TLS.
>> * HBaseIO is now able to do dynamic work rebalancing
>> * KinesisIO uses a more accurate watermark (based on
>> approximateArrivalTimestamp)
>> * TextIO now supports custom delimiter and like AvroIO, supports the
>> readAll pattern,
>> * Performance improvements on JdbcIO when it has to read lot of rows
>> * Kafka write supports Exactly-Once pattern (introduce in Kafka 0.11.x)
>>
>> - A new DSL has been introduced: the SQL DSL !
>>
>> We are now focus on 2.3.0 release with new improvements and features !
>>
>> Stay tuned !
>>
>> JB on behalf of the Apache Beam community.
>>
>> On 12/02/2017 11:40 PM, Reuven Lax wrote:
>>
>>> The Apache Beam community is pleased to announce the availability of the
>>> 2.2.0 release.
>>>
>>> This release adds support for generic file sources and sinks (beyond
>>> TextIO and AvroIO) using FileIO, including support for dynamic filenames
>>> using readAll; this allows streaming pipelines to now read from files by
>>> continuously monitoring a directory for new filw. Many other IOs are
>>> improved, notably including exactly-once support for the Kafka sink.
>>> Initial support for BEAM-SQL is also included in this release. For a
>>> more-complete list of major changes in the release, please refer to the
>>> release notes [2].
>>>
>>> The 2.2.0 release is now the recommended version; we encourage everyone
>>> to
>>> upgrade from any earlier releases.
>>>
>>> We’d like to invite everyone to try out Apache Beam today and consider
>>> joining our vibrant community. We welcome feedback, contribution and
>>> participation through our mailing lists, issue tracker, pull requests,
>>> and
>>> events.
>>>
>>> - Reuven Lax, on behalf of the Apache Beam community.
>>>
>>> [1] https://beam.apache.org/get-started/downloads/
>>> [2]
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>>> ctId=12319527&version=12341044
>>>
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Apache Beam, version 2.2.0
Posted by Eugene Kirpichov <ki...@google.com>.
Thanks JB for sending the detailed notes about new stuff in 2.2.0! A lot of
exciting things indeed.
Regarding Java 8: I thought our consensus was to have the release notes say
that we're *considering* going Java8-only, and use that to get more
opinions from the user community - but I can't find the emails that made me
think so.
+Ismaël Mejía <ie...@gmail.com> - do you think we should formally
conclude the vote on the thread [VOTE] [DISCUSSION] Remove support for Java
7?
Or should we take more steps - e.g. perhaps tweet a link to that thread
from the Beam twitter account, ask people to chime in, and wait for say 2
weeks before declaring a conclusion?
Let's also have a process JIRA for going Java8. I've filed one:
https://issues.apache.org/jira/browse/BEAM-3285
On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Just an important note that we forgot to mention.
>
> !! The 2.2.0 release will be the last one supporting Spark 1.x and Java 7
> !!
>
> Starting from Beam 2.3.0, the Spark runner will work only with Spark 2.x
> and we
> will focus only Java 8.
>
> Regards
> JB
>
> On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
> > Thanks Reuven !
> >
> > I would like to emphasize on some highlights in 2.2.0 release:
> >
> > - New IOs have been introduced:
> > * TikaIO leveraging Apache Tika, allowing the deal with a lot of
> different
> > data formats
> > * RedisIO to read and write key/value pairs from a Redis server. This
> IO will
> > be soon extended to Redis PubSub.
> > * FileIO provides transforms for working with files (raw). Especially,
> it
> > provides matching file patterns and read on patterns. It can be easily
> extended
> > for a specific format (like we do in AvroIO or TextIO now).
> > * SolrIO to interact with Apache Solr (Lucene)
> >
> > - On the other hand, improvements have been performed on existing IOs:
> > * We started to introduce readAll pattern in IOs (AvroIO, TextIO,
> JdbcIO,
> > ...), allowing to pass "request" arguments via an input PCollection.
> > * ElasticsearchIO has an improved support of different Elasticsearch
> version
> > (including Elasticsearch 5.x). It also now supports SSL/TLS.
> > * HBaseIO is now able to do dynamic work rebalancing
> > * KinesisIO uses a more accurate watermark (based on
> approximateArrivalTimestamp)
> > * TextIO now supports custom delimiter and like AvroIO, supports the
> readAll
> > pattern,
> > * Performance improvements on JdbcIO when it has to read lot of rows
> > * Kafka write supports Exactly-Once pattern (introduce in Kafka 0.11.x)
> >
> > - A new DSL has been introduced: the SQL DSL !
> >
> > We are now focus on 2.3.0 release with new improvements and features !
> >
> > Stay tuned !
> >
> > JB on behalf of the Apache Beam community.
> >
> > On 12/02/2017 11:40 PM, Reuven Lax wrote:
> >> The Apache Beam community is pleased to announce the availability of the
> >> 2.2.0 release.
> >>
> >> This release adds support for generic file sources and sinks (beyond
> TextIO
> >> and AvroIO) using FileIO, including support for dynamic filenames using
> >> readAll; this allows streaming pipelines to now read from files by
> >> continuously monitoring a directory for new filw. Many other IOs are
> improved,
> >> notably including exactly-once support for the Kafka sink. Initial
> support for
> >> BEAM-SQL is also included in this release. For a more-complete list of
> major
> >> changes in the release, please refer to the release notes [2].
> >>
> >> The 2.2.0 release is now the recommended version; we encourage everyone
> to
> >> upgrade from any earlier releases.
> >>
> >> We’d like to invite everyone to try out Apache Beam today and consider
> >> joining our vibrant community. We welcome feedback, contribution and
> >> participation through our mailing lists, issue tracker, pull requests,
> and
> >> events.
> >>
> >> - Reuven Lax, on behalf of the Apache Beam community.
> >>
> >> [1] https://beam.apache.org/get-started/downloads/
> >> [2]
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Apache Beam, version 2.2.0
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Just an important note that we forgot to mention.
!! The 2.2.0 release will be the last one supporting Spark 1.x and Java 7 !!
Starting from Beam 2.3.0, the Spark runner will work only with Spark 2.x and we
will focus only Java 8.
Regards
JB
On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
> Thanks Reuven !
>
> I would like to emphasize on some highlights in 2.2.0 release:
>
> - New IOs have been introduced:
> * TikaIO leveraging Apache Tika, allowing the deal with a lot of different
> data formats
> * RedisIO to read and write key/value pairs from a Redis server. This IO will
> be soon extended to Redis PubSub.
> * FileIO provides transforms for working with files (raw). Especially, it
> provides matching file patterns and read on patterns. It can be easily extended
> for a specific format (like we do in AvroIO or TextIO now).
> * SolrIO to interact with Apache Solr (Lucene)
>
> - On the other hand, improvements have been performed on existing IOs:
> * We started to introduce readAll pattern in IOs (AvroIO, TextIO, JdbcIO,
> ...), allowing to pass "request" arguments via an input PCollection.
> * ElasticsearchIO has an improved support of different Elasticsearch version
> (including Elasticsearch 5.x). It also now supports SSL/TLS.
> * HBaseIO is now able to do dynamic work rebalancing
> * KinesisIO uses a more accurate watermark (based on approximateArrivalTimestamp)
> * TextIO now supports custom delimiter and like AvroIO, supports the readAll
> pattern,
> * Performance improvements on JdbcIO when it has to read lot of rows
> * Kafka write supports Exactly-Once pattern (introduce in Kafka 0.11.x)
>
> - A new DSL has been introduced: the SQL DSL !
>
> We are now focus on 2.3.0 release with new improvements and features !
>
> Stay tuned !
>
> JB on behalf of the Apache Beam community.
>
> On 12/02/2017 11:40 PM, Reuven Lax wrote:
>> The Apache Beam community is pleased to announce the availability of the
>> 2.2.0 release.
>>
>> This release adds support for generic file sources and sinks (beyond TextIO
>> and AvroIO) using FileIO, including support for dynamic filenames using
>> readAll; this allows streaming pipelines to now read from files by
>> continuously monitoring a directory for new filw. Many other IOs are improved,
>> notably including exactly-once support for the Kafka sink. Initial support for
>> BEAM-SQL is also included in this release. For a more-complete list of major
>> changes in the release, please refer to the release notes [2].
>>
>> The 2.2.0 release is now the recommended version; we encourage everyone to
>> upgrade from any earlier releases.
>>
>> We’d like to invite everyone to try out Apache Beam today and consider
>> joining our vibrant community. We welcome feedback, contribution and
>> participation through our mailing lists, issue tracker, pull requests, and
>> events.
>>
>> - Reuven Lax, on behalf of the Apache Beam community.
>>
>> [1] https://beam.apache.org/get-started/downloads/
>> [2]
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Apache Beam, version 2.2.0
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Just an important note that we forgot to mention.
!! The 2.2.0 release will be the last one supporting Spark 1.x and Java 7 !!
Starting from Beam 2.3.0, the Spark runner will work only with Spark 2.x and we
will focus only Java 8.
Regards
JB
On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
> Thanks Reuven !
>
> I would like to emphasize on some highlights in 2.2.0 release:
>
> - New IOs have been introduced:
> * TikaIO leveraging Apache Tika, allowing the deal with a lot of different
> data formats
> * RedisIO to read and write key/value pairs from a Redis server. This IO will
> be soon extended to Redis PubSub.
> * FileIO provides transforms for working with files (raw). Especially, it
> provides matching file patterns and read on patterns. It can be easily extended
> for a specific format (like we do in AvroIO or TextIO now).
> * SolrIO to interact with Apache Solr (Lucene)
>
> - On the other hand, improvements have been performed on existing IOs:
> * We started to introduce readAll pattern in IOs (AvroIO, TextIO, JdbcIO,
> ...), allowing to pass "request" arguments via an input PCollection.
> * ElasticsearchIO has an improved support of different Elasticsearch version
> (including Elasticsearch 5.x). It also now supports SSL/TLS.
> * HBaseIO is now able to do dynamic work rebalancing
> * KinesisIO uses a more accurate watermark (based on approximateArrivalTimestamp)
> * TextIO now supports custom delimiter and like AvroIO, supports the readAll
> pattern,
> * Performance improvements on JdbcIO when it has to read lot of rows
> * Kafka write supports Exactly-Once pattern (introduce in Kafka 0.11.x)
>
> - A new DSL has been introduced: the SQL DSL !
>
> We are now focus on 2.3.0 release with new improvements and features !
>
> Stay tuned !
>
> JB on behalf of the Apache Beam community.
>
> On 12/02/2017 11:40 PM, Reuven Lax wrote:
>> The Apache Beam community is pleased to announce the availability of the
>> 2.2.0 release.
>>
>> This release adds support for generic file sources and sinks (beyond TextIO
>> and AvroIO) using FileIO, including support for dynamic filenames using
>> readAll; this allows streaming pipelines to now read from files by
>> continuously monitoring a directory for new filw. Many other IOs are improved,
>> notably including exactly-once support for the Kafka sink. Initial support for
>> BEAM-SQL is also included in this release. For a more-complete list of major
>> changes in the release, please refer to the release notes [2].
>>
>> The 2.2.0 release is now the recommended version; we encourage everyone to
>> upgrade from any earlier releases.
>>
>> We’d like to invite everyone to try out Apache Beam today and consider
>> joining our vibrant community. We welcome feedback, contribution and
>> participation through our mailing lists, issue tracker, pull requests, and
>> events.
>>
>> - Reuven Lax, on behalf of the Apache Beam community.
>>
>> [1] https://beam.apache.org/get-started/downloads/
>> [2]
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Apache Beam, version 2.2.0
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Thanks Reuven !
I would like to emphasize on some highlights in 2.2.0 release:
- New IOs have been introduced:
* TikaIO leveraging Apache Tika, allowing the deal with a lot of different
data formats
* RedisIO to read and write key/value pairs from a Redis server. This IO will
be soon extended to Redis PubSub.
* FileIO provides transforms for working with files (raw). Especially, it
provides matching file patterns and read on patterns. It can be easily extended
for a specific format (like we do in AvroIO or TextIO now).
* SolrIO to interact with Apache Solr (Lucene)
- On the other hand, improvements have been performed on existing IOs:
* We started to introduce readAll pattern in IOs (AvroIO, TextIO, JdbcIO,
...), allowing to pass "request" arguments via an input PCollection.
* ElasticsearchIO has an improved support of different Elasticsearch version
(including Elasticsearch 5.x). It also now supports SSL/TLS.
* HBaseIO is now able to do dynamic work rebalancing
* KinesisIO uses a more accurate watermark (based on approximateArrivalTimestamp)
* TextIO now supports custom delimiter and like AvroIO, supports the readAll
pattern,
* Performance improvements on JdbcIO when it has to read lot of rows
* Kafka write supports Exactly-Once pattern (introduce in Kafka 0.11.x)
- A new DSL has been introduced: the SQL DSL !
We are now focus on 2.3.0 release with new improvements and features !
Stay tuned !
JB on behalf of the Apache Beam community.
On 12/02/2017 11:40 PM, Reuven Lax wrote:
> The Apache Beam community is pleased to announce the availability of the
> 2.2.0 release.
>
> This release adds support for generic file sources and sinks (beyond TextIO and
> AvroIO) using FileIO, including support for dynamic filenames using readAll;
> this allows streaming pipelines to now read from files by continuously
> monitoring a directory for new filw. Many other IOs are improved, notably
> including exactly-once support for the Kafka sink. Initial support for BEAM-SQL
> is also included in this release. For a more-complete list of major changes in
> the release, please refer to the release notes [2].
>
> The 2.2.0 release is now the recommended version; we encourage everyone to
> upgrade from any earlier releases.
>
> We’d like to invite everyone to try out Apache Beam today and consider
> joining our vibrant community. We welcome feedback, contribution and
> participation through our mailing lists, issue tracker, pull requests, and
> events.
>
> - Reuven Lax, on behalf of the Apache Beam community.
>
> [1] https://beam.apache.org/get-started/downloads/
> [2]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Apache Beam, version 2.2.0
Posted by Reuven Lax <re...@google.com>.
As this is my work gmail, I don't think I'm able to alter the SMTP servers
used to send email.
Reuven
On Sat, Dec 2, 2017 at 10:42 PM, Thomas Weise <th...@apache.org> wrote:
> You can send via mail-relay.apache.org:
>
> https://reference.apache.org/committer/email#
> sendingemailfromyourapacheorgemailaddress
>
>
> On Sat, Dec 2, 2017 at 7:13 PM, Reuven Lax <re...@google.com> wrote:
>
>> I've only ever had my @apache.org address set up as a forwarding
>> address. How do I go about sending from it?
>>
>> announce@apache.org bounces with the same error.
>>
>> On Dec 2, 2017 7:48 PM, "Thomas Weise" <th...@apache.org> wrote:
>>
>>> That's probably because the email wasn't sent from your @apache.org
>>> address. Shouldn't it also go to announce@apache.org ?
>>>
>>> On Sat, Dec 2, 2017 at 2:41 PM, Reuven Lax <re...@google.com> wrote:
>>>
>>>> My attempt to send this to the user list bounced.
>>>>
>>>> s
>>>>
>>>> On Sat, Dec 2, 2017 at 5:40 PM, Reuven Lax <re...@google.com> wrote:
>>>>
>>>>> The Apache Beam community is pleased to announce the availability of
>>>>> the
>>>>> 2.2.0 release.
>>>>>
>>>>> This release adds support for generic file sources and sinks (beyond
>>>>> TextIO and AvroIO) using FileIO, including support for dynamic filenames
>>>>> using readAll; this allows streaming pipelines to now read from files by
>>>>> continuously monitoring a directory for new filw. Many other IOs are
>>>>> improved, notably including exactly-once support for the Kafka sink.
>>>>> Initial support for BEAM-SQL is also included in this release. For a
>>>>> more-complete list of major changes in the release, please refer to the
>>>>> release notes [2].
>>>>>
>>>>> The 2.2.0 release is now the recommended version; we encourage
>>>>> everyone to
>>>>> upgrade from any earlier releases.
>>>>>
>>>>> We’d like to invite everyone to try out Apache Beam today and consider
>>>>> joining our vibrant community. We welcome feedback, contribution and
>>>>> participation through our mailing lists, issue tracker, pull requests,
>>>>> and
>>>>> events.
>>>>>
>>>>> - Reuven Lax, on behalf of the Apache Beam community.
>>>>>
>>>>> [1] https://beam.apache.org/get-started/downloads/
>>>>> [2]
>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>>>>> ctId=12319527&version=12341044
>>>>>
>>>>
>>>>
>>>
>
Re: Apache Beam, version 2.2.0
Posted by Thomas Weise <th...@apache.org>.
You can send via mail-relay.apache.org:
https://reference.apache.org/committer/email#sendingemailfromyourapacheorgemailaddress
On Sat, Dec 2, 2017 at 7:13 PM, Reuven Lax <re...@google.com> wrote:
> I've only ever had my @apache.org address set up as a forwarding address.
> How do I go about sending from it?
>
> announce@apache.org bounces with the same error.
>
> On Dec 2, 2017 7:48 PM, "Thomas Weise" <th...@apache.org> wrote:
>
>> That's probably because the email wasn't sent from your @apache.org
>> address. Shouldn't it also go to announce@apache.org ?
>>
>> On Sat, Dec 2, 2017 at 2:41 PM, Reuven Lax <re...@google.com> wrote:
>>
>>> My attempt to send this to the user list bounced.
>>>
>>> s
>>>
>>> On Sat, Dec 2, 2017 at 5:40 PM, Reuven Lax <re...@google.com> wrote:
>>>
>>>> The Apache Beam community is pleased to announce the availability of the
>>>> 2.2.0 release.
>>>>
>>>> This release adds support for generic file sources and sinks (beyond
>>>> TextIO and AvroIO) using FileIO, including support for dynamic filenames
>>>> using readAll; this allows streaming pipelines to now read from files by
>>>> continuously monitoring a directory for new filw. Many other IOs are
>>>> improved, notably including exactly-once support for the Kafka sink.
>>>> Initial support for BEAM-SQL is also included in this release. For a
>>>> more-complete list of major changes in the release, please refer to the
>>>> release notes [2].
>>>>
>>>> The 2.2.0 release is now the recommended version; we encourage everyone
>>>> to
>>>> upgrade from any earlier releases.
>>>>
>>>> We’d like to invite everyone to try out Apache Beam today and consider
>>>> joining our vibrant community. We welcome feedback, contribution and
>>>> participation through our mailing lists, issue tracker, pull requests,
>>>> and
>>>> events.
>>>>
>>>> - Reuven Lax, on behalf of the Apache Beam community.
>>>>
>>>> [1] https://beam.apache.org/get-started/downloads/
>>>> [2]
>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>>>> ctId=12319527&version=12341044
>>>>
>>>
>>>
>>
Re: Apache Beam, version 2.2.0
Posted by Reuven Lax <re...@google.com>.
I've only ever had my @apache.org address set up as a forwarding address.
How do I go about sending from it?
announce@apache.org bounces with the same error.
On Dec 2, 2017 7:48 PM, "Thomas Weise" <th...@apache.org> wrote:
> That's probably because the email wasn't sent from your @apache.org
> address. Shouldn't it also go to announce@apache.org ?
>
> On Sat, Dec 2, 2017 at 2:41 PM, Reuven Lax <re...@google.com> wrote:
>
>> My attempt to send this to the user list bounced.
>>
>> s
>>
>> On Sat, Dec 2, 2017 at 5:40 PM, Reuven Lax <re...@google.com> wrote:
>>
>>> The Apache Beam community is pleased to announce the availability of the
>>> 2.2.0 release.
>>>
>>> This release adds support for generic file sources and sinks (beyond
>>> TextIO and AvroIO) using FileIO, including support for dynamic filenames
>>> using readAll; this allows streaming pipelines to now read from files by
>>> continuously monitoring a directory for new filw. Many other IOs are
>>> improved, notably including exactly-once support for the Kafka sink.
>>> Initial support for BEAM-SQL is also included in this release. For a
>>> more-complete list of major changes in the release, please refer to the
>>> release notes [2].
>>>
>>> The 2.2.0 release is now the recommended version; we encourage everyone
>>> to
>>> upgrade from any earlier releases.
>>>
>>> We’d like to invite everyone to try out Apache Beam today and consider
>>> joining our vibrant community. We welcome feedback, contribution and
>>> participation through our mailing lists, issue tracker, pull requests,
>>> and
>>> events.
>>>
>>> - Reuven Lax, on behalf of the Apache Beam community.
>>>
>>> [1] https://beam.apache.org/get-started/downloads/
>>> [2]
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>>> ctId=12319527&version=12341044
>>>
>>
>>
>
Re: Apache Beam, version 2.2.0
Posted by Thomas Weise <th...@apache.org>.
That's probably because the email wasn't sent from your @apache.org
address. Shouldn't it also go to announce@apache.org ?
On Sat, Dec 2, 2017 at 2:41 PM, Reuven Lax <re...@google.com> wrote:
> My attempt to send this to the user list bounced.
>
> s
>
> On Sat, Dec 2, 2017 at 5:40 PM, Reuven Lax <re...@google.com> wrote:
>
>> The Apache Beam community is pleased to announce the availability of the
>> 2.2.0 release.
>>
>> This release adds support for generic file sources and sinks (beyond
>> TextIO and AvroIO) using FileIO, including support for dynamic filenames
>> using readAll; this allows streaming pipelines to now read from files by
>> continuously monitoring a directory for new filw. Many other IOs are
>> improved, notably including exactly-once support for the Kafka sink.
>> Initial support for BEAM-SQL is also included in this release. For a
>> more-complete list of major changes in the release, please refer to the
>> release notes [2].
>>
>> The 2.2.0 release is now the recommended version; we encourage everyone to
>> upgrade from any earlier releases.
>>
>> We’d like to invite everyone to try out Apache Beam today and consider
>> joining our vibrant community. We welcome feedback, contribution and
>> participation through our mailing lists, issue tracker, pull requests, and
>> events.
>>
>> - Reuven Lax, on behalf of the Apache Beam community.
>>
>> [1] https://beam.apache.org/get-started/downloads/
>> [2]
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>> ctId=12319527&version=12341044
>>
>
>
Re: Apache Beam, version 2.2.0
Posted by Eugene Kirpichov <ki...@google.com>.
Woohoo!!
When you're able to send to user@ - note that we should also link to the
thread with voting on removal of Java7 support
https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
On Sat, Dec 2, 2017 at 2:41 PM Reuven Lax <re...@google.com> wrote:
> My attempt to send this to the user list bounced.
>
> s
>
> On Sat, Dec 2, 2017 at 5:40 PM, Reuven Lax <re...@google.com> wrote:
>
>> The Apache Beam community is pleased to announce the availability of the
>> 2.2.0 release.
>>
>> This release adds support for generic file sources and sinks (beyond
>> TextIO and AvroIO) using FileIO, including support for dynamic filenames
>> using readAll; this allows streaming pipelines to now read from files by
>> continuously monitoring a directory for new filw. Many other IOs are
>> improved, notably including exactly-once support for the Kafka sink.
>> Initial support for BEAM-SQL is also included in this release. For a
>> more-complete list of major changes in the release, please refer to the
>> release notes [2].
>>
>> The 2.2.0 release is now the recommended version; we encourage everyone to
>> upgrade from any earlier releases.
>>
>> We’d like to invite everyone to try out Apache Beam today and consider
>> joining our vibrant community. We welcome feedback, contribution and
>> participation through our mailing lists, issue tracker, pull requests, and
>> events.
>>
>> - Reuven Lax, on behalf of the Apache Beam community.
>>
>> [1] https://beam.apache.org/get-started/downloads/
>> [2]
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
>>
>
>
Re: Apache Beam, version 2.2.0
Posted by Reuven Lax <re...@google.com>.
My attempt to send this to the user list bounced.
s
On Sat, Dec 2, 2017 at 5:40 PM, Reuven Lax <re...@google.com> wrote:
> The Apache Beam community is pleased to announce the availability of the
> 2.2.0 release.
>
> This release adds support for generic file sources and sinks (beyond
> TextIO and AvroIO) using FileIO, including support for dynamic filenames
> using readAll; this allows streaming pipelines to now read from files by
> continuously monitoring a directory for new filw. Many other IOs are
> improved, notably including exactly-once support for the Kafka sink.
> Initial support for BEAM-SQL is also included in this release. For a
> more-complete list of major changes in the release, please refer to the
> release notes [2].
>
> The 2.2.0 release is now the recommended version; we encourage everyone to
> upgrade from any earlier releases.
>
> We’d like to invite everyone to try out Apache Beam today and consider
> joining our vibrant community. We welcome feedback, contribution and
> participation through our mailing lists, issue tracker, pull requests, and
> events.
>
> - Reuven Lax, on behalf of the Apache Beam community.
>
> [1] https://beam.apache.org/get-started/downloads/
> [2]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527&version=12341044
>
Re: Apache Beam, version 2.2.0
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Thanks Reuven !
I would like to emphasize on some highlights in 2.2.0 release:
- New IOs have been introduced:
* TikaIO leveraging Apache Tika, allowing the deal with a lot of different
data formats
* RedisIO to read and write key/value pairs from a Redis server. This IO will
be soon extended to Redis PubSub.
* FileIO provides transforms for working with files (raw). Especially, it
provides matching file patterns and read on patterns. It can be easily extended
for a specific format (like we do in AvroIO or TextIO now).
* SolrIO to interact with Apache Solr (Lucene)
- On the other hand, improvements have been performed on existing IOs:
* We started to introduce readAll pattern in IOs (AvroIO, TextIO, JdbcIO,
...), allowing to pass "request" arguments via an input PCollection.
* ElasticsearchIO has an improved support of different Elasticsearch version
(including Elasticsearch 5.x). It also now supports SSL/TLS.
* HBaseIO is now able to do dynamic work rebalancing
* KinesisIO uses a more accurate watermark (based on approximateArrivalTimestamp)
* TextIO now supports custom delimiter and like AvroIO, supports the readAll
pattern,
* Performance improvements on JdbcIO when it has to read lot of rows
* Kafka write supports Exactly-Once pattern (introduce in Kafka 0.11.x)
- A new DSL has been introduced: the SQL DSL !
We are now focus on 2.3.0 release with new improvements and features !
Stay tuned !
JB on behalf of the Apache Beam community.
On 12/02/2017 11:40 PM, Reuven Lax wrote:
> The Apache Beam community is pleased to announce the availability of the
> 2.2.0 release.
>
> This release adds support for generic file sources and sinks (beyond TextIO and
> AvroIO) using FileIO, including support for dynamic filenames using readAll;
> this allows streaming pipelines to now read from files by continuously
> monitoring a directory for new filw. Many other IOs are improved, notably
> including exactly-once support for the Kafka sink. Initial support for BEAM-SQL
> is also included in this release. For a more-complete list of major changes in
> the release, please refer to the release notes [2].
>
> The 2.2.0 release is now the recommended version; we encourage everyone to
> upgrade from any earlier releases.
>
> We’d like to invite everyone to try out Apache Beam today and consider
> joining our vibrant community. We welcome feedback, contribution and
> participation through our mailing lists, issue tracker, pull requests, and
> events.
>
> - Reuven Lax, on behalf of the Apache Beam community.
>
> [1] https://beam.apache.org/get-started/downloads/
> [2]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com