You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Frances Perry <fj...@google.com.INVALID> on 2016/08/01 14:36:34 UTC
[REFLECT] Beam’s Half Birthday!
Hi Beamers!
It’s been six months today since Beam was accepted into incubation. It’s
thrilling how far we’ve come since then!
I’d like to volunteer to put together a post on the Beam blog summarizing
our progress since February. Here’s a starting point... What am I missing
that we should include? What makes you proud?
By the numbers:
* 48,238 lines of preexisting code donated by Cloudera, dataArtisans, and
Google.
* 761 pull requests from 45 contributors.
* 498 Jira issues opened and 245 resolved.
* 1 incubating release (and another 1 in progress).
* 4200 hours of automated tests.
* 161 subscribers / 606 messages on user@.
* 217 subscribers / 1205 messages on dev@.
There’s been a lot of technical progress, including:
* Refactoring of the entire codebase, examples, and tests to be truly
runner-independent.
* New functionality in the Apache Flink runner for timestamps/windows in
batch and bounded sources and side inputs in streaming mode.
* Work in progress to upgrade the Apache Spark runner to use Spark 2.0.
* Several new runners from the wider Apache community -- Apache Gearpump
has its own feature branch, Apache Apex has a PR, and conversations are
starting on Apache Storm and others.
* New SDKs/DSLs -- the Python SDK from Google is in, and there are plans to
add the Scio DSL from Spotify.
* Support for new IO connectors -- Apache Kafka and JMS are in, with Amazon
Kinesis in PR.
And community-wise, we’ve:
* Started building a vibrant developer community, with detailed design
discussions on features like DoFn reuse semantics, serialization
technology, and an API for accessing state.
* Started building a user community with improvements to the website and
documentation.
* Had multiple talks on Beam at venues including ApacheCon, Hadoop Summit,
Kafka Summit, JBCN Barcelona, QCon, and Strata.
* Presented at multiple existing meetups and are starting to organize some
of our own.
This half birthday is also a good chance to take a step back and reflect on
our goals for this year -- TLP graduation and the first stable release.
Where are we on this path? What can we do better to accomplish these
high-level goals?
Frances
Re: [REFLECT] Beam’s Half Birthday!
Posted by Amit Sela <am...@gmail.com>.
Sounds great!
The only "numbers" I can think of that are missing are git stars/forks.
Thanks,
Amit
On Mon, Aug 1, 2016 at 6:36 PM Aljoscha Krettek <al...@apache.org> wrote:
> +1
>
> This sounds very good, I can't come up with anything that you missed.
>
> On Mon, 1 Aug 2016 at 08:00 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> > Happy half birthday ;)
> >
> > Very good idea Frances !!
> >
> > And the numbers are impressive indeed.
> >
> > Maybe, we can add kind of teasing about new incoming PRs like: Cassandra
> > IO (PR submitted), MongDB IO (PR submitted), MQTT IO, JDBC IO, Socket IO
> > (I'm working on these IOs), XML/JSON DSLs .
> >
> > Regards
> > JB
> >
> > On 08/01/2016 04:36 PM, Frances Perry wrote:
> > > Hi Beamers!
> > >
> > > It’s been six months today since Beam was accepted into incubation.
> It’s
> > > thrilling how far we’ve come since then!
> > >
> > > I’d like to volunteer to put together a post on the Beam blog
> summarizing
> > > our progress since February. Here’s a starting point... What am I
> missing
> > > that we should include? What makes you proud?
> > >
> > > By the numbers:
> > >
> > > * 48,238 lines of preexisting code donated by Cloudera, dataArtisans,
> and
> > > Google.
> > >
> > > * 761 pull requests from 45 contributors.
> > >
> > > * 498 Jira issues opened and 245 resolved.
> > >
> > > * 1 incubating release (and another 1 in progress).
> > >
> > > * 4200 hours of automated tests.
> > >
> > > * 161 subscribers / 606 messages on user@.
> > >
> > > * 217 subscribers / 1205 messages on dev@.
> > >
> > > There’s been a lot of technical progress, including:
> > >
> > > * Refactoring of the entire codebase, examples, and tests to be truly
> > > runner-independent.
> > >
> > > * New functionality in the Apache Flink runner for timestamps/windows
> in
> > > batch and bounded sources and side inputs in streaming mode.
> > >
> > > * Work in progress to upgrade the Apache Spark runner to use Spark 2.0.
> > >
> > > * Several new runners from the wider Apache community -- Apache
> Gearpump
> > > has its own feature branch, Apache Apex has a PR, and conversations are
> > > starting on Apache Storm and others.
> > >
> > > * New SDKs/DSLs -- the Python SDK from Google is in, and there are
> plans
> > to
> > > add the Scio DSL from Spotify.
> > >
> > > * Support for new IO connectors -- Apache Kafka and JMS are in, with
> > Amazon
> > > Kinesis in PR.
> > >
> > > And community-wise, we’ve:
> > >
> > > * Started building a vibrant developer community, with detailed design
> > > discussions on features like DoFn reuse semantics, serialization
> > > technology, and an API for accessing state.
> > >
> > > * Started building a user community with improvements to the website
> and
> > > documentation.
> > >
> > > * Had multiple talks on Beam at venues including ApacheCon, Hadoop
> > Summit,
> > > Kafka Summit, JBCN Barcelona, QCon, and Strata.
> > >
> > > * Presented at multiple existing meetups and are starting to organize
> > some
> > > of our own.
> > >
> > > This half birthday is also a good chance to take a step back and
> reflect
> > on
> > > our goals for this year -- TLP graduation and the first stable release.
> > > Where are we on this path? What can we do better to accomplish these
> > > high-level goals?
> > >
> > > Frances
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
Re: [REFLECT] Beam’s Half Birthday!
Posted by Frances Perry <fj...@google.com.INVALID>.
Thanks for the suggestions, folks! Incorporated them and posted the blog:
http://beam.incubator.apache.org/blog/2016/08/03/six-months.html
On Tue, Aug 2, 2016 at 12:42 AM, Ismaël Mejía <ie...@gmail.com> wrote:
> Hello,
>
> Nice reminder of the work done, I feel quite proud of what this community
> has
> accomplished in this short time (and of course of been a recent member of
> it).
>
> One missing statistic that it is probably hard to measure is how some Beam
> ideas
> have helped to improve other Apache projects. I know this is the case for
> Flink
> for example, but it is easy to imagine that this continues to happen as
> well
> with other Apache projects.
>
> Other statistic that surprised me is the number of members of the mailing
> lists,
> probably it is normal at this time of the project to have more users in the
> dev
> list than in the user one, and this clearly reflects a healthy dev
> community,
> but we have to continue with the good work, so we can have thriving user
> community too.
>
> Congratulations and Happy Half Birthday Beamers.
> Ismaël
>
>
> On Tue, Aug 2, 2016 at 2:20 AM, Ahmet Altay <al...@google.com.invalid>
> wrote:
>
> > Happy half-birthday!
> >
> > As one of the new comers of the Python SDK, it would be great to have it
> in
> > the main branch. We are getting closer to that goal everyday.
> >
> > Thanks,
> > Ahmet
> >
> > On Mon, Aug 1, 2016 at 10:02 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Fully agree with Dan.
> > >
> > > Regards
> > > JB
> > >
> > >
> > > On 08/01/2016 06:56 PM, Dan Halperin wrote:
> > >
> > >> +1 (binding? ;)
> > >>
> > >> On this part of the email:
> > >>
> > >> This half birthday is also a good chance to take a step back and
> reflect
> > >>>>
> > >>> on
> > >>>
> > >>>> our goals for this year -- TLP graduation and the first stable
> > release.
> > >>>> Where are we on this path? What can we do better to accomplish these
> > >>>> high-level goals?
> > >>>>
> > >>>
> > >> I think we really want to finish as many backwards-incompatible
> changes
> > as
> > >> possible. Here's a seed for that list.
> > >>
> > >>
> > >>
> > >> - DoFn setup/teardown
> > >> - new DoFn proposal
> > >> - Continuing to move google-specific IO from SDK into
> > >> google-cloud-platform IO module
> > >> - Any changes to fundamental style (PTransform.apply rename?
> Removing
> > >> the .Bound wrappers in various transforms?)
> > >>
> > >> I'd also really like to see Gearpump runner (maybe also Apex) and
> Python
> > >> SDK in the main branch.
> > >>
> > >> Thanks,
> > >> Dan
> > >>
> > >>
> > >> On Mon, Aug 1, 2016 at 8:36 AM, Aljoscha Krettek <aljoscha@apache.org
> >
> > >> wrote:
> > >>
> > >> +1
> > >>>
> > >>> This sounds very good, I can't come up with anything that you missed.
> > >>>
> > >>> On Mon, 1 Aug 2016 at 08:00 Jean-Baptiste Onofré <jb...@nanthrax.net>
> > >>> wrote:
> > >>>
> > >>> Happy half birthday ;)
> > >>>>
> > >>>> Very good idea Frances !!
> > >>>>
> > >>>> And the numbers are impressive indeed.
> > >>>>
> > >>>> Maybe, we can add kind of teasing about new incoming PRs like:
> > Cassandra
> > >>>> IO (PR submitted), MongDB IO (PR submitted), MQTT IO, JDBC IO,
> Socket
> > IO
> > >>>> (I'm working on these IOs), XML/JSON DSLs .
> > >>>>
> > >>>> Regards
> > >>>> JB
> > >>>>
> > >>>> On 08/01/2016 04:36 PM, Frances Perry wrote:
> > >>>>
> > >>>>> Hi Beamers!
> > >>>>>
> > >>>>> It’s been six months today since Beam was accepted into incubation.
> > >>>>>
> > >>>> It’s
> > >>>
> > >>>> thrilling how far we’ve come since then!
> > >>>>>
> > >>>>> I’d like to volunteer to put together a post on the Beam blog
> > >>>>>
> > >>>> summarizing
> > >>>
> > >>>> our progress since February. Here’s a starting point... What am I
> > >>>>>
> > >>>> missing
> > >>>
> > >>>> that we should include? What makes you proud?
> > >>>>>
> > >>>>> By the numbers:
> > >>>>>
> > >>>>> * 48,238 lines of preexisting code donated by Cloudera,
> dataArtisans,
> > >>>>>
> > >>>> and
> > >>>
> > >>>> Google.
> > >>>>>
> > >>>>> * 761 pull requests from 45 contributors.
> > >>>>>
> > >>>>> * 498 Jira issues opened and 245 resolved.
> > >>>>>
> > >>>>> * 1 incubating release (and another 1 in progress).
> > >>>>>
> > >>>>> * 4200 hours of automated tests.
> > >>>>>
> > >>>>> * 161 subscribers / 606 messages on user@.
> > >>>>>
> > >>>>> * 217 subscribers / 1205 messages on dev@.
> > >>>>>
> > >>>>> There’s been a lot of technical progress, including:
> > >>>>>
> > >>>>> * Refactoring of the entire codebase, examples, and tests to be
> truly
> > >>>>> runner-independent.
> > >>>>>
> > >>>>> * New functionality in the Apache Flink runner for
> timestamps/windows
> > >>>>>
> > >>>> in
> > >>>
> > >>>> batch and bounded sources and side inputs in streaming mode.
> > >>>>>
> > >>>>> * Work in progress to upgrade the Apache Spark runner to use Spark
> > 2.0.
> > >>>>>
> > >>>>> * Several new runners from the wider Apache community -- Apache
> > >>>>>
> > >>>> Gearpump
> > >>>
> > >>>> has its own feature branch, Apache Apex has a PR, and conversations
> > are
> > >>>>> starting on Apache Storm and others.
> > >>>>>
> > >>>>> * New SDKs/DSLs -- the Python SDK from Google is in, and there are
> > >>>>>
> > >>>> plans
> > >>>
> > >>>> to
> > >>>>
> > >>>>> add the Scio DSL from Spotify.
> > >>>>>
> > >>>>> * Support for new IO connectors -- Apache Kafka and JMS are in,
> with
> > >>>>>
> > >>>> Amazon
> > >>>>
> > >>>>> Kinesis in PR.
> > >>>>>
> > >>>>> And community-wise, we’ve:
> > >>>>>
> > >>>>> * Started building a vibrant developer community, with detailed
> > design
> > >>>>> discussions on features like DoFn reuse semantics, serialization
> > >>>>> technology, and an API for accessing state.
> > >>>>>
> > >>>>> * Started building a user community with improvements to the
> website
> > >>>>>
> > >>>> and
> > >>>
> > >>>> documentation.
> > >>>>>
> > >>>>> * Had multiple talks on Beam at venues including ApacheCon, Hadoop
> > >>>>>
> > >>>> Summit,
> > >>>>
> > >>>>> Kafka Summit, JBCN Barcelona, QCon, and Strata.
> > >>>>>
> > >>>>> * Presented at multiple existing meetups and are starting to
> organize
> > >>>>>
> > >>>> some
> > >>>>
> > >>>>> of our own.
> > >>>>>
> > >>>>> This half birthday is also a good chance to take a step back and
> > >>>>>
> > >>>> reflect
> > >>>
> > >>>> on
> > >>>>
> > >>>>> our goals for this year -- TLP graduation and the first stable
> > release.
> > >>>>> Where are we on this path? What can we do better to accomplish
> these
> > >>>>> high-level goals?
> > >>>>>
> > >>>>> Frances
> > >>>>>
> > >>>>>
> > >>>> --
> > >>>> Jean-Baptiste Onofré
> > >>>> jbonofre@apache.org
> > >>>> http://blog.nanthrax.net
> > >>>> Talend - http://www.talend.com
> > >>>>
> > >>>>
> > >>>
> > >>
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
>
Re: [REFLECT] Beam’s Half Birthday!
Posted by Ismaël Mejía <ie...@gmail.com>.
Hello,
Nice reminder of the work done, I feel quite proud of what this community
has
accomplished in this short time (and of course of been a recent member of
it).
One missing statistic that it is probably hard to measure is how some Beam
ideas
have helped to improve other Apache projects. I know this is the case for
Flink
for example, but it is easy to imagine that this continues to happen as well
with other Apache projects.
Other statistic that surprised me is the number of members of the mailing
lists,
probably it is normal at this time of the project to have more users in the
dev
list than in the user one, and this clearly reflects a healthy dev
community,
but we have to continue with the good work, so we can have thriving user
community too.
Congratulations and Happy Half Birthday Beamers.
Ismaël
On Tue, Aug 2, 2016 at 2:20 AM, Ahmet Altay <al...@google.com.invalid>
wrote:
> Happy half-birthday!
>
> As one of the new comers of the Python SDK, it would be great to have it in
> the main branch. We are getting closer to that goal everyday.
>
> Thanks,
> Ahmet
>
> On Mon, Aug 1, 2016 at 10:02 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Fully agree with Dan.
> >
> > Regards
> > JB
> >
> >
> > On 08/01/2016 06:56 PM, Dan Halperin wrote:
> >
> >> +1 (binding? ;)
> >>
> >> On this part of the email:
> >>
> >> This half birthday is also a good chance to take a step back and reflect
> >>>>
> >>> on
> >>>
> >>>> our goals for this year -- TLP graduation and the first stable
> release.
> >>>> Where are we on this path? What can we do better to accomplish these
> >>>> high-level goals?
> >>>>
> >>>
> >> I think we really want to finish as many backwards-incompatible changes
> as
> >> possible. Here's a seed for that list.
> >>
> >>
> >>
> >> - DoFn setup/teardown
> >> - new DoFn proposal
> >> - Continuing to move google-specific IO from SDK into
> >> google-cloud-platform IO module
> >> - Any changes to fundamental style (PTransform.apply rename? Removing
> >> the .Bound wrappers in various transforms?)
> >>
> >> I'd also really like to see Gearpump runner (maybe also Apex) and Python
> >> SDK in the main branch.
> >>
> >> Thanks,
> >> Dan
> >>
> >>
> >> On Mon, Aug 1, 2016 at 8:36 AM, Aljoscha Krettek <al...@apache.org>
> >> wrote:
> >>
> >> +1
> >>>
> >>> This sounds very good, I can't come up with anything that you missed.
> >>>
> >>> On Mon, 1 Aug 2016 at 08:00 Jean-Baptiste Onofré <jb...@nanthrax.net>
> >>> wrote:
> >>>
> >>> Happy half birthday ;)
> >>>>
> >>>> Very good idea Frances !!
> >>>>
> >>>> And the numbers are impressive indeed.
> >>>>
> >>>> Maybe, we can add kind of teasing about new incoming PRs like:
> Cassandra
> >>>> IO (PR submitted), MongDB IO (PR submitted), MQTT IO, JDBC IO, Socket
> IO
> >>>> (I'm working on these IOs), XML/JSON DSLs .
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On 08/01/2016 04:36 PM, Frances Perry wrote:
> >>>>
> >>>>> Hi Beamers!
> >>>>>
> >>>>> It’s been six months today since Beam was accepted into incubation.
> >>>>>
> >>>> It’s
> >>>
> >>>> thrilling how far we’ve come since then!
> >>>>>
> >>>>> I’d like to volunteer to put together a post on the Beam blog
> >>>>>
> >>>> summarizing
> >>>
> >>>> our progress since February. Here’s a starting point... What am I
> >>>>>
> >>>> missing
> >>>
> >>>> that we should include? What makes you proud?
> >>>>>
> >>>>> By the numbers:
> >>>>>
> >>>>> * 48,238 lines of preexisting code donated by Cloudera, dataArtisans,
> >>>>>
> >>>> and
> >>>
> >>>> Google.
> >>>>>
> >>>>> * 761 pull requests from 45 contributors.
> >>>>>
> >>>>> * 498 Jira issues opened and 245 resolved.
> >>>>>
> >>>>> * 1 incubating release (and another 1 in progress).
> >>>>>
> >>>>> * 4200 hours of automated tests.
> >>>>>
> >>>>> * 161 subscribers / 606 messages on user@.
> >>>>>
> >>>>> * 217 subscribers / 1205 messages on dev@.
> >>>>>
> >>>>> There’s been a lot of technical progress, including:
> >>>>>
> >>>>> * Refactoring of the entire codebase, examples, and tests to be truly
> >>>>> runner-independent.
> >>>>>
> >>>>> * New functionality in the Apache Flink runner for timestamps/windows
> >>>>>
> >>>> in
> >>>
> >>>> batch and bounded sources and side inputs in streaming mode.
> >>>>>
> >>>>> * Work in progress to upgrade the Apache Spark runner to use Spark
> 2.0.
> >>>>>
> >>>>> * Several new runners from the wider Apache community -- Apache
> >>>>>
> >>>> Gearpump
> >>>
> >>>> has its own feature branch, Apache Apex has a PR, and conversations
> are
> >>>>> starting on Apache Storm and others.
> >>>>>
> >>>>> * New SDKs/DSLs -- the Python SDK from Google is in, and there are
> >>>>>
> >>>> plans
> >>>
> >>>> to
> >>>>
> >>>>> add the Scio DSL from Spotify.
> >>>>>
> >>>>> * Support for new IO connectors -- Apache Kafka and JMS are in, with
> >>>>>
> >>>> Amazon
> >>>>
> >>>>> Kinesis in PR.
> >>>>>
> >>>>> And community-wise, we’ve:
> >>>>>
> >>>>> * Started building a vibrant developer community, with detailed
> design
> >>>>> discussions on features like DoFn reuse semantics, serialization
> >>>>> technology, and an API for accessing state.
> >>>>>
> >>>>> * Started building a user community with improvements to the website
> >>>>>
> >>>> and
> >>>
> >>>> documentation.
> >>>>>
> >>>>> * Had multiple talks on Beam at venues including ApacheCon, Hadoop
> >>>>>
> >>>> Summit,
> >>>>
> >>>>> Kafka Summit, JBCN Barcelona, QCon, and Strata.
> >>>>>
> >>>>> * Presented at multiple existing meetups and are starting to organize
> >>>>>
> >>>> some
> >>>>
> >>>>> of our own.
> >>>>>
> >>>>> This half birthday is also a good chance to take a step back and
> >>>>>
> >>>> reflect
> >>>
> >>>> on
> >>>>
> >>>>> our goals for this year -- TLP graduation and the first stable
> release.
> >>>>> Where are we on this path? What can we do better to accomplish these
> >>>>> high-level goals?
> >>>>>
> >>>>> Frances
> >>>>>
> >>>>>
> >>>> --
> >>>> Jean-Baptiste Onofré
> >>>> jbonofre@apache.org
> >>>> http://blog.nanthrax.net
> >>>> Talend - http://www.talend.com
> >>>>
> >>>>
> >>>
> >>
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
Re: [REFLECT] Beam’s Half Birthday!
Posted by Ahmet Altay <al...@google.com.INVALID>.
Happy half-birthday!
As one of the new comers of the Python SDK, it would be great to have it in
the main branch. We are getting closer to that goal everyday.
Thanks,
Ahmet
On Mon, Aug 1, 2016 at 10:02 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Fully agree with Dan.
>
> Regards
> JB
>
>
> On 08/01/2016 06:56 PM, Dan Halperin wrote:
>
>> +1 (binding? ;)
>>
>> On this part of the email:
>>
>> This half birthday is also a good chance to take a step back and reflect
>>>>
>>> on
>>>
>>>> our goals for this year -- TLP graduation and the first stable release.
>>>> Where are we on this path? What can we do better to accomplish these
>>>> high-level goals?
>>>>
>>>
>> I think we really want to finish as many backwards-incompatible changes as
>> possible. Here's a seed for that list.
>>
>>
>>
>> - DoFn setup/teardown
>> - new DoFn proposal
>> - Continuing to move google-specific IO from SDK into
>> google-cloud-platform IO module
>> - Any changes to fundamental style (PTransform.apply rename? Removing
>> the .Bound wrappers in various transforms?)
>>
>> I'd also really like to see Gearpump runner (maybe also Apex) and Python
>> SDK in the main branch.
>>
>> Thanks,
>> Dan
>>
>>
>> On Mon, Aug 1, 2016 at 8:36 AM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>> +1
>>>
>>> This sounds very good, I can't come up with anything that you missed.
>>>
>>> On Mon, 1 Aug 2016 at 08:00 Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>> Happy half birthday ;)
>>>>
>>>> Very good idea Frances !!
>>>>
>>>> And the numbers are impressive indeed.
>>>>
>>>> Maybe, we can add kind of teasing about new incoming PRs like: Cassandra
>>>> IO (PR submitted), MongDB IO (PR submitted), MQTT IO, JDBC IO, Socket IO
>>>> (I'm working on these IOs), XML/JSON DSLs .
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 08/01/2016 04:36 PM, Frances Perry wrote:
>>>>
>>>>> Hi Beamers!
>>>>>
>>>>> It’s been six months today since Beam was accepted into incubation.
>>>>>
>>>> It’s
>>>
>>>> thrilling how far we’ve come since then!
>>>>>
>>>>> I’d like to volunteer to put together a post on the Beam blog
>>>>>
>>>> summarizing
>>>
>>>> our progress since February. Here’s a starting point... What am I
>>>>>
>>>> missing
>>>
>>>> that we should include? What makes you proud?
>>>>>
>>>>> By the numbers:
>>>>>
>>>>> * 48,238 lines of preexisting code donated by Cloudera, dataArtisans,
>>>>>
>>>> and
>>>
>>>> Google.
>>>>>
>>>>> * 761 pull requests from 45 contributors.
>>>>>
>>>>> * 498 Jira issues opened and 245 resolved.
>>>>>
>>>>> * 1 incubating release (and another 1 in progress).
>>>>>
>>>>> * 4200 hours of automated tests.
>>>>>
>>>>> * 161 subscribers / 606 messages on user@.
>>>>>
>>>>> * 217 subscribers / 1205 messages on dev@.
>>>>>
>>>>> There’s been a lot of technical progress, including:
>>>>>
>>>>> * Refactoring of the entire codebase, examples, and tests to be truly
>>>>> runner-independent.
>>>>>
>>>>> * New functionality in the Apache Flink runner for timestamps/windows
>>>>>
>>>> in
>>>
>>>> batch and bounded sources and side inputs in streaming mode.
>>>>>
>>>>> * Work in progress to upgrade the Apache Spark runner to use Spark 2.0.
>>>>>
>>>>> * Several new runners from the wider Apache community -- Apache
>>>>>
>>>> Gearpump
>>>
>>>> has its own feature branch, Apache Apex has a PR, and conversations are
>>>>> starting on Apache Storm and others.
>>>>>
>>>>> * New SDKs/DSLs -- the Python SDK from Google is in, and there are
>>>>>
>>>> plans
>>>
>>>> to
>>>>
>>>>> add the Scio DSL from Spotify.
>>>>>
>>>>> * Support for new IO connectors -- Apache Kafka and JMS are in, with
>>>>>
>>>> Amazon
>>>>
>>>>> Kinesis in PR.
>>>>>
>>>>> And community-wise, we’ve:
>>>>>
>>>>> * Started building a vibrant developer community, with detailed design
>>>>> discussions on features like DoFn reuse semantics, serialization
>>>>> technology, and an API for accessing state.
>>>>>
>>>>> * Started building a user community with improvements to the website
>>>>>
>>>> and
>>>
>>>> documentation.
>>>>>
>>>>> * Had multiple talks on Beam at venues including ApacheCon, Hadoop
>>>>>
>>>> Summit,
>>>>
>>>>> Kafka Summit, JBCN Barcelona, QCon, and Strata.
>>>>>
>>>>> * Presented at multiple existing meetups and are starting to organize
>>>>>
>>>> some
>>>>
>>>>> of our own.
>>>>>
>>>>> This half birthday is also a good chance to take a step back and
>>>>>
>>>> reflect
>>>
>>>> on
>>>>
>>>>> our goals for this year -- TLP graduation and the first stable release.
>>>>> Where are we on this path? What can we do better to accomplish these
>>>>> high-level goals?
>>>>>
>>>>> Frances
>>>>>
>>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>>>
>>>>
>>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [REFLECT] Beam’s Half Birthday!
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Fully agree with Dan.
Regards
JB
On 08/01/2016 06:56 PM, Dan Halperin wrote:
> +1 (binding? ;)
>
> On this part of the email:
>
>>> This half birthday is also a good chance to take a step back and reflect
>> on
>>> our goals for this year -- TLP graduation and the first stable release.
>>> Where are we on this path? What can we do better to accomplish these
>>> high-level goals?
>
> I think we really want to finish as many backwards-incompatible changes as
> possible. Here's a seed for that list.
>
>
>
> - DoFn setup/teardown
> - new DoFn proposal
> - Continuing to move google-specific IO from SDK into
> google-cloud-platform IO module
> - Any changes to fundamental style (PTransform.apply rename? Removing
> the .Bound wrappers in various transforms?)
>
> I'd also really like to see Gearpump runner (maybe also Apex) and Python
> SDK in the main branch.
>
> Thanks,
> Dan
>
>
> On Mon, Aug 1, 2016 at 8:36 AM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> +1
>>
>> This sounds very good, I can't come up with anything that you missed.
>>
>> On Mon, 1 Aug 2016 at 08:00 Jean-Baptiste Onofr� <jb...@nanthrax.net> wrote:
>>
>>> Happy half birthday ;)
>>>
>>> Very good idea Frances !!
>>>
>>> And the numbers are impressive indeed.
>>>
>>> Maybe, we can add kind of teasing about new incoming PRs like: Cassandra
>>> IO (PR submitted), MongDB IO (PR submitted), MQTT IO, JDBC IO, Socket IO
>>> (I'm working on these IOs), XML/JSON DSLs .
>>>
>>> Regards
>>> JB
>>>
>>> On 08/01/2016 04:36 PM, Frances Perry wrote:
>>>> Hi Beamers!
>>>>
>>>> It\u2019s been six months today since Beam was accepted into incubation.
>> It\u2019s
>>>> thrilling how far we\u2019ve come since then!
>>>>
>>>> I\u2019d like to volunteer to put together a post on the Beam blog
>> summarizing
>>>> our progress since February. Here\u2019s a starting point... What am I
>> missing
>>>> that we should include? What makes you proud?
>>>>
>>>> By the numbers:
>>>>
>>>> * 48,238 lines of preexisting code donated by Cloudera, dataArtisans,
>> and
>>>> Google.
>>>>
>>>> * 761 pull requests from 45 contributors.
>>>>
>>>> * 498 Jira issues opened and 245 resolved.
>>>>
>>>> * 1 incubating release (and another 1 in progress).
>>>>
>>>> * 4200 hours of automated tests.
>>>>
>>>> * 161 subscribers / 606 messages on user@.
>>>>
>>>> * 217 subscribers / 1205 messages on dev@.
>>>>
>>>> There\u2019s been a lot of technical progress, including:
>>>>
>>>> * Refactoring of the entire codebase, examples, and tests to be truly
>>>> runner-independent.
>>>>
>>>> * New functionality in the Apache Flink runner for timestamps/windows
>> in
>>>> batch and bounded sources and side inputs in streaming mode.
>>>>
>>>> * Work in progress to upgrade the Apache Spark runner to use Spark 2.0.
>>>>
>>>> * Several new runners from the wider Apache community -- Apache
>> Gearpump
>>>> has its own feature branch, Apache Apex has a PR, and conversations are
>>>> starting on Apache Storm and others.
>>>>
>>>> * New SDKs/DSLs -- the Python SDK from Google is in, and there are
>> plans
>>> to
>>>> add the Scio DSL from Spotify.
>>>>
>>>> * Support for new IO connectors -- Apache Kafka and JMS are in, with
>>> Amazon
>>>> Kinesis in PR.
>>>>
>>>> And community-wise, we\u2019ve:
>>>>
>>>> * Started building a vibrant developer community, with detailed design
>>>> discussions on features like DoFn reuse semantics, serialization
>>>> technology, and an API for accessing state.
>>>>
>>>> * Started building a user community with improvements to the website
>> and
>>>> documentation.
>>>>
>>>> * Had multiple talks on Beam at venues including ApacheCon, Hadoop
>>> Summit,
>>>> Kafka Summit, JBCN Barcelona, QCon, and Strata.
>>>>
>>>> * Presented at multiple existing meetups and are starting to organize
>>> some
>>>> of our own.
>>>>
>>>> This half birthday is also a good chance to take a step back and
>> reflect
>>> on
>>>> our goals for this year -- TLP graduation and the first stable release.
>>>> Where are we on this path? What can we do better to accomplish these
>>>> high-level goals?
>>>>
>>>> Frances
>>>>
>>>
>>> --
>>> Jean-Baptiste Onofr�
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [REFLECT] Beam’s Half Birthday!
Posted by Dan Halperin <dh...@google.com.INVALID>.
+1 (binding? ;)
On this part of the email:
> > This half birthday is also a good chance to take a step back and reflect
> on
> > our goals for this year -- TLP graduation and the first stable release.
> > Where are we on this path? What can we do better to accomplish these
> > high-level goals?
I think we really want to finish as many backwards-incompatible changes as
possible. Here's a seed for that list.
- DoFn setup/teardown
- new DoFn proposal
- Continuing to move google-specific IO from SDK into
google-cloud-platform IO module
- Any changes to fundamental style (PTransform.apply rename? Removing
the .Bound wrappers in various transforms?)
I'd also really like to see Gearpump runner (maybe also Apex) and Python
SDK in the main branch.
Thanks,
Dan
On Mon, Aug 1, 2016 at 8:36 AM, Aljoscha Krettek <al...@apache.org>
wrote:
> +1
>
> This sounds very good, I can't come up with anything that you missed.
>
> On Mon, 1 Aug 2016 at 08:00 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> > Happy half birthday ;)
> >
> > Very good idea Frances !!
> >
> > And the numbers are impressive indeed.
> >
> > Maybe, we can add kind of teasing about new incoming PRs like: Cassandra
> > IO (PR submitted), MongDB IO (PR submitted), MQTT IO, JDBC IO, Socket IO
> > (I'm working on these IOs), XML/JSON DSLs .
> >
> > Regards
> > JB
> >
> > On 08/01/2016 04:36 PM, Frances Perry wrote:
> > > Hi Beamers!
> > >
> > > It’s been six months today since Beam was accepted into incubation.
> It’s
> > > thrilling how far we’ve come since then!
> > >
> > > I’d like to volunteer to put together a post on the Beam blog
> summarizing
> > > our progress since February. Here’s a starting point... What am I
> missing
> > > that we should include? What makes you proud?
> > >
> > > By the numbers:
> > >
> > > * 48,238 lines of preexisting code donated by Cloudera, dataArtisans,
> and
> > > Google.
> > >
> > > * 761 pull requests from 45 contributors.
> > >
> > > * 498 Jira issues opened and 245 resolved.
> > >
> > > * 1 incubating release (and another 1 in progress).
> > >
> > > * 4200 hours of automated tests.
> > >
> > > * 161 subscribers / 606 messages on user@.
> > >
> > > * 217 subscribers / 1205 messages on dev@.
> > >
> > > There’s been a lot of technical progress, including:
> > >
> > > * Refactoring of the entire codebase, examples, and tests to be truly
> > > runner-independent.
> > >
> > > * New functionality in the Apache Flink runner for timestamps/windows
> in
> > > batch and bounded sources and side inputs in streaming mode.
> > >
> > > * Work in progress to upgrade the Apache Spark runner to use Spark 2.0.
> > >
> > > * Several new runners from the wider Apache community -- Apache
> Gearpump
> > > has its own feature branch, Apache Apex has a PR, and conversations are
> > > starting on Apache Storm and others.
> > >
> > > * New SDKs/DSLs -- the Python SDK from Google is in, and there are
> plans
> > to
> > > add the Scio DSL from Spotify.
> > >
> > > * Support for new IO connectors -- Apache Kafka and JMS are in, with
> > Amazon
> > > Kinesis in PR.
> > >
> > > And community-wise, we’ve:
> > >
> > > * Started building a vibrant developer community, with detailed design
> > > discussions on features like DoFn reuse semantics, serialization
> > > technology, and an API for accessing state.
> > >
> > > * Started building a user community with improvements to the website
> and
> > > documentation.
> > >
> > > * Had multiple talks on Beam at venues including ApacheCon, Hadoop
> > Summit,
> > > Kafka Summit, JBCN Barcelona, QCon, and Strata.
> > >
> > > * Presented at multiple existing meetups and are starting to organize
> > some
> > > of our own.
> > >
> > > This half birthday is also a good chance to take a step back and
> reflect
> > on
> > > our goals for this year -- TLP graduation and the first stable release.
> > > Where are we on this path? What can we do better to accomplish these
> > > high-level goals?
> > >
> > > Frances
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
Re: [REFLECT] Beam’s Half Birthday!
Posted by Aljoscha Krettek <al...@apache.org>.
+1
This sounds very good, I can't come up with anything that you missed.
On Mon, 1 Aug 2016 at 08:00 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Happy half birthday ;)
>
> Very good idea Frances !!
>
> And the numbers are impressive indeed.
>
> Maybe, we can add kind of teasing about new incoming PRs like: Cassandra
> IO (PR submitted), MongDB IO (PR submitted), MQTT IO, JDBC IO, Socket IO
> (I'm working on these IOs), XML/JSON DSLs .
>
> Regards
> JB
>
> On 08/01/2016 04:36 PM, Frances Perry wrote:
> > Hi Beamers!
> >
> > It’s been six months today since Beam was accepted into incubation. It’s
> > thrilling how far we’ve come since then!
> >
> > I’d like to volunteer to put together a post on the Beam blog summarizing
> > our progress since February. Here’s a starting point... What am I missing
> > that we should include? What makes you proud?
> >
> > By the numbers:
> >
> > * 48,238 lines of preexisting code donated by Cloudera, dataArtisans, and
> > Google.
> >
> > * 761 pull requests from 45 contributors.
> >
> > * 498 Jira issues opened and 245 resolved.
> >
> > * 1 incubating release (and another 1 in progress).
> >
> > * 4200 hours of automated tests.
> >
> > * 161 subscribers / 606 messages on user@.
> >
> > * 217 subscribers / 1205 messages on dev@.
> >
> > There’s been a lot of technical progress, including:
> >
> > * Refactoring of the entire codebase, examples, and tests to be truly
> > runner-independent.
> >
> > * New functionality in the Apache Flink runner for timestamps/windows in
> > batch and bounded sources and side inputs in streaming mode.
> >
> > * Work in progress to upgrade the Apache Spark runner to use Spark 2.0.
> >
> > * Several new runners from the wider Apache community -- Apache Gearpump
> > has its own feature branch, Apache Apex has a PR, and conversations are
> > starting on Apache Storm and others.
> >
> > * New SDKs/DSLs -- the Python SDK from Google is in, and there are plans
> to
> > add the Scio DSL from Spotify.
> >
> > * Support for new IO connectors -- Apache Kafka and JMS are in, with
> Amazon
> > Kinesis in PR.
> >
> > And community-wise, we’ve:
> >
> > * Started building a vibrant developer community, with detailed design
> > discussions on features like DoFn reuse semantics, serialization
> > technology, and an API for accessing state.
> >
> > * Started building a user community with improvements to the website and
> > documentation.
> >
> > * Had multiple talks on Beam at venues including ApacheCon, Hadoop
> Summit,
> > Kafka Summit, JBCN Barcelona, QCon, and Strata.
> >
> > * Presented at multiple existing meetups and are starting to organize
> some
> > of our own.
> >
> > This half birthday is also a good chance to take a step back and reflect
> on
> > our goals for this year -- TLP graduation and the first stable release.
> > Where are we on this path? What can we do better to accomplish these
> > high-level goals?
> >
> > Frances
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [REFLECT] Beam’s Half Birthday!
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Happy half birthday ;)
Very good idea Frances !!
And the numbers are impressive indeed.
Maybe, we can add kind of teasing about new incoming PRs like: Cassandra
IO (PR submitted), MongDB IO (PR submitted), MQTT IO, JDBC IO, Socket IO
(I'm working on these IOs), XML/JSON DSLs .
Regards
JB
On 08/01/2016 04:36 PM, Frances Perry wrote:
> Hi Beamers!
>
> It\u2019s been six months today since Beam was accepted into incubation. It\u2019s
> thrilling how far we\u2019ve come since then!
>
> I\u2019d like to volunteer to put together a post on the Beam blog summarizing
> our progress since February. Here\u2019s a starting point... What am I missing
> that we should include? What makes you proud?
>
> By the numbers:
>
> * 48,238 lines of preexisting code donated by Cloudera, dataArtisans, and
> Google.
>
> * 761 pull requests from 45 contributors.
>
> * 498 Jira issues opened and 245 resolved.
>
> * 1 incubating release (and another 1 in progress).
>
> * 4200 hours of automated tests.
>
> * 161 subscribers / 606 messages on user@.
>
> * 217 subscribers / 1205 messages on dev@.
>
> There\u2019s been a lot of technical progress, including:
>
> * Refactoring of the entire codebase, examples, and tests to be truly
> runner-independent.
>
> * New functionality in the Apache Flink runner for timestamps/windows in
> batch and bounded sources and side inputs in streaming mode.
>
> * Work in progress to upgrade the Apache Spark runner to use Spark 2.0.
>
> * Several new runners from the wider Apache community -- Apache Gearpump
> has its own feature branch, Apache Apex has a PR, and conversations are
> starting on Apache Storm and others.
>
> * New SDKs/DSLs -- the Python SDK from Google is in, and there are plans to
> add the Scio DSL from Spotify.
>
> * Support for new IO connectors -- Apache Kafka and JMS are in, with Amazon
> Kinesis in PR.
>
> And community-wise, we\u2019ve:
>
> * Started building a vibrant developer community, with detailed design
> discussions on features like DoFn reuse semantics, serialization
> technology, and an API for accessing state.
>
> * Started building a user community with improvements to the website and
> documentation.
>
> * Had multiple talks on Beam at venues including ApacheCon, Hadoop Summit,
> Kafka Summit, JBCN Barcelona, QCon, and Strata.
>
> * Presented at multiple existing meetups and are starting to organize some
> of our own.
>
> This half birthday is also a good chance to take a step back and reflect on
> our goals for this year -- TLP graduation and the first stable release.
> Where are we on this path? What can we do better to accomplish these
> high-level goals?
>
> Frances
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com