You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by David Cavazos <dc...@google.com> on 2018/08/08 16:38:27 UTC
[VOTE] Community Examples Repository
Hi everyone!
We discussed several options as well as some of the implications of each
option. Please vote for your favorite option, feel free to back it up with
any reasons that make you feel that way.
1) Move *all* samples to a *new *examples* repository*
2) Move *some* samples to a *new *examples* repository*
3) Leave samples where they are
Some implications to creating a new repository:
- Every example would be independent from every other example, so tests can
be run in parallel
- Examples would now show how to use Beam *externally*
- The examples repository would need a testing infrastructure
- Decoupling makes examples easier to test on different versions
- Easier to copy-paste an existing example and start from there, almost
like a template
- Smaller size for the core Beam library
- Two different repositories to maintain
- Versioning could mirror Beam's current version
Link to proposal
<https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
Re: [VOTE] Community Examples Repository
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
I would go for 1 or 3 to be consistent.
Regards
JB
On 08/08/2018 18:38, David Cavazos wrote:
> Hi everyone!
>
> We discussed several options as well as some of the implications of each
> option. Please vote for your favorite option, feel free to back it up
> with any reasons that make you feel that way.
>
> 1) Move *all* samples to a *new *examples*repository*
> 2) Move *some* samples to a *new *examples*repository*
> 3) Leave samples where they are
>
> Some implications to creating a new repository:
> - Every example would be independent from every other example, so tests
> can be run in parallel
> - Examples would now show how to use Beam /externally/
> - The examples repository would need a testing infrastructure
> - Decoupling makes examples easier to test on different versions
> - Easier to copy-paste an existing example and start from there, almost
> like a template
> - Smaller size for the core Beam library
> - Two different repositories to maintain
> - Versioning could mirror Beam's current version
>
> Link to proposal
> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Community Examples Repository
Posted by Andrea Foegler <fo...@google.com>.
I guess I'm voting for 2. Tests obviously belong in the code repo, both as
a sample usage (not "how-to", more like "man") and, well, for testing.
These pipelines might not be annotated as completely and include hitting
edge cases and other non-standard situations.
Any example where the primary purpose is external consumption seems totally
reasonable to move out.
Regardless of where they are stored, trying to share these artifacts
explicitly seems less than ideal. The cost of duplication here seems low.
On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ru...@google.com> wrote:
> 2 - examples that rely on experimental API can still stay in where they
> are because such examples could be changed.
>
> -Rui
>
> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <cc...@google.com> wrote:
>
>> 3 - We benefit from increased test coverage by having examples together
>> with the rest of the code. As Robert mentions in the doc, hosting the Beam
>> examples in the main repository is the best way to keep the examples
>> visible, tested and maintained. Given that we recently moved to a single
>> repository for the website since that previously caused a lot of pain, it
>> makes sense to be consistent here.
>>
>> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay <al...@google.com> wrote:
>>
>>> 2 - Similar to Huygaa, I see value in keeping a core set of examples
>>> tested and maintained against head. At the same time I understand the value
>>> of a growing set of community grown examples that are targeted against a
>>> pre-defined versions of Beam and not necessarily updated at every release.
>>>
>>> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan <ba...@google.com>
>>> wrote:
>>>
>>>> 2 - I like the idea of having a separate repo where we can have more
>>>> freedom to check in examples. However, we benefit from having immediate
>>>> core examples in Beam for testing purposes.
>>>>
>>>> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos <dc...@google.com>
>>>> wrote:
>>>>
>>>>> Hi everyone!
>>>>>
>>>>> We discussed several options as well as some of the implications of
>>>>> each option. Please vote for your favorite option, feel free to back it up
>>>>> with any reasons that make you feel that way.
>>>>>
>>>>> 1) Move *all* samples to a *new *examples* repository*
>>>>> 2) Move *some* samples to a *new *examples* repository*
>>>>> 3) Leave samples where they are
>>>>>
>>>>> Some implications to creating a new repository:
>>>>> - Every example would be independent from every other example, so
>>>>> tests can be run in parallel
>>>>> - Examples would now show how to use Beam *externally*
>>>>> - The examples repository would need a testing infrastructure
>>>>> - Decoupling makes examples easier to test on different versions
>>>>> - Easier to copy-paste an existing example and start from there,
>>>>> almost like a template
>>>>> - Smaller size for the core Beam library
>>>>> - Two different repositories to maintain
>>>>> - Versioning could mirror Beam's current version
>>>>>
>>>>> Link to proposal
>>>>> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
>>>>>
>>>>
>>>
Re: [VOTE] Community Examples Repository
Posted by Ahmet Altay <al...@google.com>.
Charles, I agree with your comments and questions. I want to add one more
benefit that was mentioned earlier:
A place for people to contribute example that is not tied to the beam
release cycle. Such a repository could be a place for casual contributors
to add examples over time.
On Wed, Aug 8, 2018 at 2:25 PM, Charles Chen <cc...@google.com> wrote:
> It looks like the main claim is that 1 and 2 have the benefit of
> increasing visibility for examples on the Beam site. I agree with Robert's
> comments on the doc which claim that this is orthogonal to whether a
> separate repository is created (the comments are unresolved:
> https://docs.google.com/a/google.com/document/d/
> 1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY).
>
> I would add that the maintenance and testing burden has not been
> adequately addressed in the proposal (i.e. are we creating new Jenkins
> jobs?; will postcommits on the main Beam repo run examples tests?; are we
> releasing artifacts--if so, is this together with the main package or
> separately in new packages?). If we go with the half-way solution in (2),
> there is also the issue of where the threshold is--for example, if a
> user-contributed example is particularly useful, do we move it to the main
> repo?
>
> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <gr...@google.com> wrote:
>
>> I'd vote for 2.
>>
>> Giving independence to an example repository and creating the right
>> infrastructure to maintain them will give visibility to the efforts our
>> users are creating to solve their uses cases with Beam. I also want to make
>> the process of sharing common work more easily.
>>
>> Re:The examples that will remain in core, I agree that it's crucial to
>> keep some examples for testing.
>>
>>
>> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lc...@google.com> wrote:
>>
>>> I would vote for 3.
>>>
>>> My reasoning is that Java has a good mechanism to get a starter/example
>>> project going by using the the maven archetypes already. Our quickstart
>>> guide for Apache Beam for the Java SDK already covers generating the
>>> examples archetype.
>>> We could point users to the starter project at the end of the java
>>> quickstart.
>>>
>>> If python/go have a similar mechanism that is commonly used, I would go
>>> with those over creating a separate repo for examples and adding the
>>> maintenance burden involved.
>>>
>>>
>>>
>>> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ru...@google.com> wrote:
>>>
>>>> 2 - examples that rely on experimental API can still stay in where they
>>>> are because such examples could be changed.
>>>>
>>>> -Rui
>>>>
>>>> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <cc...@google.com> wrote:
>>>>
>>>>> 3 - We benefit from increased test coverage by having examples
>>>>> together with the rest of the code. As Robert mentions in the doc, hosting
>>>>> the Beam examples in the main repository is the best way to keep the
>>>>> examples visible, tested and maintained. Given that we recently moved to a
>>>>> single repository for the website since that previously caused a lot of
>>>>> pain, it makes sense to be consistent here.
>>>>>
>>>>> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay <al...@google.com> wrote:
>>>>>
>>>>>> 2 - Similar to Huygaa, I see value in keeping a core set of examples
>>>>>> tested and maintained against head. At the same time I understand the value
>>>>>> of a growing set of community grown examples that are targeted against a
>>>>>> pre-defined versions of Beam and not necessarily updated at every release.
>>>>>>
>>>>>> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan <batbat@google.com
>>>>>> > wrote:
>>>>>>
>>>>>>> 2 - I like the idea of having a separate repo where we can have more
>>>>>>> freedom to check in examples. However, we benefit from having immediate
>>>>>>> core examples in Beam for testing purposes.
>>>>>>>
>>>>>>> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos <dc...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi everyone!
>>>>>>>>
>>>>>>>> We discussed several options as well as some of the implications of
>>>>>>>> each option. Please vote for your favorite option, feel free to back it up
>>>>>>>> with any reasons that make you feel that way.
>>>>>>>>
>>>>>>>> 1) Move *all* samples to a *new *examples* repository*
>>>>>>>> 2) Move *some* samples to a *new *examples* repository*
>>>>>>>> 3) Leave samples where they are
>>>>>>>>
>>>>>>>> Some implications to creating a new repository:
>>>>>>>> - Every example would be independent from every other example, so
>>>>>>>> tests can be run in parallel
>>>>>>>> - Examples would now show how to use Beam *externally*
>>>>>>>> - The examples repository would need a testing infrastructure
>>>>>>>> - Decoupling makes examples easier to test on different versions
>>>>>>>> - Easier to copy-paste an existing example and start from there,
>>>>>>>> almost like a template
>>>>>>>> - Smaller size for the core Beam library
>>>>>>>> - Two different repositories to maintain
>>>>>>>> - Versioning could mirror Beam's current version
>>>>>>>>
>>>>>>>> Link to proposal
>>>>>>>> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
>>>>>>>>
>>>>>>>
>>>>>>
Re: [VOTE] Community Examples Repository
Posted by Yifan Zou <yi...@google.com>.
3 for all reasons above.
Keeping the examples in the Beam repository is a straightforward way to let
the examples being visible and maintainable. And as Lukasz Cwik mentioned,
the Quickstart makes a clear and easy approach for users to generate the
archetype and run examples.
I did not see enough benefits to make tons of efforts splitting an example
repository. It make sense to keep consistent.
- Yifan
On Mon, Aug 13, 2018 at 10:21 AM Andrew Pilloud <ap...@google.com> wrote:
> +1 for 2, it would be nice to have a example project in it's own GitHub
> repo. Might not need to be an "official" repo thought. Could we provide
> links to community supplied examples?
>
> On Thu, Aug 9, 2018, 2:30 PM Robert Bradshaw <ro...@google.com> wrote:
>
>> (3)
>>
>> In particular, I see a lot of value for (quoting the proposal)
>>
>> """
>> Since then, there have been
>> numerous updates, increased Python parity, and new features that do
>> not have accompanying examples employing best practices and
>> demonstrating an end-to-end experience for new users. We would like to
>> leverage the existing examples by raising their visibility and
>> auditing them.
>> """
>>
>> and I think the situation would become *worse* on all these fronts
>> with a separate repo (as well as the other issues mentioned,
>> especially complexity). We should consider lowering the bar to liking
>> to user-maintained examples that don't merit being in the main repo,
>> as well as guidelines for adding examples in the main repo itself.
>> On Thu, Aug 9, 2018 at 1:44 PM Ismaël Mejía <ie...@gmail.com> wrote:
>> >
>> > 3 for all the reasons discussed above. I think there are better ways to
>> improve the status quo without the extra maintenance of having a new repo
>> for this.
>> >
>> > On Thu, Aug 9, 2018 at 7:00 PM Ahmet Altay <al...@google.com> wrote:
>> >>
>> >> If we go forward with (3), could we actually update our documentation
>> on how we will support casual example contributions? I think we will need
>> to have information on how to add links to the new examples people want to
>> add to the set, what examples would be good additions to the Beam repo and
>> what examples would be better maintained somewhere else by their owners,
>> and what could they expect from our community when they work on such
>> examples.
>> >>
>> >> On Thu, Aug 9, 2018 at 9:41 AM, Mikhail Gryzykhin <mi...@google.com>
>> wrote:
>> >>>
>> >>> 3 (if contributors are up for voting) - We want to have beam
>> maintained examples in main repo. This will give good man to users and
>> allow us to test those easily with minimal maintenance.
>> >>>
>> >>> We can add links to opensource user repositories to our
>> documentation/wiki. This will be flexible enough to provide external
>> examples on one hand, and avoid responsibility of maintaining user code on
>> the other hand.
>> >>>
>> >>> --Mikhail
>> >>>
>> >>> Have feedback?
>> >>>
>> >>>
>> >>> On Thu, Aug 9, 2018 at 8:57 AM Rafael Fernandez <rf...@google.com>
>> wrote:
>> >>>>
>> >>>> Here is the Rose', David's, and Gris' proposal in text form, I hope
>> >>>> the copy/paste helps:
>> >>>>
>> >>>>
>> >>>> Apache Beam Examples Repository
>> >>>>
>> >>>> Authors: Rose Nguyen (rtnguyen@google.com), David Cavazos
>> >>>> (dcavazos@google.com), Gris Cuevas (gris@apache.org)
>> >>>>
>> >>>> Status: Proposal
>> >>>> Created: 2018-07-30
>> >>>> Updated: 2018-07-30
>> >>>>
>> >>>> Summary
>> >>>>
>> >>>> The Apache Beam Community creates and contributes examples to the
>> core
>> >>>> Apache Beam Github repository. We want to make the process easier and
>> >>>> less dependent in the core repository by creating a separate repo,
>> >>>> dedicated solely to Community examples, contribution guidelines and
>> >>>> add the examples to the website.
>> >>>>
>> >>>> Background
>> >>>>
>> >>>> The original batch of examples on the Apache Beam GitHub repository
>> >>>> was donated by Cloud Dataflow at the time of Java SDK 1.x to
>> >>>> demonstrate the capability of this programming model. These initial
>> >>>> examples were intended to demonstrate how a user can put together
>> >>>> their code components and try out Beam. Since then, there have been
>> >>>> numerous updates, increased Python parity, and new features that do
>> >>>> not have accompanying examples employing best practices and
>> >>>> demonstrating an end-to-end experience for new users. We would like
>> to
>> >>>> leverage the existing examples by raising their visibility and
>> >>>> auditing them. This is also an opportunity to establish
>> >>>> contribution/maintenance guidelines for community contributions and
>> to
>> >>>> start hosting the examples on the Beam site in an official
>> repository.
>> >>>> Attracting and retaining new users necessitates updated, concrete
>> >>>> examples that exhibit the range of capabilities of Beam.
>> >>>>
>> >>>> Proposed Tasks
>> >>>>
>> >>>> We would like to create a new GitHub Repository under the Apache
>> >>>> Software Foundation Org page for Apache Beam Community Examples. This
>> >>>> repo would be similar to apache/beam-site. The name we’d like to have
>> >>>> is apache/beam-examples. We will also move all current examples to
>> >>>> this repo, perform an audit to outline best practices and guidelines
>> >>>> and then publish them in the Apache Beam website.
>> >>>>
>> >>>>
>> >>>> Here is an outlined list of tasks we propose:
>> >>>>
>> >>>> Send Apache Beam Example Repository proposal to the mailing list
>> >>>> (David) - July 31
>> >>>>
>> >>>> Create the GitHub Repo (PMC would need to do this) - Request help
>> >>>> after proposal is refined/accepted
>> >>>>
>> >>>> Move current examples to new repo (David) -- 2 weeks after item 2 is
>> completed
>> >>>>
>> >>>> Add a note to let people we need to audit for best practices
>> >>>>
>> >>>> Audit current examples and define best practices (David, Rose, Gris)
>> >>>> -- Target date: week of 8/20
>> >>>>
>> >>>> Write guidelines on adding new examples and maintaining them (Gris,
>> >>>> Rose) -- Week after audit is completed
>> >>>>
>> >>>> Add examples to website (Rose) -- 1 week after guidelines are written
>> >>>>
>> >>>> Publish guidelines in website (Rose) -- 1 week after guidelines are
>> written
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Thu, Aug 9, 2018 at 6:22 AM Łukasz Gajowy <
>> lukasz.gajowy@gmail.com> wrote:
>> >>>> >
>> >>>> > I'd also vote for 3: I don't see much added value in separating
>> the repos and I see much additional effort to be done in maintaining extra
>> repo(s) (updating examples when new version of beam sdk comes out) and
>> their infrastructure (jenkins, etc). What Lukasz Cwik said about mvn
>> archetypes and how easy the examples can be to get starter examples from a
>> common repo only strengthens my opinion.
>> >>>> >
>> >>>> > Regarding 2: I think it's not good to have some official examples
>> here and some there - IMO it can make a false impression (user experience)
>> that some examples are less important than the others. Maybe a good idea is
>> to encourage users to share their (independent, non official) examples and
>> create a list of such on the beam site instead of 2?
>> >>>> >
>> >>>> > Łukasz
>> >>>> >
>> >>>> > czw., 9 sie 2018 o 11:35 Alexey Romanenko <
>> aromanenko.dev@gmail.com> napisał(a):
>> >>>> >>
>> >>>> >> 3 - I agree with JB, Charles and Lukasz arguments above saying
>> why we need to have examples and main code in the same repository (+
>> website code base will move there soon). I don’t see any huge benefits to
>> have examples aside and, at the same time, it will bring additional
>> complexity and burden for project support.
>> >>>> >>
>> >>>> >> On 9 Aug 2018, at 08:18, Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>> >>>> >>
>> >>>> >> Hi guys,
>> >>>> >>
>> >>>> >> For this kind of discussion, I would prefer to avoid Google Doc
>> and
>> >>>> >> directly put the point/proposal on the mailing list.
>> >>>> >>
>> >>>> >> It's easier for the community to follow.
>> >>>> >>
>> >>>> >> The statement is more for 3 because it's more convenient for
>> users to
>> >>>> >> easily find the examples and include in the distribution.
>> >>>> >>
>> >>>> >> Regards
>> >>>> >> JB
>> >>>> >>
>> >>>> >> On 08/08/2018 23:25, Charles Chen wrote:
>> >>>> >>
>> >>>> >> It looks like the main claim is that 1 and 2 have the benefit of
>> >>>> >> increasing visibility for examples on the Beam site. I agree with
>> >>>> >> Robert's comments on the doc which claim that this is orthogonal
>> to
>> >>>> >> whether a separate repository is created (the comments are
>> unresolved:
>> >>>> >>
>> https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY
>> ).
>> >>>> >>
>> >>>> >> I would add that the maintenance and testing burden has not been
>> >>>> >> adequately addressed in the proposal (i.e. are we creating new
>> Jenkins
>> >>>> >> jobs?; will postcommits on the main Beam repo run examples
>> tests?; are
>> >>>> >> we releasing artifacts--if so, is this together with the main
>> package or
>> >>>> >> separately in new packages?). If we go with the half-way
>> solution in
>> >>>> >> (2), there is also the issue of where the threshold is--for
>> example, if
>> >>>> >> a user-contributed example is particularly useful, do we move it
>> to the
>> >>>> >> main repo?
>> >>>> >>
>> >>>> >> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <gris@google.com
>> >>>> >> <ma...@google.com>> wrote:
>> >>>> >>
>> >>>> >> I'd vote for 2.
>> >>>> >>
>> >>>> >> Giving independence to an example repository and creating the
>> right
>> >>>> >> infrastructure to maintain them will give visibility to the
>> efforts
>> >>>> >> our users are creating to solve their uses cases with Beam. I
>> also
>> >>>> >> want to make the process of sharing common work more easily.
>> >>>> >>
>> >>>> >> Re:The examples that will remain in core, I agree that it's
>> crucial
>> >>>> >> to keep some examples for testing.
>> >>>> >>
>> >>>> >>
>> >>>> >> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lcwik@google.com
>> >>>> >> <ma...@google.com>> wrote:
>> >>>> >>
>> >>>> >> I would vote for 3.
>> >>>> >>
>> >>>> >> My reasoning is that Java has a good mechanism to get a
>> >>>> >> starter/example project going by using the the maven
>> archetypes
>> >>>> >> already. Our quickstart guide for Apache Beam for the Java
>> SDK
>> >>>> >> already covers generating the examples archetype.
>> >>>> >> We could point users to the starter project at the end of
>> the
>> >>>> >> java quickstart.
>> >>>> >>
>> >>>> >> If python/go have a similar mechanism that is commonly
>> used, I
>> >>>> >> would go with those over creating a separate repo for
>> examples
>> >>>> >> and adding the maintenance burden involved.
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <
>> ruwang@google.com
>> >>>> >> <ma...@google.com>> wrote:
>> >>>> >>
>> >>>> >> 2 - examples that rely on experimental API can still
>> stay in
>> >>>> >> where they are because such examples could be changed.
>> >>>> >>
>> >>>> >> -Rui
>> >>>> >>
>> >>>> >> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <
>> ccy@google.com
>> >>>> >> <ma...@google.com>> wrote:
>> >>>> >>
>> >>>> >> 3 - We benefit from increased test coverage by
>> having
>> >>>> >> examples together with the rest of the code. As
>> Robert
>> >>>> >> mentions in the doc, hosting the Beam examples in
>> the
>> >>>> >> main repository is the best way to keep the
>> examples
>> >>>> >> visible, tested and maintained. Given that we
>> recently
>> >>>> >> moved to a single repository for the website since
>> that
>> >>>> >> previously caused a lot of pain, it makes sense to
>> be
>> >>>> >> consistent here.
>> >>>> >>
>> >>>> >> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay
>> >>>> >> <altay@google.com <ma...@google.com>>
>> wrote:
>> >>>> >>
>> >>>> >> 2 - Similar to Huygaa, I see value in keeping
>> a core
>> >>>> >> set of examples tested and maintained against
>> head.
>> >>>> >> At the same time I understand the value of a
>> growing
>> >>>> >> set of community grown examples that are
>> targeted
>> >>>> >> against a pre-defined versions of Beam and not
>> >>>> >> necessarily updated at every release.
>> >>>> >>
>> >>>> >> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa
>> Batsaikhan
>> >>>> >> <batbat@google.com <ma...@google.com>>
>> wrote:
>> >>>> >>
>> >>>> >> 2 - I like the idea of having a separate
>> repo
>> >>>> >> where we can have more freedom to check in
>> >>>> >> examples. However, we benefit from having
>> >>>> >> immediate core examples in Beam for testing
>> >>>> >> purposes.
>> >>>> >>
>> >>>> >> On Wed, Aug 8, 2018 at 9:38 AM David
>> Cavazos
>> >>>> >> <dcavazos@google.com
>> >>>> >> <ma...@google.com>> wrote:
>> >>>> >>
>> >>>> >> Hi everyone!
>> >>>> >>
>> >>>> >> We discussed several options as well
>> as some
>> >>>> >> of the implications of each option.
>> Please
>> >>>> >> vote for your favorite option, feel
>> free to
>> >>>> >> back it up with any reasons that make
>> you
>> >>>> >> feel that way.
>> >>>> >>
>> >>>> >> 1) Move *all* samples to a *new
>> >>>> >> *examples*repository*
>> >>>> >> 2) Move *some* samples to a *new
>> >>>> >> *examples*repository*
>> >>>> >> 3) Leave samples where they are
>> >>>> >>
>> >>>> >> Some implications to creating a new
>> repository:
>> >>>> >> - Every example would be independent
>> from
>> >>>> >> every other example, so tests can be
>> run in
>> >>>> >> parallel
>> >>>> >> - Examples would now show how to use
>> Beam
>> >>>> >> /externally/
>> >>>> >> - The examples repository would need a
>> >>>> >> testing infrastructure
>> >>>> >> - Decoupling makes examples easier to
>> test
>> >>>> >> on different versions
>> >>>> >> - Easier to copy-paste an existing
>> example
>> >>>> >> and start from there, almost like a
>> template
>> >>>> >> - Smaller size for the core Beam
>> library
>> >>>> >> - Two different repositories to
>> maintain
>> >>>> >> - Versioning could mirror Beam's
>> current version
>> >>>> >>
>> >>>> >> Link to proposal
>> >>>> >> <
>> https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing
>> >
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> --
>> >>>> >> Jean-Baptiste Onofré
>> >>>> >> jbonofre@apache.org
>> >>>> >> http://blog.nanthrax.net
>> >>>> >> Talend - http://www.talend.com
>> >>>> >>
>> >>>> >>
>> >>
>> >>
>>
>
Re: [VOTE] Community Examples Repository
Posted by Andrew Pilloud <ap...@google.com>.
+1 for 2, it would be nice to have a example project in it's own GitHub
repo. Might not need to be an "official" repo thought. Could we provide
links to community supplied examples?
On Thu, Aug 9, 2018, 2:30 PM Robert Bradshaw <ro...@google.com> wrote:
> (3)
>
> In particular, I see a lot of value for (quoting the proposal)
>
> """
> Since then, there have been
> numerous updates, increased Python parity, and new features that do
> not have accompanying examples employing best practices and
> demonstrating an end-to-end experience for new users. We would like to
> leverage the existing examples by raising their visibility and
> auditing them.
> """
>
> and I think the situation would become *worse* on all these fronts
> with a separate repo (as well as the other issues mentioned,
> especially complexity). We should consider lowering the bar to liking
> to user-maintained examples that don't merit being in the main repo,
> as well as guidelines for adding examples in the main repo itself.
> On Thu, Aug 9, 2018 at 1:44 PM Ismaël Mejía <ie...@gmail.com> wrote:
> >
> > 3 for all the reasons discussed above. I think there are better ways to
> improve the status quo without the extra maintenance of having a new repo
> for this.
> >
> > On Thu, Aug 9, 2018 at 7:00 PM Ahmet Altay <al...@google.com> wrote:
> >>
> >> If we go forward with (3), could we actually update our documentation
> on how we will support casual example contributions? I think we will need
> to have information on how to add links to the new examples people want to
> add to the set, what examples would be good additions to the Beam repo and
> what examples would be better maintained somewhere else by their owners,
> and what could they expect from our community when they work on such
> examples.
> >>
> >> On Thu, Aug 9, 2018 at 9:41 AM, Mikhail Gryzykhin <mi...@google.com>
> wrote:
> >>>
> >>> 3 (if contributors are up for voting) - We want to have beam
> maintained examples in main repo. This will give good man to users and
> allow us to test those easily with minimal maintenance.
> >>>
> >>> We can add links to opensource user repositories to our
> documentation/wiki. This will be flexible enough to provide external
> examples on one hand, and avoid responsibility of maintaining user code on
> the other hand.
> >>>
> >>> --Mikhail
> >>>
> >>> Have feedback?
> >>>
> >>>
> >>> On Thu, Aug 9, 2018 at 8:57 AM Rafael Fernandez <rf...@google.com>
> wrote:
> >>>>
> >>>> Here is the Rose', David's, and Gris' proposal in text form, I hope
> >>>> the copy/paste helps:
> >>>>
> >>>>
> >>>> Apache Beam Examples Repository
> >>>>
> >>>> Authors: Rose Nguyen (rtnguyen@google.com), David Cavazos
> >>>> (dcavazos@google.com), Gris Cuevas (gris@apache.org)
> >>>>
> >>>> Status: Proposal
> >>>> Created: 2018-07-30
> >>>> Updated: 2018-07-30
> >>>>
> >>>> Summary
> >>>>
> >>>> The Apache Beam Community creates and contributes examples to the core
> >>>> Apache Beam Github repository. We want to make the process easier and
> >>>> less dependent in the core repository by creating a separate repo,
> >>>> dedicated solely to Community examples, contribution guidelines and
> >>>> add the examples to the website.
> >>>>
> >>>> Background
> >>>>
> >>>> The original batch of examples on the Apache Beam GitHub repository
> >>>> was donated by Cloud Dataflow at the time of Java SDK 1.x to
> >>>> demonstrate the capability of this programming model. These initial
> >>>> examples were intended to demonstrate how a user can put together
> >>>> their code components and try out Beam. Since then, there have been
> >>>> numerous updates, increased Python parity, and new features that do
> >>>> not have accompanying examples employing best practices and
> >>>> demonstrating an end-to-end experience for new users. We would like to
> >>>> leverage the existing examples by raising their visibility and
> >>>> auditing them. This is also an opportunity to establish
> >>>> contribution/maintenance guidelines for community contributions and to
> >>>> start hosting the examples on the Beam site in an official repository.
> >>>> Attracting and retaining new users necessitates updated, concrete
> >>>> examples that exhibit the range of capabilities of Beam.
> >>>>
> >>>> Proposed Tasks
> >>>>
> >>>> We would like to create a new GitHub Repository under the Apache
> >>>> Software Foundation Org page for Apache Beam Community Examples. This
> >>>> repo would be similar to apache/beam-site. The name we’d like to have
> >>>> is apache/beam-examples. We will also move all current examples to
> >>>> this repo, perform an audit to outline best practices and guidelines
> >>>> and then publish them in the Apache Beam website.
> >>>>
> >>>>
> >>>> Here is an outlined list of tasks we propose:
> >>>>
> >>>> Send Apache Beam Example Repository proposal to the mailing list
> >>>> (David) - July 31
> >>>>
> >>>> Create the GitHub Repo (PMC would need to do this) - Request help
> >>>> after proposal is refined/accepted
> >>>>
> >>>> Move current examples to new repo (David) -- 2 weeks after item 2 is
> completed
> >>>>
> >>>> Add a note to let people we need to audit for best practices
> >>>>
> >>>> Audit current examples and define best practices (David, Rose, Gris)
> >>>> -- Target date: week of 8/20
> >>>>
> >>>> Write guidelines on adding new examples and maintaining them (Gris,
> >>>> Rose) -- Week after audit is completed
> >>>>
> >>>> Add examples to website (Rose) -- 1 week after guidelines are written
> >>>>
> >>>> Publish guidelines in website (Rose) -- 1 week after guidelines are
> written
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Aug 9, 2018 at 6:22 AM Łukasz Gajowy <lu...@gmail.com>
> wrote:
> >>>> >
> >>>> > I'd also vote for 3: I don't see much added value in separating the
> repos and I see much additional effort to be done in maintaining extra
> repo(s) (updating examples when new version of beam sdk comes out) and
> their infrastructure (jenkins, etc). What Lukasz Cwik said about mvn
> archetypes and how easy the examples can be to get starter examples from a
> common repo only strengthens my opinion.
> >>>> >
> >>>> > Regarding 2: I think it's not good to have some official examples
> here and some there - IMO it can make a false impression (user experience)
> that some examples are less important than the others. Maybe a good idea is
> to encourage users to share their (independent, non official) examples and
> create a list of such on the beam site instead of 2?
> >>>> >
> >>>> > Łukasz
> >>>> >
> >>>> > czw., 9 sie 2018 o 11:35 Alexey Romanenko <ar...@gmail.com>
> napisał(a):
> >>>> >>
> >>>> >> 3 - I agree with JB, Charles and Lukasz arguments above saying why
> we need to have examples and main code in the same repository (+ website
> code base will move there soon). I don’t see any huge benefits to have
> examples aside and, at the same time, it will bring additional complexity
> and burden for project support.
> >>>> >>
> >>>> >> On 9 Aug 2018, at 08:18, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >>>> >>
> >>>> >> Hi guys,
> >>>> >>
> >>>> >> For this kind of discussion, I would prefer to avoid Google Doc and
> >>>> >> directly put the point/proposal on the mailing list.
> >>>> >>
> >>>> >> It's easier for the community to follow.
> >>>> >>
> >>>> >> The statement is more for 3 because it's more convenient for users
> to
> >>>> >> easily find the examples and include in the distribution.
> >>>> >>
> >>>> >> Regards
> >>>> >> JB
> >>>> >>
> >>>> >> On 08/08/2018 23:25, Charles Chen wrote:
> >>>> >>
> >>>> >> It looks like the main claim is that 1 and 2 have the benefit of
> >>>> >> increasing visibility for examples on the Beam site. I agree with
> >>>> >> Robert's comments on the doc which claim that this is orthogonal to
> >>>> >> whether a separate repository is created (the comments are
> unresolved:
> >>>> >>
> https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY
> ).
> >>>> >>
> >>>> >> I would add that the maintenance and testing burden has not been
> >>>> >> adequately addressed in the proposal (i.e. are we creating new
> Jenkins
> >>>> >> jobs?; will postcommits on the main Beam repo run examples tests?;
> are
> >>>> >> we releasing artifacts--if so, is this together with the main
> package or
> >>>> >> separately in new packages?). If we go with the half-way solution
> in
> >>>> >> (2), there is also the issue of where the threshold is--for
> example, if
> >>>> >> a user-contributed example is particularly useful, do we move it
> to the
> >>>> >> main repo?
> >>>> >>
> >>>> >> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <gris@google.com
> >>>> >> <ma...@google.com>> wrote:
> >>>> >>
> >>>> >> I'd vote for 2.
> >>>> >>
> >>>> >> Giving independence to an example repository and creating the
> right
> >>>> >> infrastructure to maintain them will give visibility to the
> efforts
> >>>> >> our users are creating to solve their uses cases with Beam. I
> also
> >>>> >> want to make the process of sharing common work more easily.
> >>>> >>
> >>>> >> Re:The examples that will remain in core, I agree that it's
> crucial
> >>>> >> to keep some examples for testing.
> >>>> >>
> >>>> >>
> >>>> >> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lcwik@google.com
> >>>> >> <ma...@google.com>> wrote:
> >>>> >>
> >>>> >> I would vote for 3.
> >>>> >>
> >>>> >> My reasoning is that Java has a good mechanism to get a
> >>>> >> starter/example project going by using the the maven
> archetypes
> >>>> >> already. Our quickstart guide for Apache Beam for the Java
> SDK
> >>>> >> already covers generating the examples archetype.
> >>>> >> We could point users to the starter project at the end of
> the
> >>>> >> java quickstart.
> >>>> >>
> >>>> >> If python/go have a similar mechanism that is commonly
> used, I
> >>>> >> would go with those over creating a separate repo for
> examples
> >>>> >> and adding the maintenance burden involved.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ruwang@google.com
> >>>> >> <ma...@google.com>> wrote:
> >>>> >>
> >>>> >> 2 - examples that rely on experimental API can still
> stay in
> >>>> >> where they are because such examples could be changed.
> >>>> >>
> >>>> >> -Rui
> >>>> >>
> >>>> >> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <
> ccy@google.com
> >>>> >> <ma...@google.com>> wrote:
> >>>> >>
> >>>> >> 3 - We benefit from increased test coverage by
> having
> >>>> >> examples together with the rest of the code. As
> Robert
> >>>> >> mentions in the doc, hosting the Beam examples in
> the
> >>>> >> main repository is the best way to keep the examples
> >>>> >> visible, tested and maintained. Given that we
> recently
> >>>> >> moved to a single repository for the website since
> that
> >>>> >> previously caused a lot of pain, it makes sense to
> be
> >>>> >> consistent here.
> >>>> >>
> >>>> >> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay
> >>>> >> <altay@google.com <ma...@google.com>> wrote:
> >>>> >>
> >>>> >> 2 - Similar to Huygaa, I see value in keeping a
> core
> >>>> >> set of examples tested and maintained against
> head.
> >>>> >> At the same time I understand the value of a
> growing
> >>>> >> set of community grown examples that are
> targeted
> >>>> >> against a pre-defined versions of Beam and not
> >>>> >> necessarily updated at every release.
> >>>> >>
> >>>> >> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa
> Batsaikhan
> >>>> >> <batbat@google.com <ma...@google.com>>
> wrote:
> >>>> >>
> >>>> >> 2 - I like the idea of having a separate
> repo
> >>>> >> where we can have more freedom to check in
> >>>> >> examples. However, we benefit from having
> >>>> >> immediate core examples in Beam for testing
> >>>> >> purposes.
> >>>> >>
> >>>> >> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos
> >>>> >> <dcavazos@google.com
> >>>> >> <ma...@google.com>> wrote:
> >>>> >>
> >>>> >> Hi everyone!
> >>>> >>
> >>>> >> We discussed several options as well as
> some
> >>>> >> of the implications of each option.
> Please
> >>>> >> vote for your favorite option, feel
> free to
> >>>> >> back it up with any reasons that make
> you
> >>>> >> feel that way.
> >>>> >>
> >>>> >> 1) Move *all* samples to a *new
> >>>> >> *examples*repository*
> >>>> >> 2) Move *some* samples to a *new
> >>>> >> *examples*repository*
> >>>> >> 3) Leave samples where they are
> >>>> >>
> >>>> >> Some implications to creating a new
> repository:
> >>>> >> - Every example would be independent
> from
> >>>> >> every other example, so tests can be
> run in
> >>>> >> parallel
> >>>> >> - Examples would now show how to use
> Beam
> >>>> >> /externally/
> >>>> >> - The examples repository would need a
> >>>> >> testing infrastructure
> >>>> >> - Decoupling makes examples easier to
> test
> >>>> >> on different versions
> >>>> >> - Easier to copy-paste an existing
> example
> >>>> >> and start from there, almost like a
> template
> >>>> >> - Smaller size for the core Beam library
> >>>> >> - Two different repositories to maintain
> >>>> >> - Versioning could mirror Beam's
> current version
> >>>> >>
> >>>> >> Link to proposal
> >>>> >> <
> https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing
> >
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> --
> >>>> >> Jean-Baptiste Onofré
> >>>> >> jbonofre@apache.org
> >>>> >> http://blog.nanthrax.net
> >>>> >> Talend - http://www.talend.com
> >>>> >>
> >>>> >>
> >>
> >>
>
Re: [VOTE] Community Examples Repository
Posted by Robert Bradshaw <ro...@google.com>.
(3)
In particular, I see a lot of value for (quoting the proposal)
"""
Since then, there have been
numerous updates, increased Python parity, and new features that do
not have accompanying examples employing best practices and
demonstrating an end-to-end experience for new users. We would like to
leverage the existing examples by raising their visibility and
auditing them.
"""
and I think the situation would become *worse* on all these fronts
with a separate repo (as well as the other issues mentioned,
especially complexity). We should consider lowering the bar to liking
to user-maintained examples that don't merit being in the main repo,
as well as guidelines for adding examples in the main repo itself.
On Thu, Aug 9, 2018 at 1:44 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
> 3 for all the reasons discussed above. I think there are better ways to improve the status quo without the extra maintenance of having a new repo for this.
>
> On Thu, Aug 9, 2018 at 7:00 PM Ahmet Altay <al...@google.com> wrote:
>>
>> If we go forward with (3), could we actually update our documentation on how we will support casual example contributions? I think we will need to have information on how to add links to the new examples people want to add to the set, what examples would be good additions to the Beam repo and what examples would be better maintained somewhere else by their owners, and what could they expect from our community when they work on such examples.
>>
>> On Thu, Aug 9, 2018 at 9:41 AM, Mikhail Gryzykhin <mi...@google.com> wrote:
>>>
>>> 3 (if contributors are up for voting) - We want to have beam maintained examples in main repo. This will give good man to users and allow us to test those easily with minimal maintenance.
>>>
>>> We can add links to opensource user repositories to our documentation/wiki. This will be flexible enough to provide external examples on one hand, and avoid responsibility of maintaining user code on the other hand.
>>>
>>> --Mikhail
>>>
>>> Have feedback?
>>>
>>>
>>> On Thu, Aug 9, 2018 at 8:57 AM Rafael Fernandez <rf...@google.com> wrote:
>>>>
>>>> Here is the Rose', David's, and Gris' proposal in text form, I hope
>>>> the copy/paste helps:
>>>>
>>>>
>>>> Apache Beam Examples Repository
>>>>
>>>> Authors: Rose Nguyen (rtnguyen@google.com), David Cavazos
>>>> (dcavazos@google.com), Gris Cuevas (gris@apache.org)
>>>>
>>>> Status: Proposal
>>>> Created: 2018-07-30
>>>> Updated: 2018-07-30
>>>>
>>>> Summary
>>>>
>>>> The Apache Beam Community creates and contributes examples to the core
>>>> Apache Beam Github repository. We want to make the process easier and
>>>> less dependent in the core repository by creating a separate repo,
>>>> dedicated solely to Community examples, contribution guidelines and
>>>> add the examples to the website.
>>>>
>>>> Background
>>>>
>>>> The original batch of examples on the Apache Beam GitHub repository
>>>> was donated by Cloud Dataflow at the time of Java SDK 1.x to
>>>> demonstrate the capability of this programming model. These initial
>>>> examples were intended to demonstrate how a user can put together
>>>> their code components and try out Beam. Since then, there have been
>>>> numerous updates, increased Python parity, and new features that do
>>>> not have accompanying examples employing best practices and
>>>> demonstrating an end-to-end experience for new users. We would like to
>>>> leverage the existing examples by raising their visibility and
>>>> auditing them. This is also an opportunity to establish
>>>> contribution/maintenance guidelines for community contributions and to
>>>> start hosting the examples on the Beam site in an official repository.
>>>> Attracting and retaining new users necessitates updated, concrete
>>>> examples that exhibit the range of capabilities of Beam.
>>>>
>>>> Proposed Tasks
>>>>
>>>> We would like to create a new GitHub Repository under the Apache
>>>> Software Foundation Org page for Apache Beam Community Examples. This
>>>> repo would be similar to apache/beam-site. The name we’d like to have
>>>> is apache/beam-examples. We will also move all current examples to
>>>> this repo, perform an audit to outline best practices and guidelines
>>>> and then publish them in the Apache Beam website.
>>>>
>>>>
>>>> Here is an outlined list of tasks we propose:
>>>>
>>>> Send Apache Beam Example Repository proposal to the mailing list
>>>> (David) - July 31
>>>>
>>>> Create the GitHub Repo (PMC would need to do this) - Request help
>>>> after proposal is refined/accepted
>>>>
>>>> Move current examples to new repo (David) -- 2 weeks after item 2 is completed
>>>>
>>>> Add a note to let people we need to audit for best practices
>>>>
>>>> Audit current examples and define best practices (David, Rose, Gris)
>>>> -- Target date: week of 8/20
>>>>
>>>> Write guidelines on adding new examples and maintaining them (Gris,
>>>> Rose) -- Week after audit is completed
>>>>
>>>> Add examples to website (Rose) -- 1 week after guidelines are written
>>>>
>>>> Publish guidelines in website (Rose) -- 1 week after guidelines are written
>>>>
>>>>
>>>>
>>>> On Thu, Aug 9, 2018 at 6:22 AM Łukasz Gajowy <lu...@gmail.com> wrote:
>>>> >
>>>> > I'd also vote for 3: I don't see much added value in separating the repos and I see much additional effort to be done in maintaining extra repo(s) (updating examples when new version of beam sdk comes out) and their infrastructure (jenkins, etc). What Lukasz Cwik said about mvn archetypes and how easy the examples can be to get starter examples from a common repo only strengthens my opinion.
>>>> >
>>>> > Regarding 2: I think it's not good to have some official examples here and some there - IMO it can make a false impression (user experience) that some examples are less important than the others. Maybe a good idea is to encourage users to share their (independent, non official) examples and create a list of such on the beam site instead of 2?
>>>> >
>>>> > Łukasz
>>>> >
>>>> > czw., 9 sie 2018 o 11:35 Alexey Romanenko <ar...@gmail.com> napisał(a):
>>>> >>
>>>> >> 3 - I agree with JB, Charles and Lukasz arguments above saying why we need to have examples and main code in the same repository (+ website code base will move there soon). I don’t see any huge benefits to have examples aside and, at the same time, it will bring additional complexity and burden for project support.
>>>> >>
>>>> >> On 9 Aug 2018, at 08:18, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>> >>
>>>> >> Hi guys,
>>>> >>
>>>> >> For this kind of discussion, I would prefer to avoid Google Doc and
>>>> >> directly put the point/proposal on the mailing list.
>>>> >>
>>>> >> It's easier for the community to follow.
>>>> >>
>>>> >> The statement is more for 3 because it's more convenient for users to
>>>> >> easily find the examples and include in the distribution.
>>>> >>
>>>> >> Regards
>>>> >> JB
>>>> >>
>>>> >> On 08/08/2018 23:25, Charles Chen wrote:
>>>> >>
>>>> >> It looks like the main claim is that 1 and 2 have the benefit of
>>>> >> increasing visibility for examples on the Beam site. I agree with
>>>> >> Robert's comments on the doc which claim that this is orthogonal to
>>>> >> whether a separate repository is created (the comments are unresolved:
>>>> >> https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY).
>>>> >>
>>>> >> I would add that the maintenance and testing burden has not been
>>>> >> adequately addressed in the proposal (i.e. are we creating new Jenkins
>>>> >> jobs?; will postcommits on the main Beam repo run examples tests?; are
>>>> >> we releasing artifacts--if so, is this together with the main package or
>>>> >> separately in new packages?). If we go with the half-way solution in
>>>> >> (2), there is also the issue of where the threshold is--for example, if
>>>> >> a user-contributed example is particularly useful, do we move it to the
>>>> >> main repo?
>>>> >>
>>>> >> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <gris@google.com
>>>> >> <ma...@google.com>> wrote:
>>>> >>
>>>> >> I'd vote for 2.
>>>> >>
>>>> >> Giving independence to an example repository and creating the right
>>>> >> infrastructure to maintain them will give visibility to the efforts
>>>> >> our users are creating to solve their uses cases with Beam. I also
>>>> >> want to make the process of sharing common work more easily.
>>>> >>
>>>> >> Re:The examples that will remain in core, I agree that it's crucial
>>>> >> to keep some examples for testing.
>>>> >>
>>>> >>
>>>> >> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lcwik@google.com
>>>> >> <ma...@google.com>> wrote:
>>>> >>
>>>> >> I would vote for 3.
>>>> >>
>>>> >> My reasoning is that Java has a good mechanism to get a
>>>> >> starter/example project going by using the the maven archetypes
>>>> >> already. Our quickstart guide for Apache Beam for the Java SDK
>>>> >> already covers generating the examples archetype.
>>>> >> We could point users to the starter project at the end of the
>>>> >> java quickstart.
>>>> >>
>>>> >> If python/go have a similar mechanism that is commonly used, I
>>>> >> would go with those over creating a separate repo for examples
>>>> >> and adding the maintenance burden involved.
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ruwang@google.com
>>>> >> <ma...@google.com>> wrote:
>>>> >>
>>>> >> 2 - examples that rely on experimental API can still stay in
>>>> >> where they are because such examples could be changed.
>>>> >>
>>>> >> -Rui
>>>> >>
>>>> >> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <ccy@google.com
>>>> >> <ma...@google.com>> wrote:
>>>> >>
>>>> >> 3 - We benefit from increased test coverage by having
>>>> >> examples together with the rest of the code. As Robert
>>>> >> mentions in the doc, hosting the Beam examples in the
>>>> >> main repository is the best way to keep the examples
>>>> >> visible, tested and maintained. Given that we recently
>>>> >> moved to a single repository for the website since that
>>>> >> previously caused a lot of pain, it makes sense to be
>>>> >> consistent here.
>>>> >>
>>>> >> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay
>>>> >> <altay@google.com <ma...@google.com>> wrote:
>>>> >>
>>>> >> 2 - Similar to Huygaa, I see value in keeping a core
>>>> >> set of examples tested and maintained against head.
>>>> >> At the same time I understand the value of a growing
>>>> >> set of community grown examples that are targeted
>>>> >> against a pre-defined versions of Beam and not
>>>> >> necessarily updated at every release.
>>>> >>
>>>> >> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan
>>>> >> <batbat@google.com <ma...@google.com>> wrote:
>>>> >>
>>>> >> 2 - I like the idea of having a separate repo
>>>> >> where we can have more freedom to check in
>>>> >> examples. However, we benefit from having
>>>> >> immediate core examples in Beam for testing
>>>> >> purposes.
>>>> >>
>>>> >> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos
>>>> >> <dcavazos@google.com
>>>> >> <ma...@google.com>> wrote:
>>>> >>
>>>> >> Hi everyone!
>>>> >>
>>>> >> We discussed several options as well as some
>>>> >> of the implications of each option. Please
>>>> >> vote for your favorite option, feel free to
>>>> >> back it up with any reasons that make you
>>>> >> feel that way.
>>>> >>
>>>> >> 1) Move *all* samples to a *new
>>>> >> *examples*repository*
>>>> >> 2) Move *some* samples to a *new
>>>> >> *examples*repository*
>>>> >> 3) Leave samples where they are
>>>> >>
>>>> >> Some implications to creating a new repository:
>>>> >> - Every example would be independent from
>>>> >> every other example, so tests can be run in
>>>> >> parallel
>>>> >> - Examples would now show how to use Beam
>>>> >> /externally/
>>>> >> - The examples repository would need a
>>>> >> testing infrastructure
>>>> >> - Decoupling makes examples easier to test
>>>> >> on different versions
>>>> >> - Easier to copy-paste an existing example
>>>> >> and start from there, almost like a template
>>>> >> - Smaller size for the core Beam library
>>>> >> - Two different repositories to maintain
>>>> >> - Versioning could mirror Beam's current version
>>>> >>
>>>> >> Link to proposal
>>>> >> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Jean-Baptiste Onofré
>>>> >> jbonofre@apache.org
>>>> >> http://blog.nanthrax.net
>>>> >> Talend - http://www.talend.com
>>>> >>
>>>> >>
>>
>>
Re: [VOTE] Community Examples Repository
Posted by Ismaël Mejía <ie...@gmail.com>.
3 for all the reasons discussed above. I think there are better ways to
improve the status quo without the extra maintenance of having a new repo
for this.
On Thu, Aug 9, 2018 at 7:00 PM Ahmet Altay <al...@google.com> wrote:
> If we go forward with (3), could we actually update our documentation on
> how we will support casual example contributions? I think we will need to
> have information on how to add links to the new examples people want to add
> to the set, what examples would be good additions to the Beam repo and what
> examples would be better maintained somewhere else by their owners, and
> what could they expect from our community when they work on such examples.
>
> On Thu, Aug 9, 2018 at 9:41 AM, Mikhail Gryzykhin <mi...@google.com>
> wrote:
>
>> 3 (if contributors are up for voting) - We want to have beam maintained
>> examples in main repo. This will give good man to users and allow us to
>> test those easily with minimal maintenance.
>>
>> We can add links to opensource user repositories to our
>> documentation/wiki. This will be flexible enough to provide external
>> examples on one hand, and avoid responsibility of maintaining user code on
>> the other hand.
>>
>> --Mikhail
>>
>> Have feedback <http://go/migryz-feedback>?
>>
>>
>> On Thu, Aug 9, 2018 at 8:57 AM Rafael Fernandez <rf...@google.com>
>> wrote:
>>
>>> Here is the Rose', David's, and Gris' proposal in text form, I hope
>>> the copy/paste helps:
>>>
>>>
>>> Apache Beam Examples Repository
>>>
>>> Authors: Rose Nguyen (rtnguyen@google.com), David Cavazos
>>> (dcavazos@google.com), Gris Cuevas (gris@apache.org)
>>>
>>> Status: Proposal
>>> Created: 2018-07-30
>>> Updated: 2018-07-30
>>>
>>> Summary
>>>
>>> The Apache Beam Community creates and contributes examples to the core
>>> Apache Beam Github repository. We want to make the process easier and
>>> less dependent in the core repository by creating a separate repo,
>>> dedicated solely to Community examples, contribution guidelines and
>>> add the examples to the website.
>>>
>>> Background
>>>
>>> The original batch of examples on the Apache Beam GitHub repository
>>> was donated by Cloud Dataflow at the time of Java SDK 1.x to
>>> demonstrate the capability of this programming model. These initial
>>> examples were intended to demonstrate how a user can put together
>>> their code components and try out Beam. Since then, there have been
>>> numerous updates, increased Python parity, and new features that do
>>> not have accompanying examples employing best practices and
>>> demonstrating an end-to-end experience for new users. We would like to
>>> leverage the existing examples by raising their visibility and
>>> auditing them. This is also an opportunity to establish
>>> contribution/maintenance guidelines for community contributions and to
>>> start hosting the examples on the Beam site in an official repository.
>>> Attracting and retaining new users necessitates updated, concrete
>>> examples that exhibit the range of capabilities of Beam.
>>>
>>> Proposed Tasks
>>>
>>> We would like to create a new GitHub Repository under the Apache
>>> Software Foundation Org page for Apache Beam Community Examples. This
>>> repo would be similar to apache/beam-site. The name we’d like to have
>>> is apache/beam-examples. We will also move all current examples to
>>> this repo, perform an audit to outline best practices and guidelines
>>> and then publish them in the Apache Beam website.
>>>
>>>
>>> Here is an outlined list of tasks we propose:
>>>
>>> Send Apache Beam Example Repository proposal to the mailing list
>>> (David) - July 31
>>>
>>> Create the GitHub Repo (PMC would need to do this) - Request help
>>> after proposal is refined/accepted
>>>
>>> Move current examples to new repo (David) -- 2 weeks after item 2 is
>>> completed
>>>
>>> Add a note to let people we need to audit for best practices
>>>
>>> Audit current examples and define best practices (David, Rose, Gris)
>>> -- Target date: week of 8/20
>>>
>>> Write guidelines on adding new examples and maintaining them (Gris,
>>> Rose) -- Week after audit is completed
>>>
>>> Add examples to website (Rose) -- 1 week after guidelines are written
>>>
>>> Publish guidelines in website (Rose) -- 1 week after guidelines are
>>> written
>>>
>>>
>>>
>>> On Thu, Aug 9, 2018 at 6:22 AM Łukasz Gajowy <lu...@gmail.com>
>>> wrote:
>>> >
>>> > I'd also vote for 3: I don't see much added value in separating the
>>> repos and I see much additional effort to be done in maintaining extra
>>> repo(s) (updating examples when new version of beam sdk comes out) and
>>> their infrastructure (jenkins, etc). What Lukasz Cwik said about mvn
>>> archetypes and how easy the examples can be to get starter examples from a
>>> common repo only strengthens my opinion.
>>> >
>>> > Regarding 2: I think it's not good to have some official examples here
>>> and some there - IMO it can make a false impression (user experience) that
>>> some examples are less important than the others. Maybe a good idea is to
>>> encourage users to share their (independent, non official) examples and
>>> create a list of such on the beam site instead of 2?
>>> >
>>> > Łukasz
>>> >
>>> > czw., 9 sie 2018 o 11:35 Alexey Romanenko <ar...@gmail.com>
>>> napisał(a):
>>> >>
>>> >> 3 - I agree with JB, Charles and Lukasz arguments above saying why we
>>> need to have examples and main code in the same repository (+ website code
>>> base will move there soon). I don’t see any huge benefits to have examples
>>> aside and, at the same time, it will bring additional complexity and burden
>>> for project support.
>>> >>
>>> >> On 9 Aug 2018, at 08:18, Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>> >>
>>> >> Hi guys,
>>> >>
>>> >> For this kind of discussion, I would prefer to avoid Google Doc and
>>> >> directly put the point/proposal on the mailing list.
>>> >>
>>> >> It's easier for the community to follow.
>>> >>
>>> >> The statement is more for 3 because it's more convenient for users to
>>> >> easily find the examples and include in the distribution.
>>> >>
>>> >> Regards
>>> >> JB
>>> >>
>>> >> On 08/08/2018 23:25, Charles Chen wrote:
>>> >>
>>> >> It looks like the main claim is that 1 and 2 have the benefit of
>>> >> increasing visibility for examples on the Beam site. I agree with
>>> >> Robert's comments on the doc which claim that this is orthogonal to
>>> >> whether a separate repository is created (the comments are unresolved:
>>> >>
>>> https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY
>>> ).
>>> >>
>>> >> I would add that the maintenance and testing burden has not been
>>> >> adequately addressed in the proposal (i.e. are we creating new Jenkins
>>> >> jobs?; will postcommits on the main Beam repo run examples tests?; are
>>> >> we releasing artifacts--if so, is this together with the main package
>>> or
>>> >> separately in new packages?). If we go with the half-way solution in
>>> >> (2), there is also the issue of where the threshold is--for example,
>>> if
>>> >> a user-contributed example is particularly useful, do we move it to
>>> the
>>> >> main repo?
>>> >>
>>> >> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <gris@google.com
>>> >> <ma...@google.com>> wrote:
>>> >>
>>> >> I'd vote for 2.
>>> >>
>>> >> Giving independence to an example repository and creating the right
>>> >> infrastructure to maintain them will give visibility to the efforts
>>> >> our users are creating to solve their uses cases with Beam. I also
>>> >> want to make the process of sharing common work more easily.
>>> >>
>>> >> Re:The examples that will remain in core, I agree that it's crucial
>>> >> to keep some examples for testing.
>>> >>
>>> >>
>>> >> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lcwik@google.com
>>> >> <ma...@google.com>> wrote:
>>> >>
>>> >> I would vote for 3.
>>> >>
>>> >> My reasoning is that Java has a good mechanism to get a
>>> >> starter/example project going by using the the maven archetypes
>>> >> already. Our quickstart guide for Apache Beam for the Java SDK
>>> >> already covers generating the examples archetype.
>>> >> We could point users to the starter project at the end of the
>>> >> java quickstart.
>>> >>
>>> >> If python/go have a similar mechanism that is commonly used, I
>>> >> would go with those over creating a separate repo for examples
>>> >> and adding the maintenance burden involved.
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ruwang@google.com
>>> >> <ma...@google.com>> wrote:
>>> >>
>>> >> 2 - examples that rely on experimental API can still stay
>>> in
>>> >> where they are because such examples could be changed.
>>> >>
>>> >> -Rui
>>> >>
>>> >> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <
>>> ccy@google.com
>>> >> <ma...@google.com>> wrote:
>>> >>
>>> >> 3 - We benefit from increased test coverage by having
>>> >> examples together with the rest of the code. As Robert
>>> >> mentions in the doc, hosting the Beam examples in the
>>> >> main repository is the best way to keep the examples
>>> >> visible, tested and maintained. Given that we recently
>>> >> moved to a single repository for the website since that
>>> >> previously caused a lot of pain, it makes sense to be
>>> >> consistent here.
>>> >>
>>> >> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay
>>> >> <altay@google.com <ma...@google.com>> wrote:
>>> >>
>>> >> 2 - Similar to Huygaa, I see value in keeping a
>>> core
>>> >> set of examples tested and maintained against head.
>>> >> At the same time I understand the value of a
>>> growing
>>> >> set of community grown examples that are targeted
>>> >> against a pre-defined versions of Beam and not
>>> >> necessarily updated at every release.
>>> >>
>>> >> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan
>>> >> <batbat@google.com <ma...@google.com>>
>>> wrote:
>>> >>
>>> >> 2 - I like the idea of having a separate repo
>>> >> where we can have more freedom to check in
>>> >> examples. However, we benefit from having
>>> >> immediate core examples in Beam for testing
>>> >> purposes.
>>> >>
>>> >> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos
>>> >> <dcavazos@google.com
>>> >> <ma...@google.com>> wrote:
>>> >>
>>> >> Hi everyone!
>>> >>
>>> >> We discussed several options as well as
>>> some
>>> >> of the implications of each option. Please
>>> >> vote for your favorite option, feel free to
>>> >> back it up with any reasons that make you
>>> >> feel that way.
>>> >>
>>> >> 1) Move *all* samples to a *new
>>> >> *examples*repository*
>>> >> 2) Move *some* samples to a *new
>>> >> *examples*repository*
>>> >> 3) Leave samples where they are
>>> >>
>>> >> Some implications to creating a new
>>> repository:
>>> >> - Every example would be independent from
>>> >> every other example, so tests can be run in
>>> >> parallel
>>> >> - Examples would now show how to use Beam
>>> >> /externally/
>>> >> - The examples repository would need a
>>> >> testing infrastructure
>>> >> - Decoupling makes examples easier to test
>>> >> on different versions
>>> >> - Easier to copy-paste an existing example
>>> >> and start from there, almost like a
>>> template
>>> >> - Smaller size for the core Beam library
>>> >> - Two different repositories to maintain
>>> >> - Versioning could mirror Beam's current
>>> version
>>> >>
>>> >> Link to proposal
>>> >> <
>>> https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing
>>> >
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Jean-Baptiste Onofré
>>> >> jbonofre@apache.org
>>> >> http://blog.nanthrax.net
>>> >> Talend - http://www.talend.com
>>> >>
>>> >>
>>>
>>
>
Re: [VOTE] Community Examples Repository
Posted by Ahmet Altay <al...@google.com>.
If we go forward with (3), could we actually update our documentation on
how we will support casual example contributions? I think we will need to
have information on how to add links to the new examples people want to add
to the set, what examples would be good additions to the Beam repo and what
examples would be better maintained somewhere else by their owners, and
what could they expect from our community when they work on such examples.
On Thu, Aug 9, 2018 at 9:41 AM, Mikhail Gryzykhin <mi...@google.com> wrote:
> 3 (if contributors are up for voting) - We want to have beam maintained
> examples in main repo. This will give good man to users and allow us to
> test those easily with minimal maintenance.
>
> We can add links to opensource user repositories to our
> documentation/wiki. This will be flexible enough to provide external
> examples on one hand, and avoid responsibility of maintaining user code on
> the other hand.
>
> --Mikhail
>
> Have feedback <http://go/migryz-feedback>?
>
>
> On Thu, Aug 9, 2018 at 8:57 AM Rafael Fernandez <rf...@google.com>
> wrote:
>
>> Here is the Rose', David's, and Gris' proposal in text form, I hope
>> the copy/paste helps:
>>
>>
>> Apache Beam Examples Repository
>>
>> Authors: Rose Nguyen (rtnguyen@google.com), David Cavazos
>> (dcavazos@google.com), Gris Cuevas (gris@apache.org)
>>
>> Status: Proposal
>> Created: 2018-07-30
>> Updated: 2018-07-30
>>
>> Summary
>>
>> The Apache Beam Community creates and contributes examples to the core
>> Apache Beam Github repository. We want to make the process easier and
>> less dependent in the core repository by creating a separate repo,
>> dedicated solely to Community examples, contribution guidelines and
>> add the examples to the website.
>>
>> Background
>>
>> The original batch of examples on the Apache Beam GitHub repository
>> was donated by Cloud Dataflow at the time of Java SDK 1.x to
>> demonstrate the capability of this programming model. These initial
>> examples were intended to demonstrate how a user can put together
>> their code components and try out Beam. Since then, there have been
>> numerous updates, increased Python parity, and new features that do
>> not have accompanying examples employing best practices and
>> demonstrating an end-to-end experience for new users. We would like to
>> leverage the existing examples by raising their visibility and
>> auditing them. This is also an opportunity to establish
>> contribution/maintenance guidelines for community contributions and to
>> start hosting the examples on the Beam site in an official repository.
>> Attracting and retaining new users necessitates updated, concrete
>> examples that exhibit the range of capabilities of Beam.
>>
>> Proposed Tasks
>>
>> We would like to create a new GitHub Repository under the Apache
>> Software Foundation Org page for Apache Beam Community Examples. This
>> repo would be similar to apache/beam-site. The name we’d like to have
>> is apache/beam-examples. We will also move all current examples to
>> this repo, perform an audit to outline best practices and guidelines
>> and then publish them in the Apache Beam website.
>>
>>
>> Here is an outlined list of tasks we propose:
>>
>> Send Apache Beam Example Repository proposal to the mailing list
>> (David) - July 31
>>
>> Create the GitHub Repo (PMC would need to do this) - Request help
>> after proposal is refined/accepted
>>
>> Move current examples to new repo (David) -- 2 weeks after item 2 is
>> completed
>>
>> Add a note to let people we need to audit for best practices
>>
>> Audit current examples and define best practices (David, Rose, Gris)
>> -- Target date: week of 8/20
>>
>> Write guidelines on adding new examples and maintaining them (Gris,
>> Rose) -- Week after audit is completed
>>
>> Add examples to website (Rose) -- 1 week after guidelines are written
>>
>> Publish guidelines in website (Rose) -- 1 week after guidelines are
>> written
>>
>>
>>
>> On Thu, Aug 9, 2018 at 6:22 AM Łukasz Gajowy <lu...@gmail.com>
>> wrote:
>> >
>> > I'd also vote for 3: I don't see much added value in separating the
>> repos and I see much additional effort to be done in maintaining extra
>> repo(s) (updating examples when new version of beam sdk comes out) and
>> their infrastructure (jenkins, etc). What Lukasz Cwik said about mvn
>> archetypes and how easy the examples can be to get starter examples from a
>> common repo only strengthens my opinion.
>> >
>> > Regarding 2: I think it's not good to have some official examples here
>> and some there - IMO it can make a false impression (user experience) that
>> some examples are less important than the others. Maybe a good idea is to
>> encourage users to share their (independent, non official) examples and
>> create a list of such on the beam site instead of 2?
>> >
>> > Łukasz
>> >
>> > czw., 9 sie 2018 o 11:35 Alexey Romanenko <ar...@gmail.com>
>> napisał(a):
>> >>
>> >> 3 - I agree with JB, Charles and Lukasz arguments above saying why we
>> need to have examples and main code in the same repository (+ website code
>> base will move there soon). I don’t see any huge benefits to have examples
>> aside and, at the same time, it will bring additional complexity and burden
>> for project support.
>> >>
>> >> On 9 Aug 2018, at 08:18, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>> >>
>> >> Hi guys,
>> >>
>> >> For this kind of discussion, I would prefer to avoid Google Doc and
>> >> directly put the point/proposal on the mailing list.
>> >>
>> >> It's easier for the community to follow.
>> >>
>> >> The statement is more for 3 because it's more convenient for users to
>> >> easily find the examples and include in the distribution.
>> >>
>> >> Regards
>> >> JB
>> >>
>> >> On 08/08/2018 23:25, Charles Chen wrote:
>> >>
>> >> It looks like the main claim is that 1 and 2 have the benefit of
>> >> increasing visibility for examples on the Beam site. I agree with
>> >> Robert's comments on the doc which claim that this is orthogonal to
>> >> whether a separate repository is created (the comments are unresolved:
>> >> https://docs.google.com/a/google.com/document/d/
>> 1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY).
>> >>
>> >> I would add that the maintenance and testing burden has not been
>> >> adequately addressed in the proposal (i.e. are we creating new Jenkins
>> >> jobs?; will postcommits on the main Beam repo run examples tests?; are
>> >> we releasing artifacts--if so, is this together with the main package
>> or
>> >> separately in new packages?). If we go with the half-way solution in
>> >> (2), there is also the issue of where the threshold is--for example, if
>> >> a user-contributed example is particularly useful, do we move it to the
>> >> main repo?
>> >>
>> >> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <gris@google.com
>> >> <ma...@google.com>> wrote:
>> >>
>> >> I'd vote for 2.
>> >>
>> >> Giving independence to an example repository and creating the right
>> >> infrastructure to maintain them will give visibility to the efforts
>> >> our users are creating to solve their uses cases with Beam. I also
>> >> want to make the process of sharing common work more easily.
>> >>
>> >> Re:The examples that will remain in core, I agree that it's crucial
>> >> to keep some examples for testing.
>> >>
>> >>
>> >> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lcwik@google.com
>> >> <ma...@google.com>> wrote:
>> >>
>> >> I would vote for 3.
>> >>
>> >> My reasoning is that Java has a good mechanism to get a
>> >> starter/example project going by using the the maven archetypes
>> >> already. Our quickstart guide for Apache Beam for the Java SDK
>> >> already covers generating the examples archetype.
>> >> We could point users to the starter project at the end of the
>> >> java quickstart.
>> >>
>> >> If python/go have a similar mechanism that is commonly used, I
>> >> would go with those over creating a separate repo for examples
>> >> and adding the maintenance burden involved.
>> >>
>> >>
>> >>
>> >> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ruwang@google.com
>> >> <ma...@google.com>> wrote:
>> >>
>> >> 2 - examples that rely on experimental API can still stay in
>> >> where they are because such examples could be changed.
>> >>
>> >> -Rui
>> >>
>> >> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <
>> ccy@google.com
>> >> <ma...@google.com>> wrote:
>> >>
>> >> 3 - We benefit from increased test coverage by having
>> >> examples together with the rest of the code. As Robert
>> >> mentions in the doc, hosting the Beam examples in the
>> >> main repository is the best way to keep the examples
>> >> visible, tested and maintained. Given that we recently
>> >> moved to a single repository for the website since that
>> >> previously caused a lot of pain, it makes sense to be
>> >> consistent here.
>> >>
>> >> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay
>> >> <altay@google.com <ma...@google.com>> wrote:
>> >>
>> >> 2 - Similar to Huygaa, I see value in keeping a core
>> >> set of examples tested and maintained against head.
>> >> At the same time I understand the value of a growing
>> >> set of community grown examples that are targeted
>> >> against a pre-defined versions of Beam and not
>> >> necessarily updated at every release.
>> >>
>> >> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan
>> >> <batbat@google.com <ma...@google.com>>
>> wrote:
>> >>
>> >> 2 - I like the idea of having a separate repo
>> >> where we can have more freedom to check in
>> >> examples. However, we benefit from having
>> >> immediate core examples in Beam for testing
>> >> purposes.
>> >>
>> >> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos
>> >> <dcavazos@google.com
>> >> <ma...@google.com>> wrote:
>> >>
>> >> Hi everyone!
>> >>
>> >> We discussed several options as well as some
>> >> of the implications of each option. Please
>> >> vote for your favorite option, feel free to
>> >> back it up with any reasons that make you
>> >> feel that way.
>> >>
>> >> 1) Move *all* samples to a *new
>> >> *examples*repository*
>> >> 2) Move *some* samples to a *new
>> >> *examples*repository*
>> >> 3) Leave samples where they are
>> >>
>> >> Some implications to creating a new
>> repository:
>> >> - Every example would be independent from
>> >> every other example, so tests can be run in
>> >> parallel
>> >> - Examples would now show how to use Beam
>> >> /externally/
>> >> - The examples repository would need a
>> >> testing infrastructure
>> >> - Decoupling makes examples easier to test
>> >> on different versions
>> >> - Easier to copy-paste an existing example
>> >> and start from there, almost like a template
>> >> - Smaller size for the core Beam library
>> >> - Two different repositories to maintain
>> >> - Versioning could mirror Beam's current
>> version
>> >>
>> >> Link to proposal
>> >> <https://docs.google.com/document/d/
>> 1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
>> >>
>> >>
>> >>
>> >> --
>> >> Jean-Baptiste Onofré
>> >> jbonofre@apache.org
>> >> http://blog.nanthrax.net
>> >> Talend - http://www.talend.com
>> >>
>> >>
>>
>
Re: [VOTE] Community Examples Repository
Posted by Mikhail Gryzykhin <mi...@google.com>.
3 (if contributors are up for voting) - We want to have beam maintained
examples in main repo. This will give good man to users and allow us to
test those easily with minimal maintenance.
We can add links to opensource user repositories to our documentation/wiki.
This will be flexible enough to provide external examples on one hand, and
avoid responsibility of maintaining user code on the other hand.
--Mikhail
Have feedback <http://go/migryz-feedback>?
On Thu, Aug 9, 2018 at 8:57 AM Rafael Fernandez <rf...@google.com> wrote:
> Here is the Rose', David's, and Gris' proposal in text form, I hope
> the copy/paste helps:
>
>
> Apache Beam Examples Repository
>
> Authors: Rose Nguyen (rtnguyen@google.com), David Cavazos
> (dcavazos@google.com), Gris Cuevas (gris@apache.org)
>
> Status: Proposal
> Created: 2018-07-30
> Updated: 2018-07-30
>
> Summary
>
> The Apache Beam Community creates and contributes examples to the core
> Apache Beam Github repository. We want to make the process easier and
> less dependent in the core repository by creating a separate repo,
> dedicated solely to Community examples, contribution guidelines and
> add the examples to the website.
>
> Background
>
> The original batch of examples on the Apache Beam GitHub repository
> was donated by Cloud Dataflow at the time of Java SDK 1.x to
> demonstrate the capability of this programming model. These initial
> examples were intended to demonstrate how a user can put together
> their code components and try out Beam. Since then, there have been
> numerous updates, increased Python parity, and new features that do
> not have accompanying examples employing best practices and
> demonstrating an end-to-end experience for new users. We would like to
> leverage the existing examples by raising their visibility and
> auditing them. This is also an opportunity to establish
> contribution/maintenance guidelines for community contributions and to
> start hosting the examples on the Beam site in an official repository.
> Attracting and retaining new users necessitates updated, concrete
> examples that exhibit the range of capabilities of Beam.
>
> Proposed Tasks
>
> We would like to create a new GitHub Repository under the Apache
> Software Foundation Org page for Apache Beam Community Examples. This
> repo would be similar to apache/beam-site. The name we’d like to have
> is apache/beam-examples. We will also move all current examples to
> this repo, perform an audit to outline best practices and guidelines
> and then publish them in the Apache Beam website.
>
>
> Here is an outlined list of tasks we propose:
>
> Send Apache Beam Example Repository proposal to the mailing list
> (David) - July 31
>
> Create the GitHub Repo (PMC would need to do this) - Request help
> after proposal is refined/accepted
>
> Move current examples to new repo (David) -- 2 weeks after item 2 is
> completed
>
> Add a note to let people we need to audit for best practices
>
> Audit current examples and define best practices (David, Rose, Gris)
> -- Target date: week of 8/20
>
> Write guidelines on adding new examples and maintaining them (Gris,
> Rose) -- Week after audit is completed
>
> Add examples to website (Rose) -- 1 week after guidelines are written
>
> Publish guidelines in website (Rose) -- 1 week after guidelines are written
>
>
>
> On Thu, Aug 9, 2018 at 6:22 AM Łukasz Gajowy <lu...@gmail.com>
> wrote:
> >
> > I'd also vote for 3: I don't see much added value in separating the
> repos and I see much additional effort to be done in maintaining extra
> repo(s) (updating examples when new version of beam sdk comes out) and
> their infrastructure (jenkins, etc). What Lukasz Cwik said about mvn
> archetypes and how easy the examples can be to get starter examples from a
> common repo only strengthens my opinion.
> >
> > Regarding 2: I think it's not good to have some official examples here
> and some there - IMO it can make a false impression (user experience) that
> some examples are less important than the others. Maybe a good idea is to
> encourage users to share their (independent, non official) examples and
> create a list of such on the beam site instead of 2?
> >
> > Łukasz
> >
> > czw., 9 sie 2018 o 11:35 Alexey Romanenko <ar...@gmail.com>
> napisał(a):
> >>
> >> 3 - I agree with JB, Charles and Lukasz arguments above saying why we
> need to have examples and main code in the same repository (+ website code
> base will move there soon). I don’t see any huge benefits to have examples
> aside and, at the same time, it will bring additional complexity and burden
> for project support.
> >>
> >> On 9 Aug 2018, at 08:18, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >>
> >> Hi guys,
> >>
> >> For this kind of discussion, I would prefer to avoid Google Doc and
> >> directly put the point/proposal on the mailing list.
> >>
> >> It's easier for the community to follow.
> >>
> >> The statement is more for 3 because it's more convenient for users to
> >> easily find the examples and include in the distribution.
> >>
> >> Regards
> >> JB
> >>
> >> On 08/08/2018 23:25, Charles Chen wrote:
> >>
> >> It looks like the main claim is that 1 and 2 have the benefit of
> >> increasing visibility for examples on the Beam site. I agree with
> >> Robert's comments on the doc which claim that this is orthogonal to
> >> whether a separate repository is created (the comments are unresolved:
> >>
> https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY
> ).
> >>
> >> I would add that the maintenance and testing burden has not been
> >> adequately addressed in the proposal (i.e. are we creating new Jenkins
> >> jobs?; will postcommits on the main Beam repo run examples tests?; are
> >> we releasing artifacts--if so, is this together with the main package or
> >> separately in new packages?). If we go with the half-way solution in
> >> (2), there is also the issue of where the threshold is--for example, if
> >> a user-contributed example is particularly useful, do we move it to the
> >> main repo?
> >>
> >> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <gris@google.com
> >> <ma...@google.com>> wrote:
> >>
> >> I'd vote for 2.
> >>
> >> Giving independence to an example repository and creating the right
> >> infrastructure to maintain them will give visibility to the efforts
> >> our users are creating to solve their uses cases with Beam. I also
> >> want to make the process of sharing common work more easily.
> >>
> >> Re:The examples that will remain in core, I agree that it's crucial
> >> to keep some examples for testing.
> >>
> >>
> >> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lcwik@google.com
> >> <ma...@google.com>> wrote:
> >>
> >> I would vote for 3.
> >>
> >> My reasoning is that Java has a good mechanism to get a
> >> starter/example project going by using the the maven archetypes
> >> already. Our quickstart guide for Apache Beam for the Java SDK
> >> already covers generating the examples archetype.
> >> We could point users to the starter project at the end of the
> >> java quickstart.
> >>
> >> If python/go have a similar mechanism that is commonly used, I
> >> would go with those over creating a separate repo for examples
> >> and adding the maintenance burden involved.
> >>
> >>
> >>
> >> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ruwang@google.com
> >> <ma...@google.com>> wrote:
> >>
> >> 2 - examples that rely on experimental API can still stay in
> >> where they are because such examples could be changed.
> >>
> >> -Rui
> >>
> >> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <ccy@google.com
> >> <ma...@google.com>> wrote:
> >>
> >> 3 - We benefit from increased test coverage by having
> >> examples together with the rest of the code. As Robert
> >> mentions in the doc, hosting the Beam examples in the
> >> main repository is the best way to keep the examples
> >> visible, tested and maintained. Given that we recently
> >> moved to a single repository for the website since that
> >> previously caused a lot of pain, it makes sense to be
> >> consistent here.
> >>
> >> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay
> >> <altay@google.com <ma...@google.com>> wrote:
> >>
> >> 2 - Similar to Huygaa, I see value in keeping a core
> >> set of examples tested and maintained against head.
> >> At the same time I understand the value of a growing
> >> set of community grown examples that are targeted
> >> against a pre-defined versions of Beam and not
> >> necessarily updated at every release.
> >>
> >> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan
> >> <batbat@google.com <ma...@google.com>>
> wrote:
> >>
> >> 2 - I like the idea of having a separate repo
> >> where we can have more freedom to check in
> >> examples. However, we benefit from having
> >> immediate core examples in Beam for testing
> >> purposes.
> >>
> >> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos
> >> <dcavazos@google.com
> >> <ma...@google.com>> wrote:
> >>
> >> Hi everyone!
> >>
> >> We discussed several options as well as some
> >> of the implications of each option. Please
> >> vote for your favorite option, feel free to
> >> back it up with any reasons that make you
> >> feel that way.
> >>
> >> 1) Move *all* samples to a *new
> >> *examples*repository*
> >> 2) Move *some* samples to a *new
> >> *examples*repository*
> >> 3) Leave samples where they are
> >>
> >> Some implications to creating a new
> repository:
> >> - Every example would be independent from
> >> every other example, so tests can be run in
> >> parallel
> >> - Examples would now show how to use Beam
> >> /externally/
> >> - The examples repository would need a
> >> testing infrastructure
> >> - Decoupling makes examples easier to test
> >> on different versions
> >> - Easier to copy-paste an existing example
> >> and start from there, almost like a template
> >> - Smaller size for the core Beam library
> >> - Two different repositories to maintain
> >> - Versioning could mirror Beam's current
> version
> >>
> >> Link to proposal
> >> <
> https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing
> >
> >>
> >>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >>
> >>
>
Re: [VOTE] Community Examples Repository
Posted by Rafael Fernandez <rf...@google.com>.
Here is the Rose', David's, and Gris' proposal in text form, I hope
the copy/paste helps:
Apache Beam Examples Repository
Authors: Rose Nguyen (rtnguyen@google.com), David Cavazos
(dcavazos@google.com), Gris Cuevas (gris@apache.org)
Status: Proposal
Created: 2018-07-30
Updated: 2018-07-30
Summary
The Apache Beam Community creates and contributes examples to the core
Apache Beam Github repository. We want to make the process easier and
less dependent in the core repository by creating a separate repo,
dedicated solely to Community examples, contribution guidelines and
add the examples to the website.
Background
The original batch of examples on the Apache Beam GitHub repository
was donated by Cloud Dataflow at the time of Java SDK 1.x to
demonstrate the capability of this programming model. These initial
examples were intended to demonstrate how a user can put together
their code components and try out Beam. Since then, there have been
numerous updates, increased Python parity, and new features that do
not have accompanying examples employing best practices and
demonstrating an end-to-end experience for new users. We would like to
leverage the existing examples by raising their visibility and
auditing them. This is also an opportunity to establish
contribution/maintenance guidelines for community contributions and to
start hosting the examples on the Beam site in an official repository.
Attracting and retaining new users necessitates updated, concrete
examples that exhibit the range of capabilities of Beam.
Proposed Tasks
We would like to create a new GitHub Repository under the Apache
Software Foundation Org page for Apache Beam Community Examples. This
repo would be similar to apache/beam-site. The name we’d like to have
is apache/beam-examples. We will also move all current examples to
this repo, perform an audit to outline best practices and guidelines
and then publish them in the Apache Beam website.
Here is an outlined list of tasks we propose:
Send Apache Beam Example Repository proposal to the mailing list
(David) - July 31
Create the GitHub Repo (PMC would need to do this) - Request help
after proposal is refined/accepted
Move current examples to new repo (David) -- 2 weeks after item 2 is completed
Add a note to let people we need to audit for best practices
Audit current examples and define best practices (David, Rose, Gris)
-- Target date: week of 8/20
Write guidelines on adding new examples and maintaining them (Gris,
Rose) -- Week after audit is completed
Add examples to website (Rose) -- 1 week after guidelines are written
Publish guidelines in website (Rose) -- 1 week after guidelines are written
On Thu, Aug 9, 2018 at 6:22 AM Łukasz Gajowy <lu...@gmail.com> wrote:
>
> I'd also vote for 3: I don't see much added value in separating the repos and I see much additional effort to be done in maintaining extra repo(s) (updating examples when new version of beam sdk comes out) and their infrastructure (jenkins, etc). What Lukasz Cwik said about mvn archetypes and how easy the examples can be to get starter examples from a common repo only strengthens my opinion.
>
> Regarding 2: I think it's not good to have some official examples here and some there - IMO it can make a false impression (user experience) that some examples are less important than the others. Maybe a good idea is to encourage users to share their (independent, non official) examples and create a list of such on the beam site instead of 2?
>
> Łukasz
>
> czw., 9 sie 2018 o 11:35 Alexey Romanenko <ar...@gmail.com> napisał(a):
>>
>> 3 - I agree with JB, Charles and Lukasz arguments above saying why we need to have examples and main code in the same repository (+ website code base will move there soon). I don’t see any huge benefits to have examples aside and, at the same time, it will bring additional complexity and burden for project support.
>>
>> On 9 Aug 2018, at 08:18, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>
>> Hi guys,
>>
>> For this kind of discussion, I would prefer to avoid Google Doc and
>> directly put the point/proposal on the mailing list.
>>
>> It's easier for the community to follow.
>>
>> The statement is more for 3 because it's more convenient for users to
>> easily find the examples and include in the distribution.
>>
>> Regards
>> JB
>>
>> On 08/08/2018 23:25, Charles Chen wrote:
>>
>> It looks like the main claim is that 1 and 2 have the benefit of
>> increasing visibility for examples on the Beam site. I agree with
>> Robert's comments on the doc which claim that this is orthogonal to
>> whether a separate repository is created (the comments are unresolved:
>> https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY).
>>
>> I would add that the maintenance and testing burden has not been
>> adequately addressed in the proposal (i.e. are we creating new Jenkins
>> jobs?; will postcommits on the main Beam repo run examples tests?; are
>> we releasing artifacts--if so, is this together with the main package or
>> separately in new packages?). If we go with the half-way solution in
>> (2), there is also the issue of where the threshold is--for example, if
>> a user-contributed example is particularly useful, do we move it to the
>> main repo?
>>
>> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <gris@google.com
>> <ma...@google.com>> wrote:
>>
>> I'd vote for 2.
>>
>> Giving independence to an example repository and creating the right
>> infrastructure to maintain them will give visibility to the efforts
>> our users are creating to solve their uses cases with Beam. I also
>> want to make the process of sharing common work more easily.
>>
>> Re:The examples that will remain in core, I agree that it's crucial
>> to keep some examples for testing.
>>
>>
>> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lcwik@google.com
>> <ma...@google.com>> wrote:
>>
>> I would vote for 3.
>>
>> My reasoning is that Java has a good mechanism to get a
>> starter/example project going by using the the maven archetypes
>> already. Our quickstart guide for Apache Beam for the Java SDK
>> already covers generating the examples archetype.
>> We could point users to the starter project at the end of the
>> java quickstart.
>>
>> If python/go have a similar mechanism that is commonly used, I
>> would go with those over creating a separate repo for examples
>> and adding the maintenance burden involved.
>>
>>
>>
>> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ruwang@google.com
>> <ma...@google.com>> wrote:
>>
>> 2 - examples that rely on experimental API can still stay in
>> where they are because such examples could be changed.
>>
>> -Rui
>>
>> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <ccy@google.com
>> <ma...@google.com>> wrote:
>>
>> 3 - We benefit from increased test coverage by having
>> examples together with the rest of the code. As Robert
>> mentions in the doc, hosting the Beam examples in the
>> main repository is the best way to keep the examples
>> visible, tested and maintained. Given that we recently
>> moved to a single repository for the website since that
>> previously caused a lot of pain, it makes sense to be
>> consistent here.
>>
>> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay
>> <altay@google.com <ma...@google.com>> wrote:
>>
>> 2 - Similar to Huygaa, I see value in keeping a core
>> set of examples tested and maintained against head.
>> At the same time I understand the value of a growing
>> set of community grown examples that are targeted
>> against a pre-defined versions of Beam and not
>> necessarily updated at every release.
>>
>> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan
>> <batbat@google.com <ma...@google.com>> wrote:
>>
>> 2 - I like the idea of having a separate repo
>> where we can have more freedom to check in
>> examples. However, we benefit from having
>> immediate core examples in Beam for testing
>> purposes.
>>
>> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos
>> <dcavazos@google.com
>> <ma...@google.com>> wrote:
>>
>> Hi everyone!
>>
>> We discussed several options as well as some
>> of the implications of each option. Please
>> vote for your favorite option, feel free to
>> back it up with any reasons that make you
>> feel that way.
>>
>> 1) Move *all* samples to a *new
>> *examples*repository*
>> 2) Move *some* samples to a *new
>> *examples*repository*
>> 3) Leave samples where they are
>>
>> Some implications to creating a new repository:
>> - Every example would be independent from
>> every other example, so tests can be run in
>> parallel
>> - Examples would now show how to use Beam
>> /externally/
>> - The examples repository would need a
>> testing infrastructure
>> - Decoupling makes examples easier to test
>> on different versions
>> - Easier to copy-paste an existing example
>> and start from there, almost like a template
>> - Smaller size for the core Beam library
>> - Two different repositories to maintain
>> - Versioning could mirror Beam's current version
>>
>> Link to proposal
>> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
>>
>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>>
Re: [VOTE] Community Examples Repository
Posted by Łukasz Gajowy <lu...@gmail.com>.
I'd also vote for 3: I don't see much added value in separating the
repos and I see much additional effort to be done in maintaining extra
repo(s) (updating examples when new version of beam sdk comes out) and
their infrastructure (jenkins, etc). What Lukasz Cwik said about mvn
archetypes and how easy the examples can be to get starter examples from a
common repo only strengthens my opinion.
Regarding 2: I think it's not good to have some official examples here and
some there - IMO it can make a false impression (user experience) that some
examples are less important than the others. Maybe a good idea is to
encourage users to share their (independent, non official) examples and
create a list of such on the beam site instead of 2?
Łukasz
czw., 9 sie 2018 o 11:35 Alexey Romanenko <ar...@gmail.com>
napisał(a):
> 3 - I agree with JB, Charles and Lukasz arguments above saying why we need
> to have examples and main code in the same repository (+ website code base
> will move there soon). I don’t see any huge benefits to have examples aside
> and, at the same time, it will bring additional complexity and burden for
> project support.
>
> On 9 Aug 2018, at 08:18, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi guys,
>
> For this kind of discussion, I would prefer to avoid Google Doc and
> directly put the point/proposal on the mailing list.
>
> It's easier for the community to follow.
>
> The statement is more for 3 because it's more convenient for users to
> easily find the examples and include in the distribution.
>
> Regards
> JB
>
> On 08/08/2018 23:25, Charles Chen wrote:
>
> It looks like the main claim is that 1 and 2 have the benefit of
> increasing visibility for examples on the Beam site. I agree with
> Robert's comments on the doc which claim that this is orthogonal to
> whether a separate repository is created (the comments are unresolved:
>
> https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY
> ).
>
> I would add that the maintenance and testing burden has not been
> adequately addressed in the proposal (i.e. are we creating new Jenkins
> jobs?; will postcommits on the main Beam repo run examples tests?; are
> we releasing artifacts--if so, is this together with the main package or
> separately in new packages?). If we go with the half-way solution in
> (2), there is also the issue of where the threshold is--for example, if
> a user-contributed example is particularly useful, do we move it to the
> main repo?
>
> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <gris@google.com
> <mailto:gris@google.com <gr...@google.com>>> wrote:
>
> I'd vote for 2.
>
> Giving independence to an example repository and creating the right
> infrastructure to maintain them will give visibility to the efforts
> our users are creating to solve their uses cases with Beam. I also
> want to make the process of sharing common work more easily.
>
> Re:The examples that will remain in core, I agree that it's crucial
> to keep some examples for testing.
>
>
> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lcwik@google.com
> <mailto:lcwik@google.com <lc...@google.com>>> wrote:
>
> I would vote for 3.
>
> My reasoning is that Java has a good mechanism to get a
> starter/example project going by using the the maven archetypes
> already. Our quickstart guide for Apache Beam for the Java SDK
> already covers generating the examples archetype.
> We could point users to the starter project at the end of the
> java quickstart.
>
> If python/go have a similar mechanism that is commonly used, I
> would go with those over creating a separate repo for examples
> and adding the maintenance burden involved.
>
>
>
> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ruwang@google.com
> <mailto:ruwang@google.com <ru...@google.com>>> wrote:
>
> 2 - examples that rely on experimental API can still stay in
> where they are because such examples could be changed.
>
> -Rui
>
> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <ccy@google.com
> <mailto:ccy@google.com <cc...@google.com>>> wrote:
>
> 3 - We benefit from increased test coverage by having
> examples together with the rest of the code. As Robert
> mentions in the doc, hosting the Beam examples in the
> main repository is the best way to keep the examples
> visible, tested and maintained. Given that we recently
> moved to a single repository for the website since that
> previously caused a lot of pain, it makes sense to be
> consistent here.
>
> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay
> <altay@google.com <mailto:altay@google.com
> <al...@google.com>>> wrote:
>
> 2 - Similar to Huygaa, I see value in keeping a core
> set of examples tested and maintained against head.
> At the same time I understand the value of a growing
> set of community grown examples that are targeted
> against a pre-defined versions of Beam and not
> necessarily updated at every release.
>
> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan
> <batbat@google.com <mailto:batbat@google.com
> <ba...@google.com>>> wrote:
>
> 2 - I like the idea of having a separate repo
> where we can have more freedom to check in
> examples. However, we benefit from having
> immediate core examples in Beam for testing
> purposes.
>
> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos
> <dcavazos@google.com
> <mailto:dcavazos@google.com <dc...@google.com>>>
> wrote:
>
> Hi everyone!
>
> We discussed several options as well as some
> of the implications of each option. Please
> vote for your favorite option, feel free to
> back it up with any reasons that make you
> feel that way.
>
> 1) Move *all* samples to a *new
> *examples*repository*
> 2) Move *some* samples to a *new
> *examples*repository*
> 3) Leave samples where they are
>
> Some implications to creating a new repository:
> - Every example would be independent from
> every other example, so tests can be run in
> parallel
> - Examples would now show how to use Beam
> /externally/
> - The examples repository would need a
> testing infrastructure
> - Decoupling makes examples easier to test
> on different versions
> - Easier to copy-paste an existing example
> and start from there, almost like a template
> - Smaller size for the core Beam library
> - Two different repositories to maintain
> - Versioning could mirror Beam's current version
>
> Link to proposal
> <
> https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing
> >
>
>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
>
>
Re: [VOTE] Community Examples Repository
Posted by Alexey Romanenko <ar...@gmail.com>.
3 - I agree with JB, Charles and Lukasz arguments above saying why we need to have examples and main code in the same repository (+ website code base will move there soon). I don’t see any huge benefits to have examples aside and, at the same time, it will bring additional complexity and burden for project support.
> On 9 Aug 2018, at 08:18, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi guys,
>
> For this kind of discussion, I would prefer to avoid Google Doc and
> directly put the point/proposal on the mailing list.
>
> It's easier for the community to follow.
>
> The statement is more for 3 because it's more convenient for users to
> easily find the examples and include in the distribution.
>
> Regards
> JB
>
> On 08/08/2018 23:25, Charles Chen wrote:
>> It looks like the main claim is that 1 and 2 have the benefit of
>> increasing visibility for examples on the Beam site. I agree with
>> Robert's comments on the doc which claim that this is orthogonal to
>> whether a separate repository is created (the comments are unresolved:
>> https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY).
>>
>> I would add that the maintenance and testing burden has not been
>> adequately addressed in the proposal (i.e. are we creating new Jenkins
>> jobs?; will postcommits on the main Beam repo run examples tests?; are
>> we releasing artifacts--if so, is this together with the main package or
>> separately in new packages?). If we go with the half-way solution in
>> (2), there is also the issue of where the threshold is--for example, if
>> a user-contributed example is particularly useful, do we move it to the
>> main repo?
>>
>> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <gris@google.com
>> <mailto:gris@google.com <ma...@google.com>>> wrote:
>>
>> I'd vote for 2.
>>
>> Giving independence to an example repository and creating the right
>> infrastructure to maintain them will give visibility to the efforts
>> our users are creating to solve their uses cases with Beam. I also
>> want to make the process of sharing common work more easily.
>>
>> Re:The examples that will remain in core, I agree that it's crucial
>> to keep some examples for testing.
>>
>>
>> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lcwik@google.com <ma...@google.com>
>> <mailto:lcwik@google.com <ma...@google.com>>> wrote:
>>
>> I would vote for 3.
>>
>> My reasoning is that Java has a good mechanism to get a
>> starter/example project going by using the the maven archetypes
>> already. Our quickstart guide for Apache Beam for the Java SDK
>> already covers generating the examples archetype.
>> We could point users to the starter project at the end of the
>> java quickstart.
>>
>> If python/go have a similar mechanism that is commonly used, I
>> would go with those over creating a separate repo for examples
>> and adding the maintenance burden involved.
>>
>>
>>
>> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ruwang@google.com <ma...@google.com>
>> <mailto:ruwang@google.com <ma...@google.com>>> wrote:
>>
>> 2 - examples that rely on experimental API can still stay in
>> where they are because such examples could be changed.
>>
>> -Rui
>>
>> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <ccy@google.com <ma...@google.com>
>> <mailto:ccy@google.com <ma...@google.com>>> wrote:
>>
>> 3 - We benefit from increased test coverage by having
>> examples together with the rest of the code. As Robert
>> mentions in the doc, hosting the Beam examples in the
>> main repository is the best way to keep the examples
>> visible, tested and maintained. Given that we recently
>> moved to a single repository for the website since that
>> previously caused a lot of pain, it makes sense to be
>> consistent here.
>>
>> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay
>> <altay@google.com <ma...@google.com> <mailto:altay@google.com <ma...@google.com>>> wrote:
>>
>> 2 - Similar to Huygaa, I see value in keeping a core
>> set of examples tested and maintained against head.
>> At the same time I understand the value of a growing
>> set of community grown examples that are targeted
>> against a pre-defined versions of Beam and not
>> necessarily updated at every release.
>>
>> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan
>> <batbat@google.com <ma...@google.com> <mailto:batbat@google.com <ma...@google.com>>> wrote:
>>
>> 2 - I like the idea of having a separate repo
>> where we can have more freedom to check in
>> examples. However, we benefit from having
>> immediate core examples in Beam for testing
>> purposes.
>>
>> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos
>> <dcavazos@google.com <ma...@google.com>
>> <mailto:dcavazos@google.com <ma...@google.com>>> wrote:
>>
>> Hi everyone!
>>
>> We discussed several options as well as some
>> of the implications of each option. Please
>> vote for your favorite option, feel free to
>> back it up with any reasons that make you
>> feel that way.
>>
>> 1) Move *all* samples to a *new
>> *examples*repository*
>> 2) Move *some* samples to a *new
>> *examples*repository*
>> 3) Leave samples where they are
>>
>> Some implications to creating a new repository:
>> - Every example would be independent from
>> every other example, so tests can be run in
>> parallel
>> - Examples would now show how to use Beam
>> /externally/
>> - The examples repository would need a
>> testing infrastructure
>> - Decoupling makes examples easier to test
>> on different versions
>> - Easier to copy-paste an existing example
>> and start from there, almost like a template
>> - Smaller size for the core Beam library
>> - Two different repositories to maintain
>> - Versioning could mirror Beam's current version
>>
>> Link to proposal
>> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>>
>>
>>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> Talend - http://www.talend.com <http://www.talend.com/>
Re: [VOTE] Community Examples Repository
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi guys,
For this kind of discussion, I would prefer to avoid Google Doc and
directly put the point/proposal on the mailing list.
It's easier for the community to follow.
The statement is more for 3 because it's more convenient for users to
easily find the examples and include in the distribution.
Regards
JB
On 08/08/2018 23:25, Charles Chen wrote:
> It looks like the main claim is that 1 and 2 have the benefit of
> increasing visibility for examples on the Beam site. I agree with
> Robert's comments on the doc which claim that this is orthogonal to
> whether a separate repository is created (the comments are unresolved:
> https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY).
>
> I would add that the maintenance and testing burden has not been
> adequately addressed in the proposal (i.e. are we creating new Jenkins
> jobs?; will postcommits on the main Beam repo run examples tests?; are
> we releasing artifacts--if so, is this together with the main package or
> separately in new packages?). If we go with the half-way solution in
> (2), there is also the issue of where the threshold is--for example, if
> a user-contributed example is particularly useful, do we move it to the
> main repo?
>
> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <gris@google.com
> <ma...@google.com>> wrote:
>
> I'd vote for 2.
>
> Giving independence to an example repository and creating the right
> infrastructure to maintain them will give visibility to the efforts
> our users are creating to solve their uses cases with Beam. I also
> want to make the process of sharing common work more easily.
>
> Re:The examples that will remain in core, I agree that it's crucial
> to keep some examples for testing.
>
>
> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lcwik@google.com
> <ma...@google.com>> wrote:
>
> I would vote for 3.
>
> My reasoning is that Java has a good mechanism to get a
> starter/example project going by using the the maven archetypes
> already. Our quickstart guide for Apache Beam for the Java SDK
> already covers generating the examples archetype.
> We could point users to the starter project at the end of the
> java quickstart.
>
> If python/go have a similar mechanism that is commonly used, I
> would go with those over creating a separate repo for examples
> and adding the maintenance burden involved.
>
>
>
> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ruwang@google.com
> <ma...@google.com>> wrote:
>
> 2 - examples that rely on experimental API can still stay in
> where they are because such examples could be changed.
>
> -Rui
>
> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <ccy@google.com
> <ma...@google.com>> wrote:
>
> 3 - We benefit from increased test coverage by having
> examples together with the rest of the code. As Robert
> mentions in the doc, hosting the Beam examples in the
> main repository is the best way to keep the examples
> visible, tested and maintained. Given that we recently
> moved to a single repository for the website since that
> previously caused a lot of pain, it makes sense to be
> consistent here.
>
> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay
> <altay@google.com <ma...@google.com>> wrote:
>
> 2 - Similar to Huygaa, I see value in keeping a core
> set of examples tested and maintained against head.
> At the same time I understand the value of a growing
> set of community grown examples that are targeted
> against a pre-defined versions of Beam and not
> necessarily updated at every release.
>
> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan
> <batbat@google.com <ma...@google.com>> wrote:
>
> 2 - I like the idea of having a separate repo
> where we can have more freedom to check in
> examples. However, we benefit from having
> immediate core examples in Beam for testing
> purposes.
>
> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos
> <dcavazos@google.com
> <ma...@google.com>> wrote:
>
> Hi everyone!
>
> We discussed several options as well as some
> of the implications of each option. Please
> vote for your favorite option, feel free to
> back it up with any reasons that make you
> feel that way.
>
> 1) Move *all* samples to a *new
> *examples*repository*
> 2) Move *some* samples to a *new
> *examples*repository*
> 3) Leave samples where they are
>
> Some implications to creating a new repository:
> - Every example would be independent from
> every other example, so tests can be run in
> parallel
> - Examples would now show how to use Beam
> /externally/
> - The examples repository would need a
> testing infrastructure
> - Decoupling makes examples easier to test
> on different versions
> - Easier to copy-paste an existing example
> and start from there, almost like a template
> - Smaller size for the core Beam library
> - Two different repositories to maintain
> - Versioning could mirror Beam's current version
>
> Link to proposal
> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Community Examples Repository
Posted by Charles Chen <cc...@google.com>.
It looks like the main claim is that 1 and 2 have the benefit of increasing
visibility for examples on the Beam site. I agree with Robert's comments
on the doc which claim that this is orthogonal to whether a separate
repository is created (the comments are unresolved:
https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY
).
I would add that the maintenance and testing burden has not been adequately
addressed in the proposal (i.e. are we creating new Jenkins jobs?; will
postcommits on the main Beam repo run examples tests?; are we releasing
artifacts--if so, is this together with the main package or separately in
new packages?). If we go with the half-way solution in (2), there is also
the issue of where the threshold is--for example, if a user-contributed
example is particularly useful, do we move it to the main repo?
On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <gr...@google.com> wrote:
> I'd vote for 2.
>
> Giving independence to an example repository and creating the right
> infrastructure to maintain them will give visibility to the efforts our
> users are creating to solve their uses cases with Beam. I also want to make
> the process of sharing common work more easily.
>
> Re:The examples that will remain in core, I agree that it's crucial to
> keep some examples for testing.
>
>
> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lc...@google.com> wrote:
>
>> I would vote for 3.
>>
>> My reasoning is that Java has a good mechanism to get a starter/example
>> project going by using the the maven archetypes already. Our quickstart
>> guide for Apache Beam for the Java SDK already covers generating the
>> examples archetype.
>> We could point users to the starter project at the end of the java
>> quickstart.
>>
>> If python/go have a similar mechanism that is commonly used, I would go
>> with those over creating a separate repo for examples and adding the
>> maintenance burden involved.
>>
>>
>>
>> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ru...@google.com> wrote:
>>
>>> 2 - examples that rely on experimental API can still stay in where they
>>> are because such examples could be changed.
>>>
>>> -Rui
>>>
>>> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <cc...@google.com> wrote:
>>>
>>>> 3 - We benefit from increased test coverage by having examples together
>>>> with the rest of the code. As Robert mentions in the doc, hosting the Beam
>>>> examples in the main repository is the best way to keep the examples
>>>> visible, tested and maintained. Given that we recently moved to a single
>>>> repository for the website since that previously caused a lot of pain, it
>>>> makes sense to be consistent here.
>>>>
>>>> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>> 2 - Similar to Huygaa, I see value in keeping a core set of examples
>>>>> tested and maintained against head. At the same time I understand the value
>>>>> of a growing set of community grown examples that are targeted against a
>>>>> pre-defined versions of Beam and not necessarily updated at every release.
>>>>>
>>>>> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan <ba...@google.com>
>>>>> wrote:
>>>>>
>>>>>> 2 - I like the idea of having a separate repo where we can have more
>>>>>> freedom to check in examples. However, we benefit from having immediate
>>>>>> core examples in Beam for testing purposes.
>>>>>>
>>>>>> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos <dc...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi everyone!
>>>>>>>
>>>>>>> We discussed several options as well as some of the implications of
>>>>>>> each option. Please vote for your favorite option, feel free to back it up
>>>>>>> with any reasons that make you feel that way.
>>>>>>>
>>>>>>> 1) Move *all* samples to a *new *examples* repository*
>>>>>>> 2) Move *some* samples to a *new *examples* repository*
>>>>>>> 3) Leave samples where they are
>>>>>>>
>>>>>>> Some implications to creating a new repository:
>>>>>>> - Every example would be independent from every other example, so
>>>>>>> tests can be run in parallel
>>>>>>> - Examples would now show how to use Beam *externally*
>>>>>>> - The examples repository would need a testing infrastructure
>>>>>>> - Decoupling makes examples easier to test on different versions
>>>>>>> - Easier to copy-paste an existing example and start from there,
>>>>>>> almost like a template
>>>>>>> - Smaller size for the core Beam library
>>>>>>> - Two different repositories to maintain
>>>>>>> - Versioning could mirror Beam's current version
>>>>>>>
>>>>>>> Link to proposal
>>>>>>> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
>>>>>>>
>>>>>>
>>>>>
Re: [VOTE] Community Examples Repository
Posted by Griselda Cuevas <gr...@google.com>.
I'd vote for 2.
Giving independence to an example repository and creating the right
infrastructure to maintain them will give visibility to the efforts our
users are creating to solve their uses cases with Beam. I also want to make
the process of sharing common work more easily.
Re:The examples that will remain in core, I agree that it's crucial to keep
some examples for testing.
On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lc...@google.com> wrote:
> I would vote for 3.
>
> My reasoning is that Java has a good mechanism to get a starter/example
> project going by using the the maven archetypes already. Our quickstart
> guide for Apache Beam for the Java SDK already covers generating the
> examples archetype.
> We could point users to the starter project at the end of the java
> quickstart.
>
> If python/go have a similar mechanism that is commonly used, I would go
> with those over creating a separate repo for examples and adding the
> maintenance burden involved.
>
>
>
> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ru...@google.com> wrote:
>
>> 2 - examples that rely on experimental API can still stay in where they
>> are because such examples could be changed.
>>
>> -Rui
>>
>> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <cc...@google.com> wrote:
>>
>>> 3 - We benefit from increased test coverage by having examples together
>>> with the rest of the code. As Robert mentions in the doc, hosting the Beam
>>> examples in the main repository is the best way to keep the examples
>>> visible, tested and maintained. Given that we recently moved to a single
>>> repository for the website since that previously caused a lot of pain, it
>>> makes sense to be consistent here.
>>>
>>> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay <al...@google.com> wrote:
>>>
>>>> 2 - Similar to Huygaa, I see value in keeping a core set of examples
>>>> tested and maintained against head. At the same time I understand the value
>>>> of a growing set of community grown examples that are targeted against a
>>>> pre-defined versions of Beam and not necessarily updated at every release.
>>>>
>>>> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan <ba...@google.com>
>>>> wrote:
>>>>
>>>>> 2 - I like the idea of having a separate repo where we can have more
>>>>> freedom to check in examples. However, we benefit from having immediate
>>>>> core examples in Beam for testing purposes.
>>>>>
>>>>> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos <dc...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi everyone!
>>>>>>
>>>>>> We discussed several options as well as some of the implications of
>>>>>> each option. Please vote for your favorite option, feel free to back it up
>>>>>> with any reasons that make you feel that way.
>>>>>>
>>>>>> 1) Move *all* samples to a *new *examples* repository*
>>>>>> 2) Move *some* samples to a *new *examples* repository*
>>>>>> 3) Leave samples where they are
>>>>>>
>>>>>> Some implications to creating a new repository:
>>>>>> - Every example would be independent from every other example, so
>>>>>> tests can be run in parallel
>>>>>> - Examples would now show how to use Beam *externally*
>>>>>> - The examples repository would need a testing infrastructure
>>>>>> - Decoupling makes examples easier to test on different versions
>>>>>> - Easier to copy-paste an existing example and start from there,
>>>>>> almost like a template
>>>>>> - Smaller size for the core Beam library
>>>>>> - Two different repositories to maintain
>>>>>> - Versioning could mirror Beam's current version
>>>>>>
>>>>>> Link to proposal
>>>>>> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
>>>>>>
>>>>>
>>>>
Re: [VOTE] Community Examples Repository
Posted by Lukasz Cwik <lc...@google.com>.
I would vote for 3.
My reasoning is that Java has a good mechanism to get a starter/example
project going by using the the maven archetypes already. Our quickstart
guide for Apache Beam for the Java SDK already covers generating the
examples archetype.
We could point users to the starter project at the end of the java
quickstart.
If python/go have a similar mechanism that is commonly used, I would go
with those over creating a separate repo for examples and adding the
maintenance burden involved.
On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ru...@google.com> wrote:
> 2 - examples that rely on experimental API can still stay in where they
> are because such examples could be changed.
>
> -Rui
>
> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <cc...@google.com> wrote:
>
>> 3 - We benefit from increased test coverage by having examples together
>> with the rest of the code. As Robert mentions in the doc, hosting the Beam
>> examples in the main repository is the best way to keep the examples
>> visible, tested and maintained. Given that we recently moved to a single
>> repository for the website since that previously caused a lot of pain, it
>> makes sense to be consistent here.
>>
>> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay <al...@google.com> wrote:
>>
>>> 2 - Similar to Huygaa, I see value in keeping a core set of examples
>>> tested and maintained against head. At the same time I understand the value
>>> of a growing set of community grown examples that are targeted against a
>>> pre-defined versions of Beam and not necessarily updated at every release.
>>>
>>> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan <ba...@google.com>
>>> wrote:
>>>
>>>> 2 - I like the idea of having a separate repo where we can have more
>>>> freedom to check in examples. However, we benefit from having immediate
>>>> core examples in Beam for testing purposes.
>>>>
>>>> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos <dc...@google.com>
>>>> wrote:
>>>>
>>>>> Hi everyone!
>>>>>
>>>>> We discussed several options as well as some of the implications of
>>>>> each option. Please vote for your favorite option, feel free to back it up
>>>>> with any reasons that make you feel that way.
>>>>>
>>>>> 1) Move *all* samples to a *new *examples* repository*
>>>>> 2) Move *some* samples to a *new *examples* repository*
>>>>> 3) Leave samples where they are
>>>>>
>>>>> Some implications to creating a new repository:
>>>>> - Every example would be independent from every other example, so
>>>>> tests can be run in parallel
>>>>> - Examples would now show how to use Beam *externally*
>>>>> - The examples repository would need a testing infrastructure
>>>>> - Decoupling makes examples easier to test on different versions
>>>>> - Easier to copy-paste an existing example and start from there,
>>>>> almost like a template
>>>>> - Smaller size for the core Beam library
>>>>> - Two different repositories to maintain
>>>>> - Versioning could mirror Beam's current version
>>>>>
>>>>> Link to proposal
>>>>> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
>>>>>
>>>>
>>>
Re: [VOTE] Community Examples Repository
Posted by Rui Wang <ru...@google.com>.
2 - examples that rely on experimental API can still stay in where they are
because such examples could be changed.
-Rui
On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <cc...@google.com> wrote:
> 3 - We benefit from increased test coverage by having examples together
> with the rest of the code. As Robert mentions in the doc, hosting the Beam
> examples in the main repository is the best way to keep the examples
> visible, tested and maintained. Given that we recently moved to a single
> repository for the website since that previously caused a lot of pain, it
> makes sense to be consistent here.
>
> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay <al...@google.com> wrote:
>
>> 2 - Similar to Huygaa, I see value in keeping a core set of examples
>> tested and maintained against head. At the same time I understand the value
>> of a growing set of community grown examples that are targeted against a
>> pre-defined versions of Beam and not necessarily updated at every release.
>>
>> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan <ba...@google.com>
>> wrote:
>>
>>> 2 - I like the idea of having a separate repo where we can have more
>>> freedom to check in examples. However, we benefit from having immediate
>>> core examples in Beam for testing purposes.
>>>
>>> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos <dc...@google.com>
>>> wrote:
>>>
>>>> Hi everyone!
>>>>
>>>> We discussed several options as well as some of the implications of
>>>> each option. Please vote for your favorite option, feel free to back it up
>>>> with any reasons that make you feel that way.
>>>>
>>>> 1) Move *all* samples to a *new *examples* repository*
>>>> 2) Move *some* samples to a *new *examples* repository*
>>>> 3) Leave samples where they are
>>>>
>>>> Some implications to creating a new repository:
>>>> - Every example would be independent from every other example, so tests
>>>> can be run in parallel
>>>> - Examples would now show how to use Beam *externally*
>>>> - The examples repository would need a testing infrastructure
>>>> - Decoupling makes examples easier to test on different versions
>>>> - Easier to copy-paste an existing example and start from there, almost
>>>> like a template
>>>> - Smaller size for the core Beam library
>>>> - Two different repositories to maintain
>>>> - Versioning could mirror Beam's current version
>>>>
>>>> Link to proposal
>>>> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
>>>>
>>>
>>
Re: [VOTE] Community Examples Repository
Posted by Charles Chen <cc...@google.com>.
3 - We benefit from increased test coverage by having examples together
with the rest of the code. As Robert mentions in the doc, hosting the Beam
examples in the main repository is the best way to keep the examples
visible, tested and maintained. Given that we recently moved to a single
repository for the website since that previously caused a lot of pain, it
makes sense to be consistent here.
On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay <al...@google.com> wrote:
> 2 - Similar to Huygaa, I see value in keeping a core set of examples
> tested and maintained against head. At the same time I understand the value
> of a growing set of community grown examples that are targeted against a
> pre-defined versions of Beam and not necessarily updated at every release.
>
> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan <ba...@google.com>
> wrote:
>
>> 2 - I like the idea of having a separate repo where we can have more
>> freedom to check in examples. However, we benefit from having immediate
>> core examples in Beam for testing purposes.
>>
>> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos <dc...@google.com> wrote:
>>
>>> Hi everyone!
>>>
>>> We discussed several options as well as some of the implications of each
>>> option. Please vote for your favorite option, feel free to back it up with
>>> any reasons that make you feel that way.
>>>
>>> 1) Move *all* samples to a *new *examples* repository*
>>> 2) Move *some* samples to a *new *examples* repository*
>>> 3) Leave samples where they are
>>>
>>> Some implications to creating a new repository:
>>> - Every example would be independent from every other example, so tests
>>> can be run in parallel
>>> - Examples would now show how to use Beam *externally*
>>> - The examples repository would need a testing infrastructure
>>> - Decoupling makes examples easier to test on different versions
>>> - Easier to copy-paste an existing example and start from there, almost
>>> like a template
>>> - Smaller size for the core Beam library
>>> - Two different repositories to maintain
>>> - Versioning could mirror Beam's current version
>>>
>>> Link to proposal
>>> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
>>>
>>
>
Re: [VOTE] Community Examples Repository
Posted by Ahmet Altay <al...@google.com>.
2 - Similar to Huygaa, I see value in keeping a core set of examples tested
and maintained against head. At the same time I understand the value of a
growing set of community grown examples that are targeted against a
pre-defined versions of Beam and not necessarily updated at every release.
On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan <ba...@google.com>
wrote:
> 2 - I like the idea of having a separate repo where we can have more
> freedom to check in examples. However, we benefit from having immediate
> core examples in Beam for testing purposes.
>
> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos <dc...@google.com> wrote:
>
>> Hi everyone!
>>
>> We discussed several options as well as some of the implications of each
>> option. Please vote for your favorite option, feel free to back it up with
>> any reasons that make you feel that way.
>>
>> 1) Move *all* samples to a *new *examples* repository*
>> 2) Move *some* samples to a *new *examples* repository*
>> 3) Leave samples where they are
>>
>> Some implications to creating a new repository:
>> - Every example would be independent from every other example, so tests
>> can be run in parallel
>> - Examples would now show how to use Beam *externally*
>> - The examples repository would need a testing infrastructure
>> - Decoupling makes examples easier to test on different versions
>> - Easier to copy-paste an existing example and start from there, almost
>> like a template
>> - Smaller size for the core Beam library
>> - Two different repositories to maintain
>> - Versioning could mirror Beam's current version
>>
>> Link to proposal
>> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
>>
>
Re: [VOTE] Community Examples Repository
Posted by Huygaa Batsaikhan <ba...@google.com>.
2 - I like the idea of having a separate repo where we can have more
freedom to check in examples. However, we benefit from having immediate
core examples in Beam for testing purposes.
On Wed, Aug 8, 2018 at 9:38 AM David Cavazos <dc...@google.com> wrote:
> Hi everyone!
>
> We discussed several options as well as some of the implications of each
> option. Please vote for your favorite option, feel free to back it up with
> any reasons that make you feel that way.
>
> 1) Move *all* samples to a *new *examples* repository*
> 2) Move *some* samples to a *new *examples* repository*
> 3) Leave samples where they are
>
> Some implications to creating a new repository:
> - Every example would be independent from every other example, so tests
> can be run in parallel
> - Examples would now show how to use Beam *externally*
> - The examples repository would need a testing infrastructure
> - Decoupling makes examples easier to test on different versions
> - Easier to copy-paste an existing example and start from there, almost
> like a template
> - Smaller size for the core Beam library
> - Two different repositories to maintain
> - Versioning could mirror Beam's current version
>
> Link to proposal
> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing>
>