You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Etienne Chauchot <ec...@apache.org> on 2019/01/23 09:00:40 UTC
Re: [PROPOSAL] allow the users to anticipate the support of
features in the targeted runner.
HI guys,
As part of our user growth, I'd like to revive this subject.I have sketched up a 2 pages proposal on this:
https://docs.google.com/document/d/1eXt54ht0h7-pPbP-MJR0N5nzmxRRlAwbFod-LXI1x0A/edit?usp=sharing
Unfortunately I have no knowledge on IDE plugin developement. Does someone have this knowledge and the envy to code it ?
Best
Etienne
Le mercredi 24 octobre 2018 à 09:45 +0200, Etienne Chauchot a écrit :
> Hi guys,
> To sum up what we said, I just opened this ticket:https://issues.apache.org/jira/browse/BEAM-5849
> Etienne
> Le jeudi 18 octobre 2018 à 12:44 +0200, Maximilian Michels a écrit :
> > Plugins for IDEs would be amazing because they could provide feedback already during pipeline construction, but I'm
> > not sure about the effort required to develop/maintain such plugins.
> > Ultimately, Runners have to decide whether they can translate the given pipeline or not. So I'm leaning more towards
> > an approach to make intermediate checking of the pipeline translation easier, e.g. by
> > - providing the target Runner already during development- running check of the Runner alongside with the
> > DirectRunner (which is typically used when developing pipelines)
> > On 17.10.18 15:57, Etienne Chauchot wrote:Hey Max, Kenn,
> > Thanks for your feedback !
> > Yes the idea was to inform the user as soon as possible, ideally while coding the pipeline. It could be done with a
> > IDE plugin (like checkstyle) that is configured with the targeted runner; that way the targeted runner conf is not
> > part of the pipeline code in an annotation which would be against Beam portability philosophy. Such a plugin could
> > color unsupported features while coding.
> > Another way could be to implement it as a javadoc but it seems weak because not automatic enough.Another way could
> > be to implement it as a validation plugin in the build system but IMHO it is already too late for the user.
> > So, long story short, I'm more in favor of an IDE plugin or similar coding-time solution.
> > BestEtienne
> > Le mercredi 17 octobre 2018 à 12:11 +0200, Maximilian Michels a écrit :This is a good idea. It needs to be fleshed
> > out how the capability of aRunner would be visible to the user (apart from the compatibility matrix).
> > A dry-run feature would be useful, i.e. the user can run an inspectionon the pipeline to see if it contains any
> > features which are notsupported by the Runner.
> > On 17.10.18 00:03, Rui Wang wrote:Sounds like a good idea.
> > Sounds like while coding, user gets a list to show if a feature issupported on different runners. User can check the
> > list for the answer.Is my understanding correct? Will this approach become slow as number ofrunner grows? (it's just
> > a question as I am not familiar the performanceof combination of a long list, annotation and IDE)
> >
> > -Rui
> > On Sat, Oct 13, 2018 at 11:56 PM Reuven Lax <relax@google.com <ma...@google.com> <mailto:relax@google.com
> > <ma...@google.com>>> wrote:
> > Sounds like a good idea. I don't think it will work for all capabilities (e.g. some of them such as
> > "exactly once" apply to all of the API surface), but useful for the ones that we can capture.
> > On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot <echauchot@apache.org <ma...@apache.org> <mailt
> > o:echauchot@apache.org <ma...@apache.org>>> wrote:
> > Hi guys, As part of our user experience improvement to attract new Beam users, I would like
> > to suggest something:
> > Today we only have the capability matrix to inform users about features support among runners. But,
> > they might discover only when the pipeline runs, when they receive an exception, that a given
> > feature is not supported by the targeted runner. I would like to suggest to translate the capability matrix
> > into the API with annotations for example, so that, while coding, the user could know that, for now,
> > a given feature is not supported on the runner he targets.
> > I know that the runner is only specified at pipeline runtime, and that adding code would be a leak
> > of runner implementation and against portability. So it could be just informative annotations
> > like @Experimental for example with no annotation processor.
> > WDYT?
> > Etienne
Re: [PROPOSAL] allow the users to anticipate the support of features
in the targeted runner.
Posted by Łukasz Gajowy <lu...@gmail.com>.
Hi
thanks for the proposal and not abandoning this thread. This topic is very
important. I left some comments.
Thanks,
Łukasz
śr., 23 sty 2019 o 10:00 Etienne Chauchot <ec...@apache.org> napisał(a):
> HI guys,
>
> As part of our user growth, I'd like to revive this subject.
> I have sketched up a 2 pages proposal on this:
> https://docs.google.com/document/d/1eXt54ht0h7-pPbP-MJR0N5nzmxRRlAwbFod-LXI1x0A/edit?usp=sharing
>
> Unfortunately I have no knowledge on IDE plugin developement.
> Does someone have this knowledge and the envy to code it ?
>
> Best
>
> Etienne
>
> Le mercredi 24 octobre 2018 à 09:45 +0200, Etienne Chauchot a écrit :
>
> Hi guys,
>
> To sum up what we said, I just opened this ticket:
> https://issues.apache.org/jira/browse/BEAM-5849
>
> Etienne
>
> Le jeudi 18 octobre 2018 à 12:44 +0200, Maximilian Michels a écrit :
>
> Plugins for IDEs would be amazing because they could provide feedback
>
> already during pipeline construction, but I'm not sure about the effort
>
> required to develop/maintain such plugins.
>
>
> Ultimately, Runners have to decide whether they can translate the given
>
> pipeline or not. So I'm leaning more towards an approach to make
>
> intermediate checking of the pipeline translation easier, e.g. by
>
>
> - providing the target Runner already during development
>
> - running check of the Runner alongside with the DirectRunner (which is
>
> typically used when developing pipelines)
>
>
> On 17.10.18 15:57, Etienne Chauchot wrote:
>
> Hey Max, Kenn,
>
>
> Thanks for your feedback !
>
>
> Yes the idea was to inform the user as soon as possible, ideally while
>
> coding the pipeline. It could be done with a IDE plugin (like
>
> checkstyle) that is configured with the targeted runner; that way the
>
> targeted runner conf is not part of the pipeline code in an annotation
>
> which would be against Beam portability philosophy. Such a plugin could
>
> color unsupported features while coding.
>
>
> Another way could be to implement it as a javadoc but it seems weak
>
> because not automatic enough.
>
> Another way could be to implement it as a validation plugin in the build
>
> system but IMHO it is already too late for the user.
>
>
> So, long story short, I'm more in favor of an IDE plugin or similar
>
> coding-time solution.
>
>
> Best
>
> Etienne
>
>
> Le mercredi 17 octobre 2018 à 12:11 +0200, Maximilian Michels a écrit :
>
> This is a good idea. It needs to be fleshed out how the capability of a
>
> Runner would be visible to the user (apart from the compatibility matrix).
>
>
> A dry-run feature would be useful, i.e. the user can run an inspection
>
> on the pipeline to see if it contains any features which are not
>
> supported by the Runner.
>
>
> On 17.10.18 00:03, Rui Wang wrote:
>
> Sounds like a good idea.
>
>
> Sounds like while coding, user gets a list to show if a feature is
>
> supported on different runners. User can check the list for the answer.
>
> Is my understanding correct? Will this approach become slow as number of
>
> runner grows? (it's just a question as I am not familiar the performance
>
> of combination of a long list, annotation and IDE)
>
>
>
> -Rui
>
>
> On Sat, Oct 13, 2018 at 11:56 PM Reuven Lax <relax@google.com <ma...@google.com>
>
> <mailto:relax@google.com <ma...@google.com>>> wrote:
>
>
> Sounds like a good idea. I don't think it will work for all
>
> capabilities (e.g. some of them such as "exactly once" apply to all
>
> of the API surface), but useful for the ones that we can capture.
>
>
> On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot
>
> <echauchot@apache.org <ma...@apache.org> <mailto:echauchot@apache.org <ma...@apache.org>>> wrote:
>
>
> Hi guys,
>
> As part of our user experience improvement to attract new Beam
>
> users, I would like to suggest something:
>
>
> Today we only have the capability matrix to inform users about
>
> features support among runners. But, they might discover only
>
> when the pipeline runs, when they receive an exception, that a
>
> given feature is not supported by the targeted runner.
>
> I would like to suggest to translate the capability matrix into
>
> the API with annotations for example, so that, while coding, the
>
> user could know that, for now, a given feature is not supported
>
> on the runner he targets.
>
>
> I know that the runner is only specified at pipeline runtime,
>
> and that adding code would be a leak of runner implementation
>
> and against portability. So it could be just informative
>
> annotations like @Experimental for example with no annotation
>
> processor.
>
>
> WDYT?
>
>
> Etienne
>
>
>