You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Etienne Chauchot <ec...@apache.org> on 2018/10/04 09:43:34 UTC
[PROPOSAL] allow the users to anticipate the support of features in
the targeted runner.
Hi guys,
As part of our user experience improvement to attract new Beam users, I would like to suggest something:
Today we only have the capability matrix to inform users about features support among runners. But, they might discover
only when the pipeline runs, when they receive an exception, that a given feature is not supported by the targeted
runner.
I would like to suggest to translate the capability matrix into the API with annotations for example, so that, whilecoding, the user could know that, for now, a given feature is not supported on the runner he targets.
I know that the runner is only specified at pipeline runtime, and that adding code would be a leak of runner
implementation and against portability. So it could be just informative annotations like @Experimental for example with
no annotation processor.
WDYT?
Etienne
Re: [PROPOSAL] allow the users to anticipate the support of features
in the targeted runner.
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1
it sounds like a good idea to me.
I would allow us to "automatize" our features coverage and give more
vision to our users.
Regards
JB
On 04/10/2018 11:43, Etienne Chauchot wrote:
> Hi guys,
> As part of our user experience improvement to attract new Beam users, I
> would like to suggest something:
>
> Today we only have the capability matrix to inform users about features
> support among runners. But, they might discover only when the pipeline
> runs, when they receive an exception, that a given feature is not
> supported by the targeted runner.
> I would like to suggest to translate the capability matrix into the API
> with annotations for example, so that, while coding, the user could know
> that, for now, a given feature is not supported on the runner he targets.
>
> I know that the runner is only specified at pipeline runtime, and that
> adding code would be a leak of runner implementation and against
> portability. So it could be just informative annotations
> like @Experimental for example with no annotation processor.
>
> WDYT?
>
> Etienne
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [PROPOSAL] allow the users to anticipate the support of features
in the targeted runner.
Posted by Maximilian Michels <mx...@apache.org>.
Similar to how we have `validate()` on the Pipeline to check the
pipeline specification, dry-run would check the pipeline translation and
report errors back to the user.
Assuming that Runners throw errors for unsupported features, that would
already give users confidence that they will be able to run their
pipelines with a specific Runner.
On 17.10.18 15:28, Robert Bradshaw wrote:
> On Wed, Oct 17, 2018 at 3:17 PM Kenneth Knowles <kenn@apache.org
> <ma...@apache.org>> wrote:
>
> On Wed, Oct 17, 2018 at 3:12 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> A dry-run feature would be useful, i.e. the user can run an
> inspection
> on the pipeline to see if it contains any features which are not
> supported by the Runner.
>
>
> This seems extremely useful independent of an annotation processor
> (which also seems useful), and pretty easy to get done quickly.
>
>
> +1, this would be very useful. (It could also be useful for cheaper
> testing of the dataflow, or other non-local, runners.)
>
> On 17.10.18 00:03, Rui Wang wrote:
> > Sounds like a good idea.
> >
> > Sounds like while coding, user gets a list to show if a
> feature is
> > supported on different runners. User can check the list for
> the answer.
> > Is my understanding correct? Will this approach become slow
> as number of
> > runner grows? (it's just a question as I am not familiar the
> performance
> > of combination of a long list, annotation and IDE)
> >
> >
> > -Rui
> >
> > On Sat, Oct 13, 2018 at 11:56 PM Reuven Lax <relax@google.com
> <ma...@google.com>
> > <mailto:relax@google.com <ma...@google.com>>> wrote:
> >
> > Sounds like a good idea. I don't think it will work for all
> > capabilities (e.g. some of them such as "exactly once"
> apply to all
> > of the API surface), but useful for the ones that we can
> capture.
> >
> > On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot
> > <echauchot@apache.org <ma...@apache.org>
> <mailto:echauchot@apache.org <ma...@apache.org>>> wrote:
> >
> > Hi guys,
> > As part of our user experience improvement to attract
> new Beam
> > users, I would like to suggest something:
> >
> > Today we only have the capability matrix to inform
> users about
> > features support among runners. But, they might
> discover only
> > when the pipeline runs, when they receive an
> exception, that a
> > given feature is not supported by the targeted runner.
> > I would like to suggest to translate the capability
> matrix into
> > the API with annotations for example, so that, while
> coding, the
> > user could know that, for now, a given feature is not
> supported
> > on the runner he targets.
> >
> > I know that the runner is only specified at pipeline
> runtime,
> > and that adding code would be a leak of runner
> implementation
> > and against portability. So it could be just informative
> > annotations like @Experimental for example with no
> annotation
> > processor.
> >
> > WDYT?
> >
> > Etienne
> >
>
Re: [PROPOSAL] allow the users to anticipate the support of features
in the targeted runner.
Posted by Robert Bradshaw <ro...@google.com>.
On Wed, Oct 17, 2018 at 3:17 PM Kenneth Knowles <ke...@apache.org> wrote:
> On Wed, Oct 17, 2018 at 3:12 AM Maximilian Michels <mx...@apache.org> wrote:
>
>> A dry-run feature would be useful, i.e. the user can run an inspection
>> on the pipeline to see if it contains any features which are not
>> supported by the Runner.
>>
>
> This seems extremely useful independent of an annotation processor (which
> also seems useful), and pretty easy to get done quickly.
>
>>
+1, this would be very useful. (It could also be useful for cheaper testing
of the dataflow, or other non-local, runners.)
> On 17.10.18 00:03, Rui Wang wrote:
>> > Sounds like a good idea.
>> >
>> > Sounds like while coding, user gets a list to show if a feature is
>> > supported on different runners. User can check the list for the answer.
>> > Is my understanding correct? Will this approach become slow as number
>> of
>> > runner grows? (it's just a question as I am not familiar the
>> performance
>> > of combination of a long list, annotation and IDE)
>> >
>> >
>> > -Rui
>> >
>> > On Sat, Oct 13, 2018 at 11:56 PM Reuven Lax <relax@google.com
>> > <ma...@google.com>> wrote:
>> >
>> > Sounds like a good idea. I don't think it will work for all
>> > capabilities (e.g. some of them such as "exactly once" apply to all
>> > of the API surface), but useful for the ones that we can capture.
>> >
>> > On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot
>> > <echauchot@apache.org <ma...@apache.org>> wrote:
>> >
>> > Hi guys,
>> > As part of our user experience improvement to attract new Beam
>> > users, I would like to suggest something:
>> >
>> > Today we only have the capability matrix to inform users about
>> > features support among runners. But, they might discover only
>> > when the pipeline runs, when they receive an exception, that a
>> > given feature is not supported by the targeted runner.
>> > I would like to suggest to translate the capability matrix into
>> > the API with annotations for example, so that, while coding, the
>> > user could know that, for now, a given feature is not supported
>> > on the runner he targets.
>> >
>> > I know that the runner is only specified at pipeline runtime,
>> > and that adding code would be a leak of runner implementation
>> > and against portability. So it could be just informative
>> > annotations like @Experimental for example with no annotation
>> > processor.
>> >
>> > WDYT?
>> >
>> > Etienne
>> >
>>
>
Re: [PROPOSAL] allow the users to anticipate the support of features
in the targeted runner.
Posted by Kenneth Knowles <ke...@apache.org>.
On Wed, Oct 17, 2018 at 3:12 AM Maximilian Michels <mx...@apache.org> wrote:
> A dry-run feature would be useful, i.e. the user can run an inspection
> on the pipeline to see if it contains any features which are not
> supported by the Runner.
>
This seems extremely useful independent of an annotation processor (which
also seems useful), and pretty easy to get done quickly.
Kenn
>
> On 17.10.18 00:03, Rui Wang wrote:
> > Sounds like a good idea.
> >
> > Sounds like while coding, user gets a list to show if a feature is
> > supported on different runners. User can check the list for the answer.
> > Is my understanding correct? Will this approach become slow as number of
> > runner grows? (it's just a question as I am not familiar the performance
> > of combination of a long list, annotation and IDE)
> >
> >
> > -Rui
> >
> > On Sat, Oct 13, 2018 at 11:56 PM Reuven Lax <relax@google.com
> > <ma...@google.com>> wrote:
> >
> > Sounds like a good idea. I don't think it will work for all
> > capabilities (e.g. some of them such as "exactly once" apply to all
> > of the API surface), but useful for the ones that we can capture.
> >
> > On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot
> > <echauchot@apache.org <ma...@apache.org>> wrote:
> >
> > Hi guys,
> > As part of our user experience improvement to attract new Beam
> > users, I would like to suggest something:
> >
> > Today we only have the capability matrix to inform users about
> > features support among runners. But, they might discover only
> > when the pipeline runs, when they receive an exception, that a
> > given feature is not supported by the targeted runner.
> > I would like to suggest to translate the capability matrix into
> > the API with annotations for example, so that, while coding, the
> > user could know that, for now, a given feature is not supported
> > on the runner he targets.
> >
> > I know that the runner is only specified at pipeline runtime,
> > and that adding code would be a leak of runner implementation
> > and against portability. So it could be just informative
> > annotations like @Experimental for example with no annotation
> > processor.
> >
> > WDYT?
> >
> > Etienne
> >
>
Re: [PROPOSAL] allow the users to anticipate the support of features
in the targeted runner.
Posted by Łukasz Gajowy <lu...@gmail.com>.
Hi
thanks for the proposal and not abandoning this thread. This topic is very
important. I left some comments.
Thanks,
Łukasz
śr., 23 sty 2019 o 10:00 Etienne Chauchot <ec...@apache.org> napisał(a):
> HI guys,
>
> As part of our user growth, I'd like to revive this subject.
> I have sketched up a 2 pages proposal on this:
> https://docs.google.com/document/d/1eXt54ht0h7-pPbP-MJR0N5nzmxRRlAwbFod-LXI1x0A/edit?usp=sharing
>
> Unfortunately I have no knowledge on IDE plugin developement.
> Does someone have this knowledge and the envy to code it ?
>
> Best
>
> Etienne
>
> Le mercredi 24 octobre 2018 à 09:45 +0200, Etienne Chauchot a écrit :
>
> Hi guys,
>
> To sum up what we said, I just opened this ticket:
> https://issues.apache.org/jira/browse/BEAM-5849
>
> Etienne
>
> Le jeudi 18 octobre 2018 à 12:44 +0200, Maximilian Michels a écrit :
>
> Plugins for IDEs would be amazing because they could provide feedback
>
> already during pipeline construction, but I'm not sure about the effort
>
> required to develop/maintain such plugins.
>
>
> Ultimately, Runners have to decide whether they can translate the given
>
> pipeline or not. So I'm leaning more towards an approach to make
>
> intermediate checking of the pipeline translation easier, e.g. by
>
>
> - providing the target Runner already during development
>
> - running check of the Runner alongside with the DirectRunner (which is
>
> typically used when developing pipelines)
>
>
> On 17.10.18 15:57, Etienne Chauchot wrote:
>
> Hey Max, Kenn,
>
>
> Thanks for your feedback !
>
>
> Yes the idea was to inform the user as soon as possible, ideally while
>
> coding the pipeline. It could be done with a IDE plugin (like
>
> checkstyle) that is configured with the targeted runner; that way the
>
> targeted runner conf is not part of the pipeline code in an annotation
>
> which would be against Beam portability philosophy. Such a plugin could
>
> color unsupported features while coding.
>
>
> Another way could be to implement it as a javadoc but it seems weak
>
> because not automatic enough.
>
> Another way could be to implement it as a validation plugin in the build
>
> system but IMHO it is already too late for the user.
>
>
> So, long story short, I'm more in favor of an IDE plugin or similar
>
> coding-time solution.
>
>
> Best
>
> Etienne
>
>
> Le mercredi 17 octobre 2018 à 12:11 +0200, Maximilian Michels a écrit :
>
> This is a good idea. It needs to be fleshed out how the capability of a
>
> Runner would be visible to the user (apart from the compatibility matrix).
>
>
> A dry-run feature would be useful, i.e. the user can run an inspection
>
> on the pipeline to see if it contains any features which are not
>
> supported by the Runner.
>
>
> On 17.10.18 00:03, Rui Wang wrote:
>
> Sounds like a good idea.
>
>
> Sounds like while coding, user gets a list to show if a feature is
>
> supported on different runners. User can check the list for the answer.
>
> Is my understanding correct? Will this approach become slow as number of
>
> runner grows? (it's just a question as I am not familiar the performance
>
> of combination of a long list, annotation and IDE)
>
>
>
> -Rui
>
>
> On Sat, Oct 13, 2018 at 11:56 PM Reuven Lax <relax@google.com <ma...@google.com>
>
> <mailto:relax@google.com <ma...@google.com>>> wrote:
>
>
> Sounds like a good idea. I don't think it will work for all
>
> capabilities (e.g. some of them such as "exactly once" apply to all
>
> of the API surface), but useful for the ones that we can capture.
>
>
> On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot
>
> <echauchot@apache.org <ma...@apache.org> <mailto:echauchot@apache.org <ma...@apache.org>>> wrote:
>
>
> Hi guys,
>
> As part of our user experience improvement to attract new Beam
>
> users, I would like to suggest something:
>
>
> Today we only have the capability matrix to inform users about
>
> features support among runners. But, they might discover only
>
> when the pipeline runs, when they receive an exception, that a
>
> given feature is not supported by the targeted runner.
>
> I would like to suggest to translate the capability matrix into
>
> the API with annotations for example, so that, while coding, the
>
> user could know that, for now, a given feature is not supported
>
> on the runner he targets.
>
>
> I know that the runner is only specified at pipeline runtime,
>
> and that adding code would be a leak of runner implementation
>
> and against portability. So it could be just informative
>
> annotations like @Experimental for example with no annotation
>
> processor.
>
>
> WDYT?
>
>
> Etienne
>
>
>
Re: [PROPOSAL] allow the users to anticipate the support of
features in the targeted runner.
Posted by Etienne Chauchot <ec...@apache.org>.
HI guys,
As part of our user growth, I'd like to revive this subject.I have sketched up a 2 pages proposal on this:
https://docs.google.com/document/d/1eXt54ht0h7-pPbP-MJR0N5nzmxRRlAwbFod-LXI1x0A/edit?usp=sharing
Unfortunately I have no knowledge on IDE plugin developement. Does someone have this knowledge and the envy to code it ?
Best
Etienne
Le mercredi 24 octobre 2018 à 09:45 +0200, Etienne Chauchot a écrit :
> Hi guys,
> To sum up what we said, I just opened this ticket:https://issues.apache.org/jira/browse/BEAM-5849
> Etienne
> Le jeudi 18 octobre 2018 à 12:44 +0200, Maximilian Michels a écrit :
> > Plugins for IDEs would be amazing because they could provide feedback already during pipeline construction, but I'm
> > not sure about the effort required to develop/maintain such plugins.
> > Ultimately, Runners have to decide whether they can translate the given pipeline or not. So I'm leaning more towards
> > an approach to make intermediate checking of the pipeline translation easier, e.g. by
> > - providing the target Runner already during development- running check of the Runner alongside with the
> > DirectRunner (which is typically used when developing pipelines)
> > On 17.10.18 15:57, Etienne Chauchot wrote:Hey Max, Kenn,
> > Thanks for your feedback !
> > Yes the idea was to inform the user as soon as possible, ideally while coding the pipeline. It could be done with a
> > IDE plugin (like checkstyle) that is configured with the targeted runner; that way the targeted runner conf is not
> > part of the pipeline code in an annotation which would be against Beam portability philosophy. Such a plugin could
> > color unsupported features while coding.
> > Another way could be to implement it as a javadoc but it seems weak because not automatic enough.Another way could
> > be to implement it as a validation plugin in the build system but IMHO it is already too late for the user.
> > So, long story short, I'm more in favor of an IDE plugin or similar coding-time solution.
> > BestEtienne
> > Le mercredi 17 octobre 2018 à 12:11 +0200, Maximilian Michels a écrit :This is a good idea. It needs to be fleshed
> > out how the capability of aRunner would be visible to the user (apart from the compatibility matrix).
> > A dry-run feature would be useful, i.e. the user can run an inspectionon the pipeline to see if it contains any
> > features which are notsupported by the Runner.
> > On 17.10.18 00:03, Rui Wang wrote:Sounds like a good idea.
> > Sounds like while coding, user gets a list to show if a feature issupported on different runners. User can check the
> > list for the answer.Is my understanding correct? Will this approach become slow as number ofrunner grows? (it's just
> > a question as I am not familiar the performanceof combination of a long list, annotation and IDE)
> >
> > -Rui
> > On Sat, Oct 13, 2018 at 11:56 PM Reuven Lax <relax@google.com <ma...@google.com> <mailto:relax@google.com
> > <ma...@google.com>>> wrote:
> > Sounds like a good idea. I don't think it will work for all capabilities (e.g. some of them such as
> > "exactly once" apply to all of the API surface), but useful for the ones that we can capture.
> > On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot <echauchot@apache.org <ma...@apache.org> <mailt
> > o:echauchot@apache.org <ma...@apache.org>>> wrote:
> > Hi guys, As part of our user experience improvement to attract new Beam users, I would like
> > to suggest something:
> > Today we only have the capability matrix to inform users about features support among runners. But,
> > they might discover only when the pipeline runs, when they receive an exception, that a given
> > feature is not supported by the targeted runner. I would like to suggest to translate the capability matrix
> > into the API with annotations for example, so that, while coding, the user could know that, for now,
> > a given feature is not supported on the runner he targets.
> > I know that the runner is only specified at pipeline runtime, and that adding code would be a leak
> > of runner implementation and against portability. So it could be just informative annotations
> > like @Experimental for example with no annotation processor.
> > WDYT?
> > Etienne
Re: [PROPOSAL] allow the users to anticipate the support of
features in the targeted runner.
Posted by Etienne Chauchot <ec...@apache.org>.
Hi guys,
To sum up what we said, I just opened this ticket:https://issues.apache.org/jira/browse/BEAM-5849
Etienne
Le jeudi 18 octobre 2018 à 12:44 +0200, Maximilian Michels a écrit :
> Plugins for IDEs would be amazing because they could provide feedback already during pipeline construction, but I'm
> not sure about the effort required to develop/maintain such plugins.
> Ultimately, Runners have to decide whether they can translate the given pipeline or not. So I'm leaning more towards
> an approach to make intermediate checking of the pipeline translation easier, e.g. by
> - providing the target Runner already during development- running check of the Runner alongside with the DirectRunner
> (which is typically used when developing pipelines)
> On 17.10.18 15:57, Etienne Chauchot wrote:
> Hey Max, Kenn,
> Thanks for your feedback !
> Yes the idea was to inform the user as soon as possible, ideally while coding the pipeline. It could be done with a
> IDE plugin (like checkstyle) that is configured with the targeted runner; that way the targeted runner conf is not
> part of the pipeline code in an annotation which would be against Beam portability philosophy. Such a plugin could
> color unsupported features while coding.
> Another way could be to implement it as a javadoc but it seems weak because not automatic enough.Another way could be
> to implement it as a validation plugin in the build system but IMHO it is already too late for the user.
> So, long story short, I'm more in favor of an IDE plugin or similar coding-time solution.
> BestEtienne
> Le mercredi 17 octobre 2018 à 12:11 +0200, Maximilian Michels a écrit :
> This is a good idea. It needs to be fleshed out how the capability of aRunner would be visible to the user (apart from
> the compatibility matrix).
> A dry-run feature would be useful, i.e. the user can run an inspectionon the pipeline to see if it contains any
> features which are notsupported by the Runner.
> On 17.10.18 00:03, Rui Wang wrote:Sounds like a good idea.
> Sounds like while coding, user gets a list to show if a feature issupported on different runners. User can check the
> list for the answer.Is my understanding correct? Will this approach become slow as number ofrunner grows? (it's just a
> question as I am not familiar the performanceof combination of a long list, annotation and IDE)
>
> -Rui
> On Sat, Oct 13, 2018 at 11:56 PM Reuven Lax <relax@google.com <ma...@google.com> <mailto:relax@google.com
> <ma...@google.com>>> wrote:
> Sounds like a good idea. I don't think it will work for all capabilities (e.g. some of them such as "exactly
> once" apply to all of the API surface), but useful for the ones that we can capture.
> On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot <echauchot@apache.org <ma...@apache.org> <mailto:
> echauchot@apache.org <ma...@apache.org>>> wrote:
> Hi guys, As part of our user experience improvement to attract new Beam users, I would like
> to suggest something:
> Today we only have the capability matrix to inform users about features support among runners. But,
> they might discover only when the pipeline runs, when they receive an exception, that a given feature
> is not supported by the targeted runner. I would like to suggest to translate the capability matrix
> into the API with annotations for example, so that, while coding, the user could know that, for now, a
> given feature is not supported on the runner he targets.
> I know that the runner is only specified at pipeline runtime, and that adding code would be a leak of
> runner implementation and against portability. So it could be just informative annotations
> like @Experimental for example with no annotation processor.
> WDYT?
> Etienne
Re: [PROPOSAL] allow the users to anticipate the support of features
in the targeted runner.
Posted by Maximilian Michels <mx...@apache.org>.
Plugins for IDEs would be amazing because they could provide feedback
already during pipeline construction, but I'm not sure about the effort
required to develop/maintain such plugins.
Ultimately, Runners have to decide whether they can translate the given
pipeline or not. So I'm leaning more towards an approach to make
intermediate checking of the pipeline translation easier, e.g. by
- providing the target Runner already during development
- running check of the Runner alongside with the DirectRunner (which is
typically used when developing pipelines)
On 17.10.18 15:57, Etienne Chauchot wrote:
> Hey Max, Kenn,
>
> Thanks for your feedback !
>
> Yes the idea was to inform the user as soon as possible, ideally while
> coding the pipeline. It could be done with a IDE plugin (like
> checkstyle) that is configured with the targeted runner; that way the
> targeted runner conf is not part of the pipeline code in an annotation
> which would be against Beam portability philosophy. Such a plugin could
> color unsupported features while coding.
>
> Another way could be to implement it as a javadoc but it seems weak
> because not automatic enough.
> Another way could be to implement it as a validation plugin in the build
> system but IMHO it is already too late for the user.
>
> So, long story short, I'm more in favor of an IDE plugin or similar
> coding-time solution.
>
> Best
> Etienne
>
> Le mercredi 17 octobre 2018 à 12:11 +0200, Maximilian Michels a écrit :
>> This is a good idea. It needs to be fleshed out how the capability of a
>> Runner would be visible to the user (apart from the compatibility matrix).
>>
>> A dry-run feature would be useful, i.e. the user can run an inspection
>> on the pipeline to see if it contains any features which are not
>> supported by the Runner.
>>
>> On 17.10.18 00:03, Rui Wang wrote:
>> Sounds like a good idea.
>>
>> Sounds like while coding, user gets a list to show if a feature is
>> supported on different runners. User can check the list for the answer.
>> Is my understanding correct? Will this approach become slow as number of
>> runner grows? (it's just a question as I am not familiar the performance
>> of combination of a long list, annotation and IDE)
>>
>>
>> -Rui
>>
>> On Sat, Oct 13, 2018 at 11:56 PM Reuven Lax <relax@google.com <ma...@google.com>
>> <mailto:relax@google.com <ma...@google.com>>> wrote:
>>
>> Sounds like a good idea. I don't think it will work for all
>> capabilities (e.g. some of them such as "exactly once" apply to all
>> of the API surface), but useful for the ones that we can capture.
>>
>> On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot
>> <echauchot@apache.org <ma...@apache.org> <mailto:echauchot@apache.org <ma...@apache.org>>> wrote:
>>
>> Hi guys,
>> As part of our user experience improvement to attract new Beam
>> users, I would like to suggest something:
>>
>> Today we only have the capability matrix to inform users about
>> features support among runners. But, they might discover only
>> when the pipeline runs, when they receive an exception, that a
>> given feature is not supported by the targeted runner.
>> I would like to suggest to translate the capability matrix into
>> the API with annotations for example, so that, while coding, the
>> user could know that, for now, a given feature is not supported
>> on the runner he targets.
>>
>> I know that the runner is only specified at pipeline runtime,
>> and that adding code would be a leak of runner implementation
>> and against portability. So it could be just informative
>> annotations like @Experimental for example with no annotation
>> processor.
>>
>> WDYT?
>>
>> Etienne
>>
Re: [PROPOSAL] allow the users to anticipate the support of
features in the targeted runner.
Posted by Etienne Chauchot <ec...@apache.org>.
Hey Max, Kenn,
Thanks for your feedback !
Yes the idea was to inform the user as soon as possible, ideally while coding the pipeline. It could be done with a IDE
plugin (like checkstyle) that is configured with the targeted runner; that way the targeted runner conf is not part of
the pipeline code in an annotation which would be against Beam portability philosophy. Such a plugin could color
unsupported features while coding.
Another way could be to implement it as a javadoc but it seems weak because not automatic enough.
Another way could be to implement it as a validation plugin in the build system but IMHO it is already too late for the
user.
So, long story short, I'm more in favor of an IDE plugin or similar coding-time solution.
Best
Etienne
Le mercredi 17 octobre 2018 à 12:11 +0200, Maximilian Michels a écrit :
> This is a good idea. It needs to be fleshed out how the capability of a
> Runner would be visible to the user (apart from the compatibility matrix).
>
> A dry-run feature would be useful, i.e. the user can run an inspection
> on the pipeline to see if it contains any features which are not
> supported by the Runner.
>
> On 17.10.18 00:03, Rui Wang wrote:
> Sounds like a good idea.
>
> Sounds like while coding, user gets a list to show if a feature is
> supported on different runners. User can check the list for the answer.
> Is my understanding correct? Will this approach become slow as number of
> runner grows? (it's just a question as I am not familiar the performance
> of combination of a long list, annotation and IDE)
>
>
> -Rui
>
> On Sat, Oct 13, 2018 at 11:56 PM Reuven Lax <relax@google.com
> <ma...@google.com>> wrote:
>
> Sounds like a good idea. I don't think it will work for all
> capabilities (e.g. some of them such as "exactly once" apply to all
> of the API surface), but useful for the ones that we can capture.
>
> On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot
> <echauchot@apache.org <ma...@apache.org>> wrote:
>
> Hi guys,
> As part of our user experience improvement to attract new Beam
> users, I would like to suggest something:
>
> Today we only have the capability matrix to inform users about
> features support among runners. But, they might discover only
> when the pipeline runs, when they receive an exception, that a
> given feature is not supported by the targeted runner.
> I would like to suggest to translate the capability matrix into
> the API with annotations for example, so that, while coding, the
> user could know that, for now, a given feature is not supported
> on the runner he targets.
>
> I know that the runner is only specified at pipeline runtime,
> and that adding code would be a leak of runner implementation
> and against portability. So it could be just informative
> annotations like @Experimental for example with no annotation
> processor.
>
> WDYT?
>
> Etienne
>
Re: [PROPOSAL] allow the users to anticipate the support of features
in the targeted runner.
Posted by Maximilian Michels <mx...@apache.org>.
This is a good idea. It needs to be fleshed out how the capability of a
Runner would be visible to the user (apart from the compatibility matrix).
A dry-run feature would be useful, i.e. the user can run an inspection
on the pipeline to see if it contains any features which are not
supported by the Runner.
On 17.10.18 00:03, Rui Wang wrote:
> Sounds like a good idea.
>
> Sounds like while coding, user gets a list to show if a feature is
> supported on different runners. User can check the list for the answer.
> Is my understanding correct? Will this approach become slow as number of
> runner grows? (it's just a question as I am not familiar the performance
> of combination of a long list, annotation and IDE)
>
>
> -Rui
>
> On Sat, Oct 13, 2018 at 11:56 PM Reuven Lax <relax@google.com
> <ma...@google.com>> wrote:
>
> Sounds like a good idea. I don't think it will work for all
> capabilities (e.g. some of them such as "exactly once" apply to all
> of the API surface), but useful for the ones that we can capture.
>
> On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot
> <echauchot@apache.org <ma...@apache.org>> wrote:
>
> Hi guys,
> As part of our user experience improvement to attract new Beam
> users, I would like to suggest something:
>
> Today we only have the capability matrix to inform users about
> features support among runners. But, they might discover only
> when the pipeline runs, when they receive an exception, that a
> given feature is not supported by the targeted runner.
> I would like to suggest to translate the capability matrix into
> the API with annotations for example, so that, while coding, the
> user could know that, for now, a given feature is not supported
> on the runner he targets.
>
> I know that the runner is only specified at pipeline runtime,
> and that adding code would be a leak of runner implementation
> and against portability. So it could be just informative
> annotations like @Experimental for example with no annotation
> processor.
>
> WDYT?
>
> Etienne
>
Re: [PROPOSAL] allow the users to anticipate the support of
features in the targeted runner.
Posted by Etienne Chauchot <ec...@apache.org>.
Yes indeed, your understanding is correct. This is what I had in mind.
PS: I have no idea on perf right now.
Etienne
Le mardi 16 octobre 2018 à 15:03 -0700, Rui Wang a écrit :
> Sounds like a good idea.
> Sounds like while coding, user gets a list to show if a feature is supported on different runners. User can check the
> list for the answer. Is my understanding correct? Will this approach become slow as number of runner grows? (it's just
> a question as I am not familiar the performance of combination of a long list, annotation and IDE)
>
>
> -Rui
> On Sat, Oct 13, 2018 at 11:56 PM Reuven Lax <re...@google.com> wrote:
> > Sounds like a good idea. I don't think it will work for all capabilities (e.g. some of them such as "exactly once"
> > apply to all of the API surface), but useful for the ones that we can capture.
> >
> > On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot <ec...@apache.org> wrote:
> > > Hi guys,
> > > As part of our user experience improvement to attract new Beam users, I would like to suggest something:
> > >
> > > Today we only have the capability matrix to inform users about features support among runners. But, they might
> > > discover only when the pipeline runs, when they receive an exception, that a given feature is not supported by the
> > > targeted runner.
> > > I would like to suggest to translate the capability matrix into the API with annotations for example, so that,
> > > while coding, the user could know that, for now, a given feature is not supported on the runner he targets.
> > >
> > > I know that the runner is only specified at pipeline runtime, and that adding code would be a leak of runner
> > > implementation and against portability. So it could be just informative annotations like @Experimental for example
> > > with no annotation processor.
> > >
> > > WDYT?
> > >
> > > Etienne
Re: [PROPOSAL] allow the users to anticipate the support of features
in the targeted runner.
Posted by Rui Wang <ru...@google.com>.
Sounds like a good idea.
Sounds like while coding, user gets a list to show if a feature is
supported on different runners. User can check the list for the answer. Is
my understanding correct? Will this approach become slow as number of
runner grows? (it's just a question as I am not familiar the performance of
combination of a long list, annotation and IDE)
-Rui
On Sat, Oct 13, 2018 at 11:56 PM Reuven Lax <re...@google.com> wrote:
> Sounds like a good idea. I don't think it will work for all capabilities
> (e.g. some of them such as "exactly once" apply to all of the API surface),
> but useful for the ones that we can capture.
>
> On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot <ec...@apache.org>
> wrote:
>
>> Hi guys,
>> As part of our user experience improvement to attract new Beam users, I
>> would like to suggest something:
>>
>> Today we only have the capability matrix to inform users about features
>> support among runners. But, they might discover only when the pipeline
>> runs, when they receive an exception, that a given feature is not supported
>> by the targeted runner.
>> I would like to suggest to translate the capability matrix into the API
>> with annotations for example, so that, while coding, the user could know
>> that, for now, a given feature is not supported on the runner he targets.
>>
>> I know that the runner is only specified at pipeline runtime, and that
>> adding code would be a leak of runner implementation and against
>> portability. So it could be just informative annotations like @Experimental
>> for example with no annotation processor.
>>
>> WDYT?
>>
>> Etienne
>>
>
Re: [PROPOSAL] allow the users to anticipate the support of features
in the targeted runner.
Posted by Reuven Lax <re...@google.com>.
Sounds like a good idea. I don't think it will work for all capabilities
(e.g. some of them such as "exactly once" apply to all of the API surface),
but useful for the ones that we can capture.
On Thu, Oct 4, 2018 at 2:43 AM Etienne Chauchot <ec...@apache.org>
wrote:
> Hi guys,
> As part of our user experience improvement to attract new Beam users, I
> would like to suggest something:
>
> Today we only have the capability matrix to inform users about features
> support among runners. But, they might discover only when the pipeline
> runs, when they receive an exception, that a given feature is not supported
> by the targeted runner.
> I would like to suggest to translate the capability matrix into the API
> with annotations for example, so that, while coding, the user could know
> that, for now, a given feature is not supported on the runner he targets.
>
> I know that the runner is only specified at pipeline runtime, and that
> adding code would be a leak of runner implementation and against
> portability. So it could be just informative annotations like @Experimental
> for example with no annotation processor.
>
> WDYT?
>
> Etienne
>