You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <Ja...@polidea.com> on 2019/07/23 16:34:01 UTC
[VOTE] Changes in import paths
Hello everyone,
This email is calling a vote on the changes in import paths. It's been
discussed in
https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
The vote will last for at least 1 week (July 30th 6pm CEST), and at least
three +1 (binding) votes have been cast.
The proposal to vote is based on the document from Kamil
<https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#>
The proposed solution is:
- *Case 1: B: Contrib vs not: we move all that are "well" tested and
rename contrib to "incubating" or similar.*
- *Case 2: B: Airflow.operators.foo_operator.FooOperator could
become airflow.operators.foo.FooOperator*
- *Case 3: C:
airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator could
become airflow.gcp.operators.bigtable.BigTableOperator*
- *Case 4: B: no namespace introduction*
- *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class names*
- *Case 6: We will treat isolated cases on case-by-case (and my team can
do the job of GCP-related operators)*
This is my (binding) +1 vote.
Best regards,
J.
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Changes in import paths
Posted by Jarek Potiuk <Ja...@polidea.com>.
Hey Fokko,
Just to alleviate your concerns a bit - we are going to make it backwards
compatible for a start. And we will just start with GCP operators together
with Kamil and Tomek, to show how it works (and that it's not that
difficult in fact - maybe even automate it it if needed - as we usually
do). I think with the recent changes on Pylint / Py3 it will be harder and
harder to cherry-pick to v1-10 anyway, so it looks like it's a good time to
make some bolder moves.
J.
On Mon, Aug 5, 2019 at 8:35 PM Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> Hi Jarek,
>
> No worries. I must admit that I've lost track a bit when the case 3+4 got
> combined. Maybe for the future better to let the current vote run and maybe
> reinitiate a new vote if there are new options.
>
> Currently, it is quite easy to think of a few obvious groups, but it is so
> hard to keep this consistent in the future. I'm not saying that one giant
> operator package is ideal, but I think it is easy to maintain. Two/three
> years ago the guys for Databricks wrote all the operators for their
> platform, and it was in such a state that it was eligible to move to the
> core (non-contrib), similar with GCP now. But it is hard to actually do it.
> Putting everything into one place makes this from my perspective simpler.
> If Ali-Cloud comes with some operators, should we put them in a separate
> package right away? Prefixing would be a nice compromise for me.
>
> If I'm the only one against adding the additional namespaces, then I don't
> want to stop the rest of the community of making this happen. I don't like
> to veto stuff, but I think we should think this through properly :-)
>
> Cheers, Fokko
>
>
> Op ma 5 aug. 2019 om 17:55 schreef Jarek Potiuk <Jarek.Potiuk@polidea.com
> >:
>
> > Hey Fokko,
> >
> > Really sorry, to miss this . I see now that you have not voted because
> you
> > thought it will be transferred automatically -it was thought as not and
> > additional option but combined replacement for (3+4) cases.
> >
> > This was after I realised that there was a veto for namespaces from Ash
> and
> > I realised from it that namespaces were a bit confusing and overlapping
> > with grouping per-cloud-provider (see the voting "chapter
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> > >").
> > I re-fined the choice back then - whether to not group them (A) or
> whether
> > to group first by type of entity or first by cloud provider (B-E in
> various
> > ways). Maybe I was not very clear in my message - apologies.
> >
> > I now re-added your vote and also Ben's vote who also voted on leaving
> the
> > prefix and not grouping operators (Ben please correct me if I was wrong).
> > Still the result is 4:2 (binding), 6:2 (including non-binding) in favour
> of
> > grouping per cloud provider rather than using prefixes. And no vetoes.
> >
> > Re - the point of which operators should be grouped. From the
> discussions -
> > I do not think it's easy to do a general and always applicable rule. But
> I
> > believe we agreed that we group the "obvious" choices - like "gcp",
> > "azure", "aws". It's for pure "Action" type operators - but still we
> leave
> > the cross-platform "transfer" ones or non-obvious ones in the old place
> > (which we keep as bag of "everything else"). Personally - I am all for
> > grouping the qubole/databricks operators in packages same as other cloud
> > providers. But with the "softness" of that rule this might (and should)
> be
> > an individual decision of people who maintain it. And it is quite OK -
> not
> > everything has to be decided upfront. I don't think we have a "hard" rule
> > now that covers all the cases and binds us. It would be very complex one
> if
> > we have it (and we can defer introducing it for later if needed
> especially
> > if it requires more than just renaming/moving but also architectural
> > changes - as Ash proposed for the transfer operators). But at least we
> can
> > decrease an entropy by grouping the obvious cases, and we can move
> forward
> > leaving the Airflow world a bit better, one step at a time.
> >
> > J.
> >
> > On Mon, Aug 5, 2019 at 3:58 PM Driesprong, Fokko <fo...@driesprong.frl>
> > wrote:
> >
> > > Hi Jarek,
> > >
> > > Just wondering how the vote is unanimous. On case 3 I've voted for B. I
> > > still believe introducing namespaces is hard to maintain, as I
> explained
> > > earlier: Mostly because it isn't as trivial to decide when it gets its
> > own
> > > package or not. Besides the cloud providers, there are companies like
> > > Databricks and Qubole which might also end up with their own package
> > > because their integration is getting richer over time. Moving this is a
> > bit
> > > painful because we need to deprecate the old path and move everything
> > which
> > > introduces noise into version control etc.
> > >
> > > Adding an additional option to the list, does not invalidate the older
> > > options, right?
> > >
> > > Cheers, Fokko
> > >
> > >
> > > Op ma 5 aug. 2019 om 15:43 schreef Jarek Potiuk <
> > Jarek.Potiuk@polidea.com
> > > >:
> > >
> > > > Seems that after this months of discussion we got an unanimous voting
> > > > result. I summarised it in the AIP-21
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > > > >
> > > > and
> > > > changed its status to "Accepted". Thanks for all your feedback! We
> will
> > > > start soon moving GCP operators following those rules and we will
> > resolve
> > > > any initial problems with those - then we might provide some
> additional
> > > > learnings and see if we can easily (probably semi-automatically) move
> > all
> > > > other operators.
> > > >
> > > > Case 1
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#1airflow.contrib.{resources}
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%231airflow.contrib.%7Bresources%7D>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%231airflow.contrib.%7Bresources%7D
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%231airflow.contrib.%7Bresources%7D
> > >
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%231airflow.contrib.%7Bresources%7D
> > > >
> > > > >
> > > >
> > > > What to do with the "contrib" folder
> > > >
> > > > Case 2
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#2git*_{operator/sensor}{/s}.py
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%232git*_%7Boperator/sensor%7D%7B/s%7D.py>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%232git*_%7Boperator/sensor%7D%7B/s%7D.py
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%232git*_%7Boperator/sensor%7D%7B/s%7D.py
> > >
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%232git*_%7Boperator/sensor%7D%7B/s%7D.py
> > > >
> > > > >
> > > >
> > > > Drop modules *_operator suffix
> > > >
> > > > Case 3
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#3{aws/azure/gcp}_*
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%233%7Baws/azure/gcp%7D_*>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%233%7Baws/azure/gcp%7D_*
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%233%7Baws/azure/gcp%7D_*
> > >
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%233%7Baws/azure/gcp%7D_*
> > > >
> > > > >
> > > > + Case 4
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#4Separatenamespaceforresources
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%234Separatenamespaceforresources>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%234Separatenamespaceforresources
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%234Separatenamespaceforresources
> > >
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%234Separatenamespaceforresources
> > > >
> > > > >
> > > >
> > > > Grouping Cloud Providers operators/sensors/hooks
> > > >
> > > > Case 5
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#5*Operator
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%235*Operator>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%235*Operator
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%235*Operator
> > >
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%235*Operator
> > > >
> > > > >
> > > >
> > > > *Operator *Sensor *Hook in class name
> > > >
> > > > Case 6
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#6Otherisolatedcases
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%236Otherisolatedcases>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%236Otherisolatedcases
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%236Otherisolatedcases
> > >
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%236Otherisolatedcases
> > > >
> > > > >
> > > >
> > > > Isolated cases
> > > >
> > > > Case 7
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#7
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%237>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%237
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%237
> > >
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%237
> > > >
> > > > >
> > > >
> > > > Deprecation method
> > > >
> > > > *A.* Move everything "contrib" → "airflow"
> > > >
> > > > *A.* Drop the suffix.
> > > >
> > > > Example:
> > > >
> > > > - *airflow.operator.gcp_bigtable_operator.py*
> > > > becomes *airflow.operator.gcp_bigtable.py
> > > > <http://airflow.operator.gcp_bigtable.py>*.
> > > >
> > > > *D. * Group operators/sensors/hooks in
> > > > *airflow/<PROVIDER>*/operators(sensors,
> > > > hooks). Non-cloud provider ones are moved to
> > > > airflow/operators(sensors/hooks).
> > > > *Drop the prefix.*
> > > >
> > > > -
> > > > *airflow/contrib/operators/sns_publish_operator.py
> > > > becomes airflow/aws/operators/**sns_publish_operator.py*
> > > >
> > > >
> > > > - *airflow/contrib/operators/dataproc_operator.py*
> > > > becomes *airflow/gcp/operators/dataproc_operator.py*
> > > >
> > > >
> > > >
> > > > -
> > > > *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> > *airflow/*
> > > > *gcp/operators/bigtable_operator.py*
> > > >
> > > >
> > > >
> > > > -
> > > > *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > > > *operators/ssh_operator.py*
> > > >
> > > > *B.* No change - keep Operator/Sensor suffix in class name.
> > > >
> > > > *A.* Make individual decisions of renames for operators that do not
> > > follow
> > > > common conventions used for other operators.
> > > >
> > > > Consistency trumps compatibility.
> > > >
> > > > Examples:
> > > >
> > > > *DataProcHadoopOperator*
> > > >
> > > > renamed to:
> > > >
> > > > *DataprocHadoopOperator*
> > > > *A.* Native python method (with better IDE support and more flexible
> > > but a
> > > > bit more verbose)
> > > >
> > > > J.
> > > >
> > > > On Wed, Jul 31, 2019 at 10:30 AM Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>
> > > > wrote:
> > > >
> > > > > Yep. Agree with Ash on it. There are a number of 'action' operators
> > > > > specific for cloud providers and these should be our target. The
> > > transfer
> > > > > ones require another AIP (A lot of that already discussed in AIP-8
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
> > > > > ).
> > > > >
> > > > > J.
> > > > >
> > > > > Principal Software Engineer
> > > > > Phone: +48660796129
> > > > >
> > > > > śr., 31 lip 2019, 10:01 użytkownik Ash Berlin-Taylor <
> ash@apache.org
> > >
> > > > > napisał:
> > > > >
> > > > >> This is a good idea for now. I'm also not overly concerned about
> > these
> > > > >> few non-cloud examples - FTPtoS3Operator can stay where it is and
> > > > doesn't
> > > > >> have to move under 'aws.' to my mind.
> > > > >>
> > > > >> Longer term I'd like to go back to making the
> > > "transfer/copy/transform"
> > > > >> operators "composable" so that we can have a single
> > DataCopyOperator,
> > > > and
> > > > >> give it a source and destination, and it uses hooks to do the
> work.
> > > > This is
> > > > >> not a small undertaking though to do well and efficiently though.
> > > > >>
> > > > >> -ash
> > > > >>
> > > > >> > On 30 Jul 2019, at 20:54, Tomasz Urbaszek <
> > > > tomasz.urbaszek@polidea.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > Maybe we can put all those AtoB operators under one name like
> > > > >> “transfer”,
> > > > >> > then it would be easier to look for such operator?
> > > > >> >
> > > > >> > Best,
> > > > >> > Tomek
> > > > >> >
> > > > >> > On Tue, 30 Jul 2019 at 21:39, Chen Tong <ci...@gmail.com>
> wrote:
> > > > >> >
> > > > >> >> Daniel mentioned a good point. Such composed operator may also
> > > > involves
> > > > >> >> both cloud and non-cloud provider saying FTPtoS3Operator.
> Should
> > it
> > > > in
> > > > >> AWS
> > > > >> >> folder?
> > > > >> >>
> > > > >> >> Also, saying in the future, another cloud provider is growing
> > large
> > > > >> enough.
> > > > >> >> Will we rename all plugins related to this provider? What's the
> > > > >> criteria we
> > > > >> >> say it's big enough? And option D gives an impression like
> these
> > > > tools
> > > > >> may
> > > > >> >> be maintained and supported by the Cloud provider. I am not
> sure
> > if
> > > > it
> > > > >> will
> > > > >> >> be a truth or not.
> > > > >> >>
> > > > >> >> Best,
> > > > >> >>
> > > > >> >>
> > > > >> >> On Tue, Jul 30, 2019 at 11:18 AM Daniel Standish <
> > > > dpstandish@gmail.com
> > > > >> >
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >>> One wrinkle with case 3+4 option D is inter-provider
> operators.
> > > > >> Mainly
> > > > >> >>> it's storage I think e.g. XToS3Operator or XToGCSOperator
> where
> > > the
> > > > X
> > > > >> is
> > > > >> >> a
> > > > >> >>> service some different provider.
> > > > >> >>>
> > > > >> >>> Maybe the rule should be to locate the operator according to
> the
> > > > first
> > > > >> >>> provider referenced. So e.g. s3_to_gcs_transfer_operator.py
> > would
> > > > go
> > > > >> to
> > > > >> >>> aws.
> > > > >> >>>
> > > > >> >>>
> > > > >> >>> On Tue, Jul 30, 2019 at 6:21 AM Kamil Breguła <
> > > > >> kamil.bregula@polidea.com
> > > > >> >>>
> > > > >> >>> wrote:
> > > > >> >>>
> > > > >> >>>> Yes. All changes will be backwards compatible. In the case of
> > > using
> > > > >> >>>> the old path, a message containing a proposal for change will
> > be
> > > > >> >>>> reported to the user.
> > > > >> >>>>
> > > > >> >>>> I prepared an example of how to change the name of a class
> in a
> > > > case
> > > > >> >>>> with the use of a native solution.
> > > > >> >>>> Source code:
> > > > >> >>>>
> > > > >> >>
> > > > >>
> > > >
> > https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename
> > > > >> >>>> Preview from IDE: https://imgur.com/a/45NxS5W
> > > > >> >>>>
> > > > >> >>>> On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <
> > > ash@apache.org>
> > > > >> >>> wrote:
> > > > >> >>>>>
> > > > >> >>>>> Just cos I'm not sure it's _explicitly_ stated, but all of
> the
> > > > moves
> > > > >> >>>> will have a deprecation of the old name right?
> > > > >> >>>>>
> > > > >> >>>>> 3+4 case D gets my vote too.
> > > > >> >>>>>
> > > > >> >>>>> -a
> > > > >> >>>>>
> > > > >> >>>>>> On 30 Jul 2019, at 09:58, Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com
> > > > >
> > > > >> >>>> wrote:
> > > > >> >>>>>>
> > > > >> >>>>>> I went ahead and updated the page (just to speed it up) as
> I
> > > > think
> > > > >> >> it
> > > > >> >>>>>> really makes sense to join those two cases (and I do not
> see
> > > any
> > > > >> >>>> drawbacks
> > > > >> >>>>>> - I think the options we have cover all possible
> approaches)
> > > and
> > > > we
> > > > >> >>> can
> > > > >> >>>>>> always go back if we need to.
> > > > >> >>>>>>
> > > > >> >>>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > > > >> >>>>>>
> > > > >> >>>>>> My vote is *D*.
> > > > >> >>>>>>
> > > > >> >>>>>> - airflow/contrib/operators/gcp_bigtable_operator.py →
> > > > >> >>> airflow/gcp
> > > > >> >>>>>> /operators/bigtable_operator.py
> > > > >> >>>>>> - airflow/contrib/operators/ssh_operator.py ->
> > > > >> >>>>>> airflow/operators/ssh_operator.py
> > > > >> >>>>>>
> > > > >> >>>>>> I updated the page with my vote.
> > > > >> >>>>>> I propose that we vote till Friday on that one (all the
> rest
> > is
> > > > >> >>> already
> > > > >> >>>>>> agreed).
> > > > >> >>>>>>
> > > > >> >>>>>> J.
> > > > >> >>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>> On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <
> > > > >> >>> Jarek.Potiuk@polidea.com
> > > > >> >>>>>
> > > > >> >>>>>> wrote:
> > > > >> >>>>>>
> > > > >> >>>>>>> I think almost everyone voted and we have almost perfect
> > > > >> >> consensus.
> > > > >> >>>> We all
> > > > >> >>>>>>> agree amongst other on moving all operators out of contrib
> > > > >> >> (Great!).
> > > > >> >>>>>>>
> > > > >> >>>>>>> The only doubts are for *Case 3* (Cloud provider prefix)
> and
> > > > *Case
> > > > >> >>> 4*
> > > > >> >>>>>>> (Using Namespaces).
> > > > >> >>>>>>> I think there was actually an overlap between those two.
> > Also
> > > > >> >> Ash's
> > > > >> >>>>>>> comment/veto on that is quite clear.
> > > > >> >>>>>>> So I have final (I hope!) proposal to join those two cases
> > and
> > > > >> >> have
> > > > >> >>>> one
> > > > >> >>>>>>> voting (till Friday) only on that.
> > > > >> >>>>>>>
> > > > >> >>>>>>> I will update the doc if you all agree with me that makes
> > more
> > > > >> >> sense
> > > > >> >>>> as
> > > > >> >>>>>>> single case to vote on:
> > > > >> >>>>>>>
> > > > >> >>>>>>> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
> > > > >> >>>>>>>
> > > > >> >>>>>>> *Option A*. Keep operators/sensors/hooks in
> > > > >> >>> airflow/operators(sensors,
> > > > >> >>>>>>> hooks) and keep/add prefixes in file names.
> > > > >> >>>>>>>
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> becomes
> > > > >> >>>>>>> airflow/operators/**aws_sns_publish_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > > > >> >>>>>>> becomes *airflow/operators/gcp_dataproc_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes
> > > *airflow/*
> > > > >> >>>>>>> *hooks/gcp_bigtable_hook.py*
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> > > *airflow/*
> > > > >> >>>>>>> *operators/ssh_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> *Option B*. Group operators/sensors/hooks in
> > > > >> >>>> airflow/operators(sensors,
> > > > >> >>>>>>> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
> > > > >> >>>>>>> airflow/operators(sensors/hooks), Drop the prefix.
> > > > >> >>>>>>>
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> > > > >> >>>>>>> becomes airflow/operators/**aws/sns_publish_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > > > >> >>>>>>> becomes *airflow/operators/gcp/dataproc_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py
> > *becomes
> > > > >> >>>> *airflow/*
> > > > >> >>>>>>> *operators/gcp/bigtable_operator.py*
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> > > *airflow/*
> > > > >> >>>>>>> *operators/ssh_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> *Option C*. Group operators/sensors/hooks in
> > > > >> >>>> airflow/operators(sensors,
> > > > >> >>>>>>> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
> > > > >> >>>>>>> airflow/operators(sensors/hooks), Keep the prefix.
> > > > >> >>>>>>>
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> > > > >> >>>>>>> becomes
> > airflow/operators/**aws/aws_sns_publish_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > > > >> >>>>>>> becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py
> > *becomes
> > > > >> >>>> *airflow/*
> > > > >> >>>>>>> *operators/gcp/gcp_bigtable_operator.py*
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> > > *airflow/*
> > > > >> >>>>>>> *operators/ssh_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> *Option D.* Group operators/sensors/hooks in
> > > > >> >>>> *airflow/<PROVIDER>*/operators(sensors,
> > > > >> >>>>>>> hooks). Non-cloud provider ones are moved to
> > > > >> >>>>>>> airflow/operators(sensors/hooks). Drop the prefix.
> > > > >> >>>>>>>
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> > > > >> >>>>>>> becomes airflow/aws/operators/**sns_publish_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > > > >> >>>>>>> becomes *airflow/gcp/operators/dataproc_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py
> > *becomes
> > > > >> >>>> *airflow/*
> > > > >> >>>>>>> *gcp/operators/bigtable_operator.py*
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> > > *airflow/*
> > > > >> >>>>>>> *operators/ssh_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> *Option E.* Group operators/sensors/hooks in
> > > > >> >>>> *airflow/<PROVIDER>*/operators(sensors,
> > > > >> >>>>>>> hooks). Non-cloud provider ones are moved to
> > > > >> >>>>>>> airflow/operators(sensors/hooks).* Keep the prefix*.
> > > > >> >>>>>>>
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> > > > >> >>>>>>> becomes
> > airflow/aws/operators/aws_**sns_publish_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > > > >> >>>>>>> becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py
> > *becomes
> > > > >> >>>> *airflow/*
> > > > >> >>>>>>> *gcp/operators/gcp_bigtable_operator.py*
> > > > >> >>>>>>> -
> > > > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> > > *airflow/*
> > > > >> >>>>>>> *operators/ssh_operator.py*
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> Shall I proceed with updating the page ?
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> J.
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <
> > > > kaxilnaik@gmail.com>
> > > > >> >>>> wrote:
> > > > >> >>>>>>>
> > > > >> >>>>>>>> Thanks Jarek, adding my vote.
> > > > >> >>>>>>>>
> > > > >> >>>>>>>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <
> > > > >> >> ash@apache.org>
> > > > >> >>>> wrote:
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>> Thanks Jarek, much clearer. I have voted there too.
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> -ash
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko
> > > > >> >> <fokko@driesprong.frl
> > > > >> >>>>
> > > > >> >>>>>>>>> wrote:
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> Thanks Jarek, the Wiki works much better for me. I've
> > added
> > > > my
> > > > >> >>> vote
> > > > >> >>>>>>>> there
> > > > >> >>>>>>>>>> as well.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> You've convinced me on the second case. I've changed my
> > > vote
> > > > on
> > > > >> >>>> Case 2
> > > > >> >>>>>>>>> from
> > > > >> >>>>>>>>>> A to B :-)
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> Maybe we should leave the vote open a bit longer since
> it
> > > > >> >>> involves
> > > > >> >>>>>>>> quite
> > > > >> >>>>>>>>>> some reading, and I would like to get some proper
> > feedback
> > > > and
> > > > >> >>>>>>>> opinions
> > > > >> >>>>>>>>> on
> > > > >> >>>>>>>>>> this. Plus I only see two votes right now. If we don't
> > > think
> > > > >> >> this
> > > > >> >>>>>>>>> through,
> > > > >> >>>>>>>>>> it might be that we're having this vote again in the
> near
> > > > >> >> future
> > > > >> >>>> :-)
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> Cheers, Fokko
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
> > > > >> >>>>>>>>> Jarek.Potiuk@polidea.com>:
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>>> I updated the AIP-21 proposal in the way that should
> > > answer
> > > > >> >> all
> > > > >> >>>> the
> > > > >> >>>>>>>>>>> concerns in this thread.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> NOTE! There is an action for all of those who are
> > > > interested:
> > > > >> >>>> Please
> > > > >> >>>>>>>>>>> fill-in your voting in the voting table till Tuesday
> > 30th
> > > of
> > > > >> >>> July
> > > > >> >>>> 6pm
> > > > >> >>>>>>>>> CEST
> > > > >> >>>>>>>>>>> (5pm BST, 9 am PST).
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> I created a summary of the proposal
> > > > >> >>>>>>>>>>> <
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>> in table form - which contains also examples for each
> > > case.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> I added a voting table
> > > > >> >>>>>>>>>>> <
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>> where you should add your votes:
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> I propose this voting mechanism:
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm
> > CEST
> > > > (5
> > > > >> >>> pm
> > > > >> >>>>>>>> BST,
> > > > >> >>>>>>>>> 9am
> > > > >> >>>>>>>>>>> PST) .
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> After the choice, the final consistent set of choices
> > will
> > > > be
> > > > >> >>>>>>>> announced
> > > > >> >>>>>>>>>>> (taking into account majority of binding vote, also
> > > > including
> > > > >> >>>>>>>> potential
> > > > >> >>>>>>>>>>> vetos and consistency between the choices. Non-binding
> > > votes
> > > > >> >>> will
> > > > >> >>>> be
> > > > >> >>>>>>>>> taken
> > > > >> >>>>>>>>>>> into account in case there is a draw. The final set of
> > > > choices
> > > > >> >>>> will
> > > > >> >>>>>>>> be
> > > > >> >>>>>>>>>>> announced at devlist thread
> > > > >> >>>>>>>>>>> <
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>> after
> > > > >> >>>>>>>>>>> the voting completes.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> Unless there is a veto raised to the final proposal,
> the
> > > > final
> > > > >> >>>>>>>> proposal
> > > > >> >>>>>>>>> is
> > > > >> >>>>>>>>>>> accepted by Lazy Consensus
> > > > >> >>>>>>>>>>> <
> > > https://community.apache.org/committers/lazyConsensus.html
> > > > >
> > > > >> >>> on
> > > > >> >>>>>>>>> *Friday*
> > > > >> >>>>>>>>>>> 02
> > > > >> >>>>>>>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> Please let me know if what I proposed can be further
> > > > improved
> > > > >> >> to
> > > > >> >>>>>>>> avoid
> > > > >> >>>>>>>>>>> chaos.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> J.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
> > > > >> >>>>>>>> Jarek.Potiuk@polidea.com
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>> Ok. I see that indeed voting could have been
> organised
> > a
> > > > bit
> > > > >> >>>> better
> > > > >> >>>>>>>> -
> > > > >> >>>>>>>>> dev
> > > > >> >>>>>>>>>>>> list does not seem to cope well with such a complex
> > case
> > > > :).
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> As Kamil noticed - we already have a formal AIP-21
> > > > >> >>>>>>>>>>>> <
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> in the Wiki - which I should have mentioned before. I
> > > have
> > > > >> >> not
> > > > >> >>>> much
> > > > >> >>>>>>>>> time
> > > > >> >>>>>>>>>>>> today - but tomorrow I will try to summarise the
> > options
> > > -
> > > > >> >>> based
> > > > >> >>>> on
> > > > >> >>>>>>>>> some
> > > > >> >>>>>>>>>>>> real code examples (to make it easier to digest). I
> > think
> > > > the
> > > > >> >>>> best
> > > > >> >>>>>>>>>>> approach
> > > > >> >>>>>>>>>>>> will be to make a voting matrix in the AIP-21 Wiki
> page
> > > > where
> > > > >> >>>> people
> > > > >> >>>>>>>>> will
> > > > >> >>>>>>>>>>>> be able to state their preferences for each case (by
> > > > adding a
> > > > >> >>>> row to
> > > > >> >>>>>>>>> the
> > > > >> >>>>>>>>>>>> table). I think that might provide much better
> clarity
> > > and
> > > > a
> > > > >> >>>> single
> > > > >> >>>>>>>>> place
> > > > >> >>>>>>>>>>>> which will show the current aggregated state of
> voting.
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> However - as Ash also stated - some of the options
> are
> > > > >> >>>>>>>> inter-dependent
> > > > >> >>>>>>>>> so
> > > > >> >>>>>>>>>>>> at the end we will have to adjust the results in case
> > > they
> > > > >> >> are
> > > > >> >>>> not
> > > > >> >>>>>>>>>>>> consistent - and make final voting on aggregated set
> > of
> > > > >> >> cases
> > > > >> >>> -
> > > > >> >>>>>>>> but I
> > > > >> >>>>>>>>>>>> think in this case following "lasy consensus" - i,e.
> if
> > > > noone
> > > > >> >>>>>>>> objects
> > > > >> >>>>>>>>> to
> > > > >> >>>>>>>>>>>> final set of cases, we will proceed with it..
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> Do you think that will work better ?
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> J.
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> Principal Software Engineer
> > > > >> >>>>>>>>>>>> Phone: +48660796129
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> > > > >> >>>>>>>>>>>> kamil.bregula@polidea.com> napisał:
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> Document is also available as AIP on wiki:
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
> > > > >> >>>>>>>> <fokko@driesprong.frl
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>> I agree with all, this is a bit chaotic. For the
> sake
> > > of
> > > > >> >>>> clarity,
> > > > >> >>>>>>>> I
> > > > >> >>>>>>>>>>>>> would
> > > > >> >>>>>>>>>>>>>> suggest moving the Google Doc to the Wiki as an
> AIP.
> > > > >> >> Create a
> > > > >> >>>>>>>>>>> structural
> > > > >> >>>>>>>>>>>>>> code improvement AIP and then and add a matrix of
> > > > >> >> users/cases
> > > > >> >>>>>>>> where
> > > > >> >>>>>>>>>>>>>> everybody can add his vote per case. This gives
> also
> > > the
> > > > >> >>>>>>>> opportunity
> > > > >> >>>>>>>>>>> for
> > > > >> >>>>>>>>>>>>>> changing your vote on a specific case. WDYT?
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>> Cheers, Fokko
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <
> > > > >> >>>> cixuuz@gmail.com>:
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>> I agree with Ash. It is clear to vote on each case
> > > > rather
> > > > >> >>> than
> > > > >> >>>>>>>> all
> > > > >> >>>>>>>>>>>>>>> together.
> > > > >> >>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor
> <
> > > > >> >>>>>>>> ash@apache.org>
> > > > >> >>>>>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>> A number of proposals overlap or add on to each
> > > other,
> > > > >> >> so I
> > > > >> >>>>>>>> think
> > > > >> >>>>>>>>>>>>> (?) a
> > > > >> >>>>>>>>>>>>>>>> single vote makes sense.
> > > > >> >>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>> To be clear, my vote is -1 until it's clear
> exactly
> > > > what
> > > > >> >> we
> > > > >> >>>> are
> > > > >> >>>>>>>>>>>>> voting
> > > > >> >>>>>>>>>>>>>> on.
> > > > >> >>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <
> > > > >> >>>> kaxilnaik@gmail.com>
> > > > >> >>>>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>>>>> I agree with Ash and instead of voting on all 7
> > > Cases
> > > > >> >>>> together,
> > > > >> >>>>>>>>>>> how
> > > > >> >>>>>>>>>>>>>>>>> about
> > > > >> >>>>>>>>>>>>>>>>> separate vote emails for all the cases? Each
> email
> > > can
> > > > >> >>> have
> > > > >> >>>> the
> > > > >> >>>>>>>>>>>>>>>>> description
> > > > >> >>>>>>>>>>>>>>>>> copied over from the doc.
> > > > >> >>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>> It would probably be a bit easier to track a
> > > decision
> > > > in
> > > > >> >>> the
> > > > >> >>>>>>>>>>> future
> > > > >> >>>>>>>>>>>>> as
> > > > >> >>>>>>>>>>>>>>>>> well.
> > > > >> >>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
> > > > >> >>>>>>>> Jarek.Potiuk@polidea.com>
> > > > >> >>>>>>>>>>>>>>>>> @Driesprong,
> > > > >> >>>>>>>>>>>>>>>>> Fokko <fo...@driesprong.frl>
> > > > >> >>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
> > > > >> >>>>>>>> kaxilnaik@gmail.com>
> > > > >> >>>>>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} :
> *Solution
> > > A *
> > > > >> >>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py :
> > > *Solution
> > > > A
> > > > >> >> *
> > > > >> >>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
> > > > >> >>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>> *Case #4* Separate namespace for resources :
> > > > >> >>>>>>>>>>>>>>>>>> *Solution A*
> > > > >> >>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
> > > > >> >>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>> *Case #7* : *Solution A*
> > > > >> >>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel
> Standish
> > > > >> >>>>>>>>>>>>>>>>> <dp...@gmail.com>
> > > > >> >>>>>>>>>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> I have tried to add some clarification to
> > Jarek's
> > > > >> >>> summary,
> > > > >> >>>>>>>> but
> > > > >> >>>>>>>>>>> I
> > > > >> >>>>>>>>>>>>> am
> > > > >> >>>>>>>>>>>>>>>>> a
> > > > >> >>>>>>>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so
> > > > hopefully
> > > > >> >>>> Jarek
> > > > >> >>>>>>>>>>> can
> > > > >> >>>>>>>>>>>>>>>>> further
> > > > >> >>>>>>>>>>>>>>>>>>> clarify.
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> According to my reading, in this motion cases
> > > 4,5,6
> > > > >> >> are
> > > > >> >>>> all
> > > > >> >>>>>>>>>>>>> either
> > > > >> >>>>>>>>>>>>>>>>>>> proposal
> > > > >> >>>>>>>>>>>>>>>>>>> *rejections* or otherwise have no effect, so
> > > > attention
> > > > >> >>>> can be
> > > > >> >>>>>>>>>>>>>>>>> focused on
> > > > >> >>>>>>>>>>>>>>>>>>> cases 1,2,3 and 7.
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> *Motion is to adopt the following:*
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
> > > > >> >>>>>>>>>>>>>>>>>>> All objects will be moved out of contrib
> folder
> > > and
> > > > >> >> into
> > > > >> >>>>>>>>>>>>> respective
> > > > >> >>>>>>>>>>>>>>>>>>> non-contrib folder.
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> Case 2 - Solution A - Remove object type
> suffix
> > > from
> > > > >> >>>> modules
> > > > >> >>>>>>>>>>>>>>>>>>> Example:
> > > > >> >>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator
> > becomes
> > > > >> >>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator
> > > > >> >>>>>>>>>>>>>>>>>>> Example:
> > > > >> >>>>>>>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
> > > > >> >>>>>>>>>>>>> airflow.hooks.foo.FooHook
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc
> > > prefixes
> > > > >> >>>>>>>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
> > > > >> >>>>>>>>>>>>>>>>>>> Example case is
> > > > >> >>>>>>>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
> > > > >> >>>>>>>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can
> > > assume
> > > > >> >> it
> > > > >> >>> is
> > > > >> >>>>>>>> now
> > > > >> >>>>>>>>>>>>> named
> > > > >> >>>>>>>>>>>>>>>>> like
> > > > >> >>>>>>>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
> > > > >> >>>>>>>>>>>>>>>>>>> So what is proposed for this case?
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> Case 4 - Solution B - *no change*
> > > > >> >>>>>>>>>>>>>>>>>>> This proposal was about namespaces but the
> > motion
> > > is
> > > > >> >> to
> > > > >> >>>>>>>> reject
> > > > >> >>>>>>>>>>>>> the
> > > > >> >>>>>>>>>>>>>>>>>>> proposal.
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> Case 5 - Solution B - *no change*
> > > > >> >>>>>>>>>>>>>>>>>>> The proposal was to remove class type suffix
> > e.g.
> > > > >> >> remove
> > > > >> >>>> the
> > > > >> >>>>>>>>>>>>>>>>> Operator in
> > > > >> >>>>>>>>>>>>>>>>>>> FooOperator, but the motion is to reject this
> > > > proposal
> > > > >> >>> --
> > > > >> >>>>>>>> i.e.
> > > > >> >>>>>>>>>>>>>>>>> motion is
> > > > >> >>>>>>>>>>>>>>>>>>> no
> > > > >> >>>>>>>>>>>>>>>>>>> change on this case, i.e. we resolve to keep
> > > > >> >> "Operator"
> > > > >> >>>> (and
> > > > >> >>>>>>>>>>>>>>>>> "Sensor")
> > > > >> >>>>>>>>>>>>>>>>>>> suffixes on class names
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> Case 6 - *No change*
> > > > >> >>>>>>>>>>>>>>>>>>> This case concerns object naming
> > inconsistencies,
> > > > but
> > > > >> >>>> motion
> > > > >> >>>>>>>>>>> does
> > > > >> >>>>>>>>>>>>>>>>> not
> > > > >> >>>>>>>>>>>>>>>>>>> enact
> > > > >> >>>>>>>>>>>>>>>>>>> any specific proposal.
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn
> template
> > > for
> > > > >> >>>>>>>>>>> deprecation
> > > > >> >>>>>>>>>>>>>>>>> warnings
> > > > >> >>>>>>>>>>>>>>>>>>> (instead of zope)
> > > > >> >>>>>>>>>>>>>>>>>>> Example:
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> # in old_package/old_mod.py
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> import warnings
> > > > >> >>>>>>>>>>>>>>>>>>> from new_package.new_mod import *
> > > > >> >>>>>>>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved
> to
> > > > >> >>>>>>>>>>>>> new_package.new_mod.
> > > > >> >>>>>>>>>>>>>>>>>>> Import
> > > > >> >>>>>>>>>>>>>>>>>>> of "
> > > > >> >>>>>>>>>>>>>>>>>>> "old_package.old_mod will become unsupported
> in
> > > > >> >> version
> > > > >> >>>> 2",
> > > > >> >>>>>>>>>>>>>>>>>>> DeprecationWarning, 2)
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> Reference implementation here:
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> I am very happy that this motion now gets rid
> of
> > > > >> >>> contrib.
> > > > >> >>>>>>>>>>> Thank
> > > > >> >>>>>>>>>>>>> you
> > > > >> >>>>>>>>>>>>>>>>>>> Sergio
> > > > >> >>>>>>>>>>>>>>>>>>> for turning the tide with your effective
> > > > argumentation
> > > > >> >>> ;)
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash
> > Berlin-Taylor
> > > <
> > > > >> >>>>>>>>>>>>> ash@apache.org>
> > > > >> >>>>>>>>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>> Could someone summarise what the proposed
> name
> > > > >> >> changes
> > > > >> >>>> are,
> > > > >> >>>>>>>>>>>>> here,
> > > > >> >>>>>>>>>>>>>>>>> in
> > > > >> >>>>>>>>>>>>>>>>>>> this
> > > > >> >>>>>>>>>>>>>>>>>>>> thread? Pointing at a google doc and only
> > giving
> > > > >> >>> partial
> > > > >> >>>>>>>>>>>>> examples
> > > > >> >>>>>>>>>>>>>>>>> is 1)
> > > > >> >>>>>>>>>>>>>>>>>>> a
> > > > >> >>>>>>>>>>>>>>>>>>>> bit difficult to quickly work out what we are
> > > > voting
> > > > >> >>> on,
> > > > >> >>>> and
> > > > >> >>>>>>>>>>> 2)
> > > > >> >>>>>>>>>>>>>>>>> using a
> > > > >> >>>>>>>>>>>>>>>>>>>> google doc isn't great for a longer term
> > record.
> > > > >> >>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>> Thanks,
> > > > >> >>>>>>>>>>>>>>>>>>>> Ash
> > > > >> >>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
> > > > >> >>>>>>>>>>>>>>>>> <Ja...@polidea.com>
> > > > >> >>>>>>>>>>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>> I would like to recast the vote. Let's start
> > > from
> > > > 0
> > > > >> >> :)
> > > > >> >>>> . I
> > > > >> >>>>>>>>>>>>> would
> > > > >> >>>>>>>>>>>>>>>>> like
> > > > >> >>>>>>>>>>>>>>>>>>> to
> > > > >> >>>>>>>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at
> least
> > > > three
> > > > >> >>> +1
> > > > >> >>>>>>>>>>>>>>>>> (binding)
> > > > >> >>>>>>>>>>>>>>>>>>> votes
> > > > >> >>>>>>>>>>>>>>>>>>>> are
> > > > >> >>>>>>>>>>>>>>>>>>>>> needed to pass. I marked in red the
> > > modifications
> > > > to
> > > > >> >>> the
> > > > >> >>>>>>>>>>>>>>>>> previous
> > > > >> >>>>>>>>>>>>>>>>>>>> proposal.
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>> Here is the proposal (details here
> > > > >> >>>>>>>>>>>>>>>>>>>>> <
> > > > >> >>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>> )
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
> > > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 2: A:
> > > > >> >>> Airflow.operators.foo_operator.FooOperator
> > > > >> >>>>>>>>>>>>> could
> > > > >> >>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>
> > > > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > > > >> >>>>>>>>>>>>>>>>>>> could
> > > > >> >>>>>>>>>>>>>>>>>>>>> become
> > > > >> >>> airflow.gcp.operators.bigtable.BigTableOperator*
> > > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
> > > > >> >> suffixes
> > > > >> >>> on
> > > > >> >>>>>>>>>>>>> class
> > > > >> >>>>>>>>>>>>>>>>>>> names*
> > > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> > > > >> >>> case-by-case
> > > > >> >>>>>>>>>>>>> (and
> > > > >> >>>>>>>>>>>>>>>>> my team
> > > > >> >>>>>>>>>>>>>>>>>>>> can
> > > > >> >>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 7: Native python solution (with
> better
> > > IDE
> > > > >> >>>>>>>>>>>>>>>>> integration)*
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>> This is my binding (+1) vote.
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek
> Potiuk
> > <
> > > > >> >>>>>>>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
> > > > >> >>>>>>>>>>>>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>> Hey Felix,
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the
> native
> > > one
> > > > >> >> as
> > > > >> >>>> well
> > > > >> >>>>>>>>>>> (I
> > > > >> >>>>>>>>>>>>>>>>> use IDE
> > > > >> >>>>>>>>>>>>>>>>>>>> quite
> > > > >> >>>>>>>>>>>>>>>>>>>>>> often ;) )
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>> J.
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong,
> > > Fokko
> > > > >> >>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document
> > > together.
> > > > I
> > > > >> >>>> wasn't
> > > > >> >>>>>>>>>>>>> able
> > > > >> >>>>>>>>>>>>>>>>> to
> > > > >> >>>>>>>>>>>>>>>>>>>> respond
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> earlier since I was giving Airflow
> workshops
> > > > >> >>>> throughout
> > > > >> >>>>>>>>>>>>> Europe
> > > > >> >>>>>>>>>>>>>>>>> :-)
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything
> > from
> > > > >> >>> contrib
> > > > >> >>>>>>>>>>> into
> > > > >> >>>>>>>>>>>>> a
> > > > >> >>>>>>>>>>>>>>>>> single
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> package.
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> To make it easier to the end-user, my
> > > preference
> > > > >> >>>> would be
> > > > >> >>>>>>>>>>>>> to
> > > > >> >>>>>>>>>>>>>>>>> get
> > > > >> >>>>>>>>>>>>>>>>>>> rid of
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> the
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> contrib package altogether. What's in
> > contrib
> > > or
> > > > >> >> not
> > > > >> >>>> is a
> > > > >> >>>>>>>>>>>>>>>>> tricky
> > > > >> >>>>>>>>>>>>>>>>>>>> question,
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> I think this is already reflected in the
> > > > document
> > > > >> >>>> since
> > > > >> >>>>>>>>>>>>>>>>> "maturity"
> > > > >> >>>>>>>>>>>>>>>>>>> is
> > > > >> >>>>>>>>>>>>>>>>>>>> in
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> quotes. What would be the moment to
> transfer
> > > the
> > > > >> >>>> operator
> > > > >> >>>>>>>>>>>>> to
> > > > >> >>>>>>>>>>>>>>>>> the
> > > > >> >>>>>>>>>>>>>>>>>>> core
> > > > >> >>>>>>>>>>>>>>>>>>>> or
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming
> contrib
> > > to
> > > > >> >>>>>>>>>>> incubating
> > > > >> >>>>>>>>>>>>>>>>> will
> > > > >> >>>>>>>>>>>>>>>>>>> confuse
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> the
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> user even more. For people who aren't as
> > known
> > > > >> >> with
> > > > >> >>>>>>>>>>>>> Airflow it
> > > > >> >>>>>>>>>>>>>>>>> isn't
> > > > >> >>>>>>>>>>>>>>>>>>>> that
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> transparent which operator lives where,
> > > > especially
> > > > >> >>> if
> > > > >> >>>> you
> > > > >> >>>>>>>>>>>>>>>>> don't use
> > > > >> >>>>>>>>>>>>>>>>>>> an
> > > > >> >>>>>>>>>>>>>>>>>>>> IDE
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think
> it
> > > is
> > > > >> >>> worth
> > > > >> >>>>>>>>>>>>> changing
> > > > >> >>>>>>>>>>>>>>>>> the
> > > > >> >>>>>>>>>>>>>>>>>>>> public
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> API just for a different name.
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote
> > > with
> > > > >> >>> this.
> > > > >> >>>> It
> > > > >> >>>>>>>>>>>>> will
> > > > >> >>>>>>>>>>>>>>>>> make
> > > > >> >>>>>>>>>>>>>>>>>>>> easier
> > > > >> >>>>>>>>>>>>>>>>>>>>>> as we will not have to define other rules
> as
> > > > those
> > > > >> >>>> that we
> > > > >> >>>>>>>>>>>>>>>>> already
> > > > >> >>>>>>>>>>>>>>>>>>> have.
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B
> > > (let's
> > > > >> >>> say a
> > > > >> >>>>>>>>>>> +0)
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> I like to search on Github with t, and
> then
> > > > search
> > > > >> >>> for
> > > > >> >>>>>>>>>>>>>>>>> BashOperator.
> > > > >> >>>>>>>>>>>>>>>>>>>> After
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> the change, you should search for
> > > Operator/Bash.
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on
> this,
> > > from
> > > > >> >>> your
> > > > >> >>>>>>>>>>> mail
> > > > >> >>>>>>>>>>>>> you
> > > > >> >>>>>>>>>>>>>>>>>>> state: -
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 2: B:
> > > > >> >>> Airflow.operators.foo_operator.FooOperator
> > > > >> >>>>>>>>>>>>> could
> > > > >> >>>>>>>>>>>>>>>>> become
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But
> the
> > B
> > > > >> >>>> solution is
> > > > >> >>>>>>>>>>>>>>>>> keeping it
> > > > >> >>>>>>>>>>>>>>>>>>>> the
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator
> > > > *would
> > > > >> >>> stay
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> airflow.operators.foo_operator*.FooOperator*
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think
> > there
> > > > is a
> > > > >> >>>> little
> > > > >> >>>>>>>>>>>>>>>>> confusion
> > > > >> >>>>>>>>>>>>>>>>>>>> here.
> > > > >> >>>>>>>>>>>>>>>>>>>>>> This case is not for changing BashOperator
> > into
> > > > >> >> Bash
> > > > >> >>>> but
> > > > >> >>>>>>>>>>> for
> > > > >> >>>>>>>>>>>>>>>>> changing
> > > > >> >>>>>>>>>>>>>>>>>>>> the
> > > > >> >>>>>>>>>>>>>>>>>>>>>> *module* name not the *class* name. So
> > > > >> >>>>>>>>>>>>>>>>> bash_operator.BashOperator ->
> > > > >> >>>>>>>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still
> search
> > > for
> > > > >> >>>>>>>>>>>>> BashOperator
> > > > >> >>>>>>>>>>>>>>>>> in this
> > > > >> >>>>>>>>>>>>>>>>>>>> case.
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to
> decide
> > > > when
> > > > >> >> it
> > > > >> >>>> gets
> > > > >> >>>>>>>>>>>>> its
> > > > >> >>>>>>>>>>>>>>>>> own
> > > > >> >>>>>>>>>>>>>>>>>>>> package
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there
> > are
> > > > >> >>>> companies
> > > > >> >>>>>>>>>>>>> like
> > > > >> >>>>>>>>>>>>>>>>>>>> Databricks
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> and Qubole which might also end up with
> > their
> > > > own
> > > > >> >>>> package
> > > > >> >>>>>>>>>>>>>>>>> because
> > > > >> >>>>>>>>>>>>>>>>>>> their
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> integration is getting richer over time.
> > > Moving
> > > > >> >> this
> > > > >> >>>> is a
> > > > >> >>>>>>>>>>>>> bit
> > > > >> >>>>>>>>>>>>>>>>>>> painful
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> because we need to deprecate the old path
> > and
> > > > move
> > > > >> >>>>>>>>>>>>> everything
> > > > >> >>>>>>>>>>>>>>>>> which
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> introduces noise into version control etc.
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed.
> > > It's
> > > > >> >> not
> > > > >> >>> as
> > > > >> >>>>>>>>>>>>>>>>> difficult as
> > > > >> >>>>>>>>>>>>>>>>>>> it
> > > > >> >>>>>>>>>>>>>>>>>>>>>> seems to group operators in packages and it
> > has
> > > > >> >>>> numerous
> > > > >> >>>>>>>>>>>>>>>>> advantages.
> > > > >> >>>>>>>>>>>>>>>>>>> The
> > > > >> >>>>>>>>>>>>>>>>>>>>>> nice thing about deprecations - we can do
> > them
> > > in
> > > > >> >> the
> > > > >> >>>>>>>>>>>>>>>>> __init__.py of
> > > > >> >>>>>>>>>>>>>>>>>>> the
> > > > >> >>>>>>>>>>>>>>>>>>>>>> packages which causes the noise fairly
> small
> > > (Git
> > > > >> >>> will
> > > > >> >>>>>>>>>>>>> actually
> > > > >> >>>>>>>>>>>>>>>>>>> realise
> > > > >> >>>>>>>>>>>>>>>>>>>>>> that the file has been moved and you can
> > follow
> > > > >> >> that.
> > > > >> >>>>>>>>>>> Maybe
> > > > >> >>>>>>>>>>>>> not
> > > > >> >>>>>>>>>>>>>>>>> as
> > > > >> >>>>>>>>>>>>>>>>>>> super
> > > > >> >>>>>>>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but
> > it's
> > > > >> >> not a
> > > > >> >>>>>>>>>>> rocket
> > > > >> >>>>>>>>>>>>>>>>> science
> > > > >> >>>>>>>>>>>>>>>>>>>> either.
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>> Yes.
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> ---
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> Apart from all, great work!
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix
> > > > >> >> Uellendall
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>> :
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>> I have no strong "No" against any
> proposed
> > > > change
> > > > >> >>> of
> > > > >> >>>>>>>>>>> these
> > > > >> >>>>>>>>>>>>>>>>> cases.
> > > > >> >>>>>>>>>>>>>>>>>>> So I
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> go
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>> with +1 (non-binding).
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up
> > again
> > > > and
> > > > >> >>> your
> > > > >> >>>>>>>>>>>>> intense
> > > > >> >>>>>>>>>>>>>>>>> work
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> towards
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil
> > for
> > > > even
> > > > >> >>>>>>>>>>> creating
> > > > >> >>>>>>>>>>>>>>>>> this
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> document. I
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>> like how the code is getting more and
> more
> > > > >> >>> consistent
> > > > >> >>>>>>>>>>> and
> > > > >> >>>>>>>>>>>>>>>>> clean :)
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>> Felix
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>> -------- Original Message --------
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk
> wrote:
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> This email is calling a vote on the
> > changes
> > > in
> > > > >> >>>> import
> > > > >> >>>>>>>>>>>>> paths.
> > > > >> >>>>>>>>>>>>>>>>> It's
> > > > >> >>>>>>>>>>>>>>>>>>>> been
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> discussed in
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week
> > (July
> > > > >> >> 30th
> > > > >> >>>> 6pm
> > > > >> >>>>>>>>>>>>> CEST),
> > > > >> >>>>>>>>>>>>>>>>> and
> > > > >> >>>>>>>>>>>>>>>>>>> at
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> least
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the
> > > document
> > > > >> >> from
> > > > >> >>>>>>>>>>> Kamil
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> <
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> The proposed solution is:
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move
> all
> > > that
> > > > >> >> are
> > > > >> >>>>>>>>>>> "well"
> > > > >> >>>>>>>>>>>>>>>>> tested
> > > > >> >>>>>>>>>>>>>>>>>>> and
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or
> > similar.*
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 2: B:
> > > > >> >>>>>>>>>>> Airflow.operators.foo_operator.FooOperator
> > > > >> >>>>>>>>>>>>>>>>> could
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> become
> airflow.operators.foo.FooOperator*
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>
> > > > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > > > >> >>>>>>>>>>>>>>>>>>>> could
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> become
> > > > >> >>>> airflow.gcp.operators.bigtable.BigTableOperator*
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and
> > "Sensor")
> > > > >> >>>> suffixes
> > > > >> >>>>>>>>>>> on
> > > > >> >>>>>>>>>>>>>>>>> class
> > > > >> >>>>>>>>>>>>>>>>>>> names*
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases
> on
> > > > >> >>>> case-by-case
> > > > >> >>>>>>>>>>>>> (and
> > > > >> >>>>>>>>>>>>>>>>> my
> > > > >> >>>>>>>>>>>>>>>>>>> team
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> can
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> J.
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> --
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> |
> > > > Principal
> > > > >> >>>>>>>>>>> Software
> > > > >> >>>>>>>>>>>>>>>>> Engineer
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
> > > > >> >>>>>>>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <
> > https://www.polidea.com/>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>> --
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> > > > >> >>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> |
> > Principal
> > > > >> >>>> Software
> > > > >> >>>>>>>>>>>>>>>>> Engineer
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > > > >> >>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/
> >
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>> --
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> > > > >> >>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> |
> > Principal
> > > > >> >>> Software
> > > > >> >>>>>>>>>>>>>> Engineer
> > > > >> >>>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > > > >> >>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > > > >> >>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>> --
> > > > >> >>>>>>>>>>>>>>>>>> *Kaxil Naik*
> > > > >> >>>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > > > >> >>>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer |
> > > *Certified*
> > > > >> >>> Apache
> > > > >> >>>>>>>>>>> Spark
> > > > >> >>>>>>>>>>>>> &
> > > > >> >>>>>>>>>>>>>>>>> Neo4j
> > > > >> >>>>>>>>>>>>>>>>>> Developer
> > > > >> >>>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > > >> >>>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>>> --
> > > > >> >>>>>>>>>>>>>>>>> *Kaxil Naik*
> > > > >> >>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > > > >> >>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer |
> > *Certified*
> > > > >> >>> Apache
> > > > >> >>>>>>>> Spark
> > > > >> >>>>>>>>>>> &
> > > > >> >>>>>>>>>>>>>>>>> Neo4j
> > > > >> >>>>>>>>>>>>>>>>> Developer
> > > > >> >>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > > >> >>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> --
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> Jarek Potiuk
> > > > >> >>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> Software
> > > > >> >>> Engineer
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > > > >> >>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>>>>> --
> > > > >> >>>>>>>> *Kaxil Naik*
> > > > >> >>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > > > >> >>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
> Apache
> > > > >> >> Spark &
> > > > >> >>>> Neo4j
> > > > >> >>>>>>>> Developer
> > > > >> >>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > > >> >>>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>> --
> > > > >> >>>>>>>
> > > > >> >>>>>>> Jarek Potiuk
> > > > >> >>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> > > > Engineer
> > > > >> >>>>>>>
> > > > >> >>>>>>> M: +48 660 796 129 <+48660796129>
> > > > >> >>>>>>> [image: Polidea] <https://www.polidea.com/>
> > > > >> >>>>>>>
> > > > >> >>>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>> --
> > > > >> >>>>>>
> > > > >> >>>>>> Jarek Potiuk
> > > > >> >>>>>> Polidea <https://www.polidea.com/> | Principal Software
> > > Engineer
> > > > >> >>>>>>
> > > > >> >>>>>> M: +48 660 796 129 <+48660796129>
> > > > >> >>>>>> [image: Polidea] <https://www.polidea.com/>
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > > >>
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Changes in import paths
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi Jarek,
No worries. I must admit that I've lost track a bit when the case 3+4 got
combined. Maybe for the future better to let the current vote run and maybe
reinitiate a new vote if there are new options.
Currently, it is quite easy to think of a few obvious groups, but it is so
hard to keep this consistent in the future. I'm not saying that one giant
operator package is ideal, but I think it is easy to maintain. Two/three
years ago the guys for Databricks wrote all the operators for their
platform, and it was in such a state that it was eligible to move to the
core (non-contrib), similar with GCP now. But it is hard to actually do it.
Putting everything into one place makes this from my perspective simpler.
If Ali-Cloud comes with some operators, should we put them in a separate
package right away? Prefixing would be a nice compromise for me.
If I'm the only one against adding the additional namespaces, then I don't
want to stop the rest of the community of making this happen. I don't like
to veto stuff, but I think we should think this through properly :-)
Cheers, Fokko
Op ma 5 aug. 2019 om 17:55 schreef Jarek Potiuk <Ja...@polidea.com>:
> Hey Fokko,
>
> Really sorry, to miss this . I see now that you have not voted because you
> thought it will be transferred automatically -it was thought as not and
> additional option but combined replacement for (3+4) cases.
>
> This was after I realised that there was a veto for namespaces from Ash and
> I realised from it that namespaces were a bit confusing and overlapping
> with grouping per-cloud-provider (see the voting "chapter
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> >").
> I re-fined the choice back then - whether to not group them (A) or whether
> to group first by type of entity or first by cloud provider (B-E in various
> ways). Maybe I was not very clear in my message - apologies.
>
> I now re-added your vote and also Ben's vote who also voted on leaving the
> prefix and not grouping operators (Ben please correct me if I was wrong).
> Still the result is 4:2 (binding), 6:2 (including non-binding) in favour of
> grouping per cloud provider rather than using prefixes. And no vetoes.
>
> Re - the point of which operators should be grouped. From the discussions -
> I do not think it's easy to do a general and always applicable rule. But I
> believe we agreed that we group the "obvious" choices - like "gcp",
> "azure", "aws". It's for pure "Action" type operators - but still we leave
> the cross-platform "transfer" ones or non-obvious ones in the old place
> (which we keep as bag of "everything else"). Personally - I am all for
> grouping the qubole/databricks operators in packages same as other cloud
> providers. But with the "softness" of that rule this might (and should) be
> an individual decision of people who maintain it. And it is quite OK - not
> everything has to be decided upfront. I don't think we have a "hard" rule
> now that covers all the cases and binds us. It would be very complex one if
> we have it (and we can defer introducing it for later if needed especially
> if it requires more than just renaming/moving but also architectural
> changes - as Ash proposed for the transfer operators). But at least we can
> decrease an entropy by grouping the obvious cases, and we can move forward
> leaving the Airflow world a bit better, one step at a time.
>
> J.
>
> On Mon, Aug 5, 2019 at 3:58 PM Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
> > Hi Jarek,
> >
> > Just wondering how the vote is unanimous. On case 3 I've voted for B. I
> > still believe introducing namespaces is hard to maintain, as I explained
> > earlier: Mostly because it isn't as trivial to decide when it gets its
> own
> > package or not. Besides the cloud providers, there are companies like
> > Databricks and Qubole which might also end up with their own package
> > because their integration is getting richer over time. Moving this is a
> bit
> > painful because we need to deprecate the old path and move everything
> which
> > introduces noise into version control etc.
> >
> > Adding an additional option to the list, does not invalidate the older
> > options, right?
> >
> > Cheers, Fokko
> >
> >
> > Op ma 5 aug. 2019 om 15:43 schreef Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >:
> >
> > > Seems that after this months of discussion we got an unanimous voting
> > > result. I summarised it in the AIP-21
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > > >
> > > and
> > > changed its status to "Accepted". Thanks for all your feedback! We will
> > > start soon moving GCP operators following those rules and we will
> resolve
> > > any initial problems with those - then we might provide some additional
> > > learnings and see if we can easily (probably semi-automatically) move
> all
> > > other operators.
> > >
> > > Case 1
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#1airflow.contrib.{resources}
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%231airflow.contrib.%7Bresources%7D>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%231airflow.contrib.%7Bresources%7D
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%231airflow.contrib.%7Bresources%7D
> > >
> > > >
> > >
> > > What to do with the "contrib" folder
> > >
> > > Case 2
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#2git*_{operator/sensor}{/s}.py
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%232git*_%7Boperator/sensor%7D%7B/s%7D.py>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%232git*_%7Boperator/sensor%7D%7B/s%7D.py
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%232git*_%7Boperator/sensor%7D%7B/s%7D.py
> > >
> > > >
> > >
> > > Drop modules *_operator suffix
> > >
> > > Case 3
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#3{aws/azure/gcp}_*
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%233%7Baws/azure/gcp%7D_*>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%233%7Baws/azure/gcp%7D_*
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%233%7Baws/azure/gcp%7D_*
> > >
> > > >
> > > + Case 4
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#4Separatenamespaceforresources
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%234Separatenamespaceforresources>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%234Separatenamespaceforresources
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%234Separatenamespaceforresources
> > >
> > > >
> > >
> > > Grouping Cloud Providers operators/sensors/hooks
> > >
> > > Case 5
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#5*Operator
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%235*Operator>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%235*Operator
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%235*Operator
> > >
> > > >
> > >
> > > *Operator *Sensor *Hook in class name
> > >
> > > Case 6
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#6Otherisolatedcases
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%236Otherisolatedcases>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%236Otherisolatedcases
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%236Otherisolatedcases
> > >
> > > >
> > >
> > > Isolated cases
> > >
> > > Case 7
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#7
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%237>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%237
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%237
> > >
> > > >
> > >
> > > Deprecation method
> > >
> > > *A.* Move everything "contrib" → "airflow"
> > >
> > > *A.* Drop the suffix.
> > >
> > > Example:
> > >
> > > - *airflow.operator.gcp_bigtable_operator.py*
> > > becomes *airflow.operator.gcp_bigtable.py
> > > <http://airflow.operator.gcp_bigtable.py>*.
> > >
> > > *D. * Group operators/sensors/hooks in
> > > *airflow/<PROVIDER>*/operators(sensors,
> > > hooks). Non-cloud provider ones are moved to
> > > airflow/operators(sensors/hooks).
> > > *Drop the prefix.*
> > >
> > > -
> > > *airflow/contrib/operators/sns_publish_operator.py
> > > becomes airflow/aws/operators/**sns_publish_operator.py*
> > >
> > >
> > > - *airflow/contrib/operators/dataproc_operator.py*
> > > becomes *airflow/gcp/operators/dataproc_operator.py*
> > >
> > >
> > >
> > > -
> > > *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> *airflow/*
> > > *gcp/operators/bigtable_operator.py*
> > >
> > >
> > >
> > > -
> > > *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > > *operators/ssh_operator.py*
> > >
> > > *B.* No change - keep Operator/Sensor suffix in class name.
> > >
> > > *A.* Make individual decisions of renames for operators that do not
> > follow
> > > common conventions used for other operators.
> > >
> > > Consistency trumps compatibility.
> > >
> > > Examples:
> > >
> > > *DataProcHadoopOperator*
> > >
> > > renamed to:
> > >
> > > *DataprocHadoopOperator*
> > > *A.* Native python method (with better IDE support and more flexible
> > but a
> > > bit more verbose)
> > >
> > > J.
> > >
> > > On Wed, Jul 31, 2019 at 10:30 AM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> > > wrote:
> > >
> > > > Yep. Agree with Ash on it. There are a number of 'action' operators
> > > > specific for cloud providers and these should be our target. The
> > transfer
> > > > ones require another AIP (A lot of that already discussed in AIP-8
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
> > > > ).
> > > >
> > > > J.
> > > >
> > > > Principal Software Engineer
> > > > Phone: +48660796129
> > > >
> > > > śr., 31 lip 2019, 10:01 użytkownik Ash Berlin-Taylor <ash@apache.org
> >
> > > > napisał:
> > > >
> > > >> This is a good idea for now. I'm also not overly concerned about
> these
> > > >> few non-cloud examples - FTPtoS3Operator can stay where it is and
> > > doesn't
> > > >> have to move under 'aws.' to my mind.
> > > >>
> > > >> Longer term I'd like to go back to making the
> > "transfer/copy/transform"
> > > >> operators "composable" so that we can have a single
> DataCopyOperator,
> > > and
> > > >> give it a source and destination, and it uses hooks to do the work.
> > > This is
> > > >> not a small undertaking though to do well and efficiently though.
> > > >>
> > > >> -ash
> > > >>
> > > >> > On 30 Jul 2019, at 20:54, Tomasz Urbaszek <
> > > tomasz.urbaszek@polidea.com>
> > > >> wrote:
> > > >> >
> > > >> > Maybe we can put all those AtoB operators under one name like
> > > >> “transfer”,
> > > >> > then it would be easier to look for such operator?
> > > >> >
> > > >> > Best,
> > > >> > Tomek
> > > >> >
> > > >> > On Tue, 30 Jul 2019 at 21:39, Chen Tong <ci...@gmail.com> wrote:
> > > >> >
> > > >> >> Daniel mentioned a good point. Such composed operator may also
> > > involves
> > > >> >> both cloud and non-cloud provider saying FTPtoS3Operator. Should
> it
> > > in
> > > >> AWS
> > > >> >> folder?
> > > >> >>
> > > >> >> Also, saying in the future, another cloud provider is growing
> large
> > > >> enough.
> > > >> >> Will we rename all plugins related to this provider? What's the
> > > >> criteria we
> > > >> >> say it's big enough? And option D gives an impression like these
> > > tools
> > > >> may
> > > >> >> be maintained and supported by the Cloud provider. I am not sure
> if
> > > it
> > > >> will
> > > >> >> be a truth or not.
> > > >> >>
> > > >> >> Best,
> > > >> >>
> > > >> >>
> > > >> >> On Tue, Jul 30, 2019 at 11:18 AM Daniel Standish <
> > > dpstandish@gmail.com
> > > >> >
> > > >> >> wrote:
> > > >> >>
> > > >> >>> One wrinkle with case 3+4 option D is inter-provider operators.
> > > >> Mainly
> > > >> >>> it's storage I think e.g. XToS3Operator or XToGCSOperator where
> > the
> > > X
> > > >> is
> > > >> >> a
> > > >> >>> service some different provider.
> > > >> >>>
> > > >> >>> Maybe the rule should be to locate the operator according to the
> > > first
> > > >> >>> provider referenced. So e.g. s3_to_gcs_transfer_operator.py
> would
> > > go
> > > >> to
> > > >> >>> aws.
> > > >> >>>
> > > >> >>>
> > > >> >>> On Tue, Jul 30, 2019 at 6:21 AM Kamil Breguła <
> > > >> kamil.bregula@polidea.com
> > > >> >>>
> > > >> >>> wrote:
> > > >> >>>
> > > >> >>>> Yes. All changes will be backwards compatible. In the case of
> > using
> > > >> >>>> the old path, a message containing a proposal for change will
> be
> > > >> >>>> reported to the user.
> > > >> >>>>
> > > >> >>>> I prepared an example of how to change the name of a class in a
> > > case
> > > >> >>>> with the use of a native solution.
> > > >> >>>> Source code:
> > > >> >>>>
> > > >> >>
> > > >>
> > >
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename
> > > >> >>>> Preview from IDE: https://imgur.com/a/45NxS5W
> > > >> >>>>
> > > >> >>>> On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <
> > ash@apache.org>
> > > >> >>> wrote:
> > > >> >>>>>
> > > >> >>>>> Just cos I'm not sure it's _explicitly_ stated, but all of the
> > > moves
> > > >> >>>> will have a deprecation of the old name right?
> > > >> >>>>>
> > > >> >>>>> 3+4 case D gets my vote too.
> > > >> >>>>>
> > > >> >>>>> -a
> > > >> >>>>>
> > > >> >>>>>> On 30 Jul 2019, at 09:58, Jarek Potiuk <
> > Jarek.Potiuk@polidea.com
> > > >
> > > >> >>>> wrote:
> > > >> >>>>>>
> > > >> >>>>>> I went ahead and updated the page (just to speed it up) as I
> > > think
> > > >> >> it
> > > >> >>>>>> really makes sense to join those two cases (and I do not see
> > any
> > > >> >>>> drawbacks
> > > >> >>>>>> - I think the options we have cover all possible approaches)
> > and
> > > we
> > > >> >>> can
> > > >> >>>>>> always go back if we need to.
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > > >> >>>>>>
> > > >> >>>>>> My vote is *D*.
> > > >> >>>>>>
> > > >> >>>>>> - airflow/contrib/operators/gcp_bigtable_operator.py →
> > > >> >>> airflow/gcp
> > > >> >>>>>> /operators/bigtable_operator.py
> > > >> >>>>>> - airflow/contrib/operators/ssh_operator.py ->
> > > >> >>>>>> airflow/operators/ssh_operator.py
> > > >> >>>>>>
> > > >> >>>>>> I updated the page with my vote.
> > > >> >>>>>> I propose that we vote till Friday on that one (all the rest
> is
> > > >> >>> already
> > > >> >>>>>> agreed).
> > > >> >>>>>>
> > > >> >>>>>> J.
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>> On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <
> > > >> >>> Jarek.Potiuk@polidea.com
> > > >> >>>>>
> > > >> >>>>>> wrote:
> > > >> >>>>>>
> > > >> >>>>>>> I think almost everyone voted and we have almost perfect
> > > >> >> consensus.
> > > >> >>>> We all
> > > >> >>>>>>> agree amongst other on moving all operators out of contrib
> > > >> >> (Great!).
> > > >> >>>>>>>
> > > >> >>>>>>> The only doubts are for *Case 3* (Cloud provider prefix) and
> > > *Case
> > > >> >>> 4*
> > > >> >>>>>>> (Using Namespaces).
> > > >> >>>>>>> I think there was actually an overlap between those two.
> Also
> > > >> >> Ash's
> > > >> >>>>>>> comment/veto on that is quite clear.
> > > >> >>>>>>> So I have final (I hope!) proposal to join those two cases
> and
> > > >> >> have
> > > >> >>>> one
> > > >> >>>>>>> voting (till Friday) only on that.
> > > >> >>>>>>>
> > > >> >>>>>>> I will update the doc if you all agree with me that makes
> more
> > > >> >> sense
> > > >> >>>> as
> > > >> >>>>>>> single case to vote on:
> > > >> >>>>>>>
> > > >> >>>>>>> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
> > > >> >>>>>>>
> > > >> >>>>>>> *Option A*. Keep operators/sensors/hooks in
> > > >> >>> airflow/operators(sensors,
> > > >> >>>>>>> hooks) and keep/add prefixes in file names.
> > > >> >>>>>>>
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py becomes
> > > >> >>>>>>> airflow/operators/**aws_sns_publish_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > > >> >>>>>>> becomes *airflow/operators/gcp_dataproc_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes
> > *airflow/*
> > > >> >>>>>>> *hooks/gcp_bigtable_hook.py*
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> > *airflow/*
> > > >> >>>>>>> *operators/ssh_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> *Option B*. Group operators/sensors/hooks in
> > > >> >>>> airflow/operators(sensors,
> > > >> >>>>>>> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
> > > >> >>>>>>> airflow/operators(sensors/hooks), Drop the prefix.
> > > >> >>>>>>>
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> > > >> >>>>>>> becomes airflow/operators/**aws/sns_publish_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > > >> >>>>>>> becomes *airflow/operators/gcp/dataproc_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py
> *becomes
> > > >> >>>> *airflow/*
> > > >> >>>>>>> *operators/gcp/bigtable_operator.py*
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> > *airflow/*
> > > >> >>>>>>> *operators/ssh_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> *Option C*. Group operators/sensors/hooks in
> > > >> >>>> airflow/operators(sensors,
> > > >> >>>>>>> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
> > > >> >>>>>>> airflow/operators(sensors/hooks), Keep the prefix.
> > > >> >>>>>>>
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> > > >> >>>>>>> becomes
> airflow/operators/**aws/aws_sns_publish_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > > >> >>>>>>> becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py
> *becomes
> > > >> >>>> *airflow/*
> > > >> >>>>>>> *operators/gcp/gcp_bigtable_operator.py*
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> > *airflow/*
> > > >> >>>>>>> *operators/ssh_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> *Option D.* Group operators/sensors/hooks in
> > > >> >>>> *airflow/<PROVIDER>*/operators(sensors,
> > > >> >>>>>>> hooks). Non-cloud provider ones are moved to
> > > >> >>>>>>> airflow/operators(sensors/hooks). Drop the prefix.
> > > >> >>>>>>>
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> > > >> >>>>>>> becomes airflow/aws/operators/**sns_publish_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > > >> >>>>>>> becomes *airflow/gcp/operators/dataproc_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py
> *becomes
> > > >> >>>> *airflow/*
> > > >> >>>>>>> *gcp/operators/bigtable_operator.py*
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> > *airflow/*
> > > >> >>>>>>> *operators/ssh_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> *Option E.* Group operators/sensors/hooks in
> > > >> >>>> *airflow/<PROVIDER>*/operators(sensors,
> > > >> >>>>>>> hooks). Non-cloud provider ones are moved to
> > > >> >>>>>>> airflow/operators(sensors/hooks).* Keep the prefix*.
> > > >> >>>>>>>
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> > > >> >>>>>>> becomes
> airflow/aws/operators/aws_**sns_publish_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > > >> >>>>>>> becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py
> *becomes
> > > >> >>>> *airflow/*
> > > >> >>>>>>> *gcp/operators/gcp_bigtable_operator.py*
> > > >> >>>>>>> -
> > > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> > *airflow/*
> > > >> >>>>>>> *operators/ssh_operator.py*
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> Shall I proceed with updating the page ?
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> J.
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <
> > > kaxilnaik@gmail.com>
> > > >> >>>> wrote:
> > > >> >>>>>>>
> > > >> >>>>>>>> Thanks Jarek, adding my vote.
> > > >> >>>>>>>>
> > > >> >>>>>>>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <
> > > >> >> ash@apache.org>
> > > >> >>>> wrote:
> > > >> >>>>>>>>
> > > >> >>>>>>>>> Thanks Jarek, much clearer. I have voted there too.
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> -ash
> > > >> >>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko
> > > >> >> <fokko@driesprong.frl
> > > >> >>>>
> > > >> >>>>>>>>> wrote:
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> Thanks Jarek, the Wiki works much better for me. I've
> added
> > > my
> > > >> >>> vote
> > > >> >>>>>>>> there
> > > >> >>>>>>>>>> as well.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> You've convinced me on the second case. I've changed my
> > vote
> > > on
> > > >> >>>> Case 2
> > > >> >>>>>>>>> from
> > > >> >>>>>>>>>> A to B :-)
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> Maybe we should leave the vote open a bit longer since it
> > > >> >>> involves
> > > >> >>>>>>>> quite
> > > >> >>>>>>>>>> some reading, and I would like to get some proper
> feedback
> > > and
> > > >> >>>>>>>> opinions
> > > >> >>>>>>>>> on
> > > >> >>>>>>>>>> this. Plus I only see two votes right now. If we don't
> > think
> > > >> >> this
> > > >> >>>>>>>>> through,
> > > >> >>>>>>>>>> it might be that we're having this vote again in the near
> > > >> >> future
> > > >> >>>> :-)
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> Cheers, Fokko
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
> > > >> >>>>>>>>> Jarek.Potiuk@polidea.com>:
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>>> I updated the AIP-21 proposal in the way that should
> > answer
> > > >> >> all
> > > >> >>>> the
> > > >> >>>>>>>>>>> concerns in this thread.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> NOTE! There is an action for all of those who are
> > > interested:
> > > >> >>>> Please
> > > >> >>>>>>>>>>> fill-in your voting in the voting table till Tuesday
> 30th
> > of
> > > >> >>> July
> > > >> >>>> 6pm
> > > >> >>>>>>>>> CEST
> > > >> >>>>>>>>>>> (5pm BST, 9 am PST).
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> I created a summary of the proposal
> > > >> >>>>>>>>>>> <
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>> in table form - which contains also examples for each
> > case.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> I added a voting table
> > > >> >>>>>>>>>>> <
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>> where you should add your votes:
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> I propose this voting mechanism:
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm
> CEST
> > > (5
> > > >> >>> pm
> > > >> >>>>>>>> BST,
> > > >> >>>>>>>>> 9am
> > > >> >>>>>>>>>>> PST) .
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> After the choice, the final consistent set of choices
> will
> > > be
> > > >> >>>>>>>> announced
> > > >> >>>>>>>>>>> (taking into account majority of binding vote, also
> > > including
> > > >> >>>>>>>> potential
> > > >> >>>>>>>>>>> vetos and consistency between the choices. Non-binding
> > votes
> > > >> >>> will
> > > >> >>>> be
> > > >> >>>>>>>>> taken
> > > >> >>>>>>>>>>> into account in case there is a draw. The final set of
> > > choices
> > > >> >>>> will
> > > >> >>>>>>>> be
> > > >> >>>>>>>>>>> announced at devlist thread
> > > >> >>>>>>>>>>> <
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>> after
> > > >> >>>>>>>>>>> the voting completes.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Unless there is a veto raised to the final proposal, the
> > > final
> > > >> >>>>>>>> proposal
> > > >> >>>>>>>>> is
> > > >> >>>>>>>>>>> accepted by Lazy Consensus
> > > >> >>>>>>>>>>> <
> > https://community.apache.org/committers/lazyConsensus.html
> > > >
> > > >> >>> on
> > > >> >>>>>>>>> *Friday*
> > > >> >>>>>>>>>>> 02
> > > >> >>>>>>>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Please let me know if what I proposed can be further
> > > improved
> > > >> >> to
> > > >> >>>>>>>> avoid
> > > >> >>>>>>>>>>> chaos.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> J.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
> > > >> >>>>>>>> Jarek.Potiuk@polidea.com
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>>> Ok. I see that indeed voting could have been organised
> a
> > > bit
> > > >> >>>> better
> > > >> >>>>>>>> -
> > > >> >>>>>>>>> dev
> > > >> >>>>>>>>>>>> list does not seem to cope well with such a complex
> case
> > > :).
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>> As Kamil noticed - we already have a formal AIP-21
> > > >> >>>>>>>>>>>> <
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>> in the Wiki - which I should have mentioned before. I
> > have
> > > >> >> not
> > > >> >>>> much
> > > >> >>>>>>>>> time
> > > >> >>>>>>>>>>>> today - but tomorrow I will try to summarise the
> options
> > -
> > > >> >>> based
> > > >> >>>> on
> > > >> >>>>>>>>> some
> > > >> >>>>>>>>>>>> real code examples (to make it easier to digest). I
> think
> > > the
> > > >> >>>> best
> > > >> >>>>>>>>>>> approach
> > > >> >>>>>>>>>>>> will be to make a voting matrix in the AIP-21 Wiki page
> > > where
> > > >> >>>> people
> > > >> >>>>>>>>> will
> > > >> >>>>>>>>>>>> be able to state their preferences for each case (by
> > > adding a
> > > >> >>>> row to
> > > >> >>>>>>>>> the
> > > >> >>>>>>>>>>>> table). I think that might provide much better clarity
> > and
> > > a
> > > >> >>>> single
> > > >> >>>>>>>>> place
> > > >> >>>>>>>>>>>> which will show the current aggregated state of voting.
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>> However - as Ash also stated - some of the options are
> > > >> >>>>>>>> inter-dependent
> > > >> >>>>>>>>> so
> > > >> >>>>>>>>>>>> at the end we will have to adjust the results in case
> > they
> > > >> >> are
> > > >> >>>> not
> > > >> >>>>>>>>>>>> consistent - and make final voting on aggregated set
> of
> > > >> >> cases
> > > >> >>> -
> > > >> >>>>>>>> but I
> > > >> >>>>>>>>>>>> think in this case following "lasy consensus" - i,e. if
> > > noone
> > > >> >>>>>>>> objects
> > > >> >>>>>>>>> to
> > > >> >>>>>>>>>>>> final set of cases, we will proceed with it..
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>> Do you think that will work better ?
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>> J.
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>> Principal Software Engineer
> > > >> >>>>>>>>>>>> Phone: +48660796129
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> > > >> >>>>>>>>>>>> kamil.bregula@polidea.com> napisał:
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>>> Document is also available as AIP on wiki:
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
> > > >> >>>>>>>> <fokko@driesprong.frl
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>> I agree with all, this is a bit chaotic. For the sake
> > of
> > > >> >>>> clarity,
> > > >> >>>>>>>> I
> > > >> >>>>>>>>>>>>> would
> > > >> >>>>>>>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP.
> > > >> >> Create a
> > > >> >>>>>>>>>>> structural
> > > >> >>>>>>>>>>>>>> code improvement AIP and then and add a matrix of
> > > >> >> users/cases
> > > >> >>>>>>>> where
> > > >> >>>>>>>>>>>>>> everybody can add his vote per case. This gives also
> > the
> > > >> >>>>>>>> opportunity
> > > >> >>>>>>>>>>> for
> > > >> >>>>>>>>>>>>>> changing your vote on a specific case. WDYT?
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>> Cheers, Fokko
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <
> > > >> >>>> cixuuz@gmail.com>:
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>> I agree with Ash. It is clear to vote on each case
> > > rather
> > > >> >>> than
> > > >> >>>>>>>> all
> > > >> >>>>>>>>>>>>>>> together.
> > > >> >>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
> > > >> >>>>>>>> ash@apache.org>
> > > >> >>>>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>> A number of proposals overlap or add on to each
> > other,
> > > >> >> so I
> > > >> >>>>>>>> think
> > > >> >>>>>>>>>>>>> (?) a
> > > >> >>>>>>>>>>>>>>>> single vote makes sense.
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly
> > > what
> > > >> >> we
> > > >> >>>> are
> > > >> >>>>>>>>>>>>> voting
> > > >> >>>>>>>>>>>>>> on.
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <
> > > >> >>>> kaxilnaik@gmail.com>
> > > >> >>>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>>>>> I agree with Ash and instead of voting on all 7
> > Cases
> > > >> >>>> together,
> > > >> >>>>>>>>>>> how
> > > >> >>>>>>>>>>>>>>>>> about
> > > >> >>>>>>>>>>>>>>>>> separate vote emails for all the cases? Each email
> > can
> > > >> >>> have
> > > >> >>>> the
> > > >> >>>>>>>>>>>>>>>>> description
> > > >> >>>>>>>>>>>>>>>>> copied over from the doc.
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>> It would probably be a bit easier to track a
> > decision
> > > in
> > > >> >>> the
> > > >> >>>>>>>>>>> future
> > > >> >>>>>>>>>>>>> as
> > > >> >>>>>>>>>>>>>>>>> well.
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
> > > >> >>>>>>>> Jarek.Potiuk@polidea.com>
> > > >> >>>>>>>>>>>>>>>>> @Driesprong,
> > > >> >>>>>>>>>>>>>>>>> Fokko <fo...@driesprong.frl>
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
> > > >> >>>>>>>> kaxilnaik@gmail.com>
> > > >> >>>>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution
> > A *
> > > >> >>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py :
> > *Solution
> > > A
> > > >> >> *
> > > >> >>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
> > > >> >>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>> *Case #4* Separate namespace for resources :
> > > >> >>>>>>>>>>>>>>>>>> *Solution A*
> > > >> >>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
> > > >> >>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>> *Case #7* : *Solution A*
> > > >> >>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> > > >> >>>>>>>>>>>>>>>>> <dp...@gmail.com>
> > > >> >>>>>>>>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> I have tried to add some clarification to
> Jarek's
> > > >> >>> summary,
> > > >> >>>>>>>> but
> > > >> >>>>>>>>>>> I
> > > >> >>>>>>>>>>>>> am
> > > >> >>>>>>>>>>>>>>>>> a
> > > >> >>>>>>>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so
> > > hopefully
> > > >> >>>> Jarek
> > > >> >>>>>>>>>>> can
> > > >> >>>>>>>>>>>>>>>>> further
> > > >> >>>>>>>>>>>>>>>>>>> clarify.
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> According to my reading, in this motion cases
> > 4,5,6
> > > >> >> are
> > > >> >>>> all
> > > >> >>>>>>>>>>>>> either
> > > >> >>>>>>>>>>>>>>>>>>> proposal
> > > >> >>>>>>>>>>>>>>>>>>> *rejections* or otherwise have no effect, so
> > > attention
> > > >> >>>> can be
> > > >> >>>>>>>>>>>>>>>>> focused on
> > > >> >>>>>>>>>>>>>>>>>>> cases 1,2,3 and 7.
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> *Motion is to adopt the following:*
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
> > > >> >>>>>>>>>>>>>>>>>>> All objects will be moved out of contrib folder
> > and
> > > >> >> into
> > > >> >>>>>>>>>>>>> respective
> > > >> >>>>>>>>>>>>>>>>>>> non-contrib folder.
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix
> > from
> > > >> >>>> modules
> > > >> >>>>>>>>>>>>>>>>>>> Example:
> > > >> >>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator
> becomes
> > > >> >>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator
> > > >> >>>>>>>>>>>>>>>>>>> Example:
> > > >> >>>>>>>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
> > > >> >>>>>>>>>>>>> airflow.hooks.foo.FooHook
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc
> > prefixes
> > > >> >>>>>>>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
> > > >> >>>>>>>>>>>>>>>>>>> Example case is
> > > >> >>>>>>>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
> > > >> >>>>>>>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can
> > assume
> > > >> >> it
> > > >> >>> is
> > > >> >>>>>>>> now
> > > >> >>>>>>>>>>>>> named
> > > >> >>>>>>>>>>>>>>>>> like
> > > >> >>>>>>>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
> > > >> >>>>>>>>>>>>>>>>>>> So what is proposed for this case?
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> Case 4 - Solution B - *no change*
> > > >> >>>>>>>>>>>>>>>>>>> This proposal was about namespaces but the
> motion
> > is
> > > >> >> to
> > > >> >>>>>>>> reject
> > > >> >>>>>>>>>>>>> the
> > > >> >>>>>>>>>>>>>>>>>>> proposal.
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> Case 5 - Solution B - *no change*
> > > >> >>>>>>>>>>>>>>>>>>> The proposal was to remove class type suffix
> e.g.
> > > >> >> remove
> > > >> >>>> the
> > > >> >>>>>>>>>>>>>>>>> Operator in
> > > >> >>>>>>>>>>>>>>>>>>> FooOperator, but the motion is to reject this
> > > proposal
> > > >> >>> --
> > > >> >>>>>>>> i.e.
> > > >> >>>>>>>>>>>>>>>>> motion is
> > > >> >>>>>>>>>>>>>>>>>>> no
> > > >> >>>>>>>>>>>>>>>>>>> change on this case, i.e. we resolve to keep
> > > >> >> "Operator"
> > > >> >>>> (and
> > > >> >>>>>>>>>>>>>>>>> "Sensor")
> > > >> >>>>>>>>>>>>>>>>>>> suffixes on class names
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> Case 6 - *No change*
> > > >> >>>>>>>>>>>>>>>>>>> This case concerns object naming
> inconsistencies,
> > > but
> > > >> >>>> motion
> > > >> >>>>>>>>>>> does
> > > >> >>>>>>>>>>>>>>>>> not
> > > >> >>>>>>>>>>>>>>>>>>> enact
> > > >> >>>>>>>>>>>>>>>>>>> any specific proposal.
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template
> > for
> > > >> >>>>>>>>>>> deprecation
> > > >> >>>>>>>>>>>>>>>>> warnings
> > > >> >>>>>>>>>>>>>>>>>>> (instead of zope)
> > > >> >>>>>>>>>>>>>>>>>>> Example:
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> # in old_package/old_mod.py
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> import warnings
> > > >> >>>>>>>>>>>>>>>>>>> from new_package.new_mod import *
> > > >> >>>>>>>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
> > > >> >>>>>>>>>>>>> new_package.new_mod.
> > > >> >>>>>>>>>>>>>>>>>>> Import
> > > >> >>>>>>>>>>>>>>>>>>> of "
> > > >> >>>>>>>>>>>>>>>>>>> "old_package.old_mod will become unsupported in
> > > >> >> version
> > > >> >>>> 2",
> > > >> >>>>>>>>>>>>>>>>>>> DeprecationWarning, 2)
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> Reference implementation here:
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> I am very happy that this motion now gets rid of
> > > >> >>> contrib.
> > > >> >>>>>>>>>>> Thank
> > > >> >>>>>>>>>>>>> you
> > > >> >>>>>>>>>>>>>>>>>>> Sergio
> > > >> >>>>>>>>>>>>>>>>>>> for turning the tide with your effective
> > > argumentation
> > > >> >>> ;)
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash
> Berlin-Taylor
> > <
> > > >> >>>>>>>>>>>>> ash@apache.org>
> > > >> >>>>>>>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>> Could someone summarise what the proposed name
> > > >> >> changes
> > > >> >>>> are,
> > > >> >>>>>>>>>>>>> here,
> > > >> >>>>>>>>>>>>>>>>> in
> > > >> >>>>>>>>>>>>>>>>>>> this
> > > >> >>>>>>>>>>>>>>>>>>>> thread? Pointing at a google doc and only
> giving
> > > >> >>> partial
> > > >> >>>>>>>>>>>>> examples
> > > >> >>>>>>>>>>>>>>>>> is 1)
> > > >> >>>>>>>>>>>>>>>>>>> a
> > > >> >>>>>>>>>>>>>>>>>>>> bit difficult to quickly work out what we are
> > > voting
> > > >> >>> on,
> > > >> >>>> and
> > > >> >>>>>>>>>>> 2)
> > > >> >>>>>>>>>>>>>>>>> using a
> > > >> >>>>>>>>>>>>>>>>>>>> google doc isn't great for a longer term
> record.
> > > >> >>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>> Thanks,
> > > >> >>>>>>>>>>>>>>>>>>>> Ash
> > > >> >>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
> > > >> >>>>>>>>>>>>>>>>> <Ja...@polidea.com>
> > > >> >>>>>>>>>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>> I would like to recast the vote. Let's start
> > from
> > > 0
> > > >> >> :)
> > > >> >>>> . I
> > > >> >>>>>>>>>>>>> would
> > > >> >>>>>>>>>>>>>>>>> like
> > > >> >>>>>>>>>>>>>>>>>>> to
> > > >> >>>>>>>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least
> > > three
> > > >> >>> +1
> > > >> >>>>>>>>>>>>>>>>> (binding)
> > > >> >>>>>>>>>>>>>>>>>>> votes
> > > >> >>>>>>>>>>>>>>>>>>>> are
> > > >> >>>>>>>>>>>>>>>>>>>>> needed to pass. I marked in red the
> > modifications
> > > to
> > > >> >>> the
> > > >> >>>>>>>>>>>>>>>>> previous
> > > >> >>>>>>>>>>>>>>>>>>>> proposal.
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>> Here is the proposal (details here
> > > >> >>>>>>>>>>>>>>>>>>>>> <
> > > >> >>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>> )
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
> > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 2: A:
> > > >> >>> Airflow.operators.foo_operator.FooOperator
> > > >> >>>>>>>>>>>>> could
> > > >> >>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>
> > > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > > >> >>>>>>>>>>>>>>>>>>> could
> > > >> >>>>>>>>>>>>>>>>>>>>> become
> > > >> >>> airflow.gcp.operators.bigtable.BigTableOperator*
> > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
> > > >> >> suffixes
> > > >> >>> on
> > > >> >>>>>>>>>>>>> class
> > > >> >>>>>>>>>>>>>>>>>>> names*
> > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> > > >> >>> case-by-case
> > > >> >>>>>>>>>>>>> (and
> > > >> >>>>>>>>>>>>>>>>> my team
> > > >> >>>>>>>>>>>>>>>>>>>> can
> > > >> >>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > > >> >>>>>>>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better
> > IDE
> > > >> >>>>>>>>>>>>>>>>> integration)*
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>> This is my binding (+1) vote.
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk
> <
> > > >> >>>>>>>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
> > > >> >>>>>>>>>>>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>> Hey Felix,
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native
> > one
> > > >> >> as
> > > >> >>>> well
> > > >> >>>>>>>>>>> (I
> > > >> >>>>>>>>>>>>>>>>> use IDE
> > > >> >>>>>>>>>>>>>>>>>>>> quite
> > > >> >>>>>>>>>>>>>>>>>>>>>> often ;) )
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>> J.
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong,
> > Fokko
> > > >> >>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document
> > together.
> > > I
> > > >> >>>> wasn't
> > > >> >>>>>>>>>>>>> able
> > > >> >>>>>>>>>>>>>>>>> to
> > > >> >>>>>>>>>>>>>>>>>>>> respond
> > > >> >>>>>>>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops
> > > >> >>>> throughout
> > > >> >>>>>>>>>>>>> Europe
> > > >> >>>>>>>>>>>>>>>>> :-)
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything
> from
> > > >> >>> contrib
> > > >> >>>>>>>>>>> into
> > > >> >>>>>>>>>>>>> a
> > > >> >>>>>>>>>>>>>>>>> single
> > > >> >>>>>>>>>>>>>>>>>>>>>>> package.
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> To make it easier to the end-user, my
> > preference
> > > >> >>>> would be
> > > >> >>>>>>>>>>>>> to
> > > >> >>>>>>>>>>>>>>>>> get
> > > >> >>>>>>>>>>>>>>>>>>> rid of
> > > >> >>>>>>>>>>>>>>>>>>>>>>> the
> > > >> >>>>>>>>>>>>>>>>>>>>>>> contrib package altogether. What's in
> contrib
> > or
> > > >> >> not
> > > >> >>>> is a
> > > >> >>>>>>>>>>>>>>>>> tricky
> > > >> >>>>>>>>>>>>>>>>>>>> question,
> > > >> >>>>>>>>>>>>>>>>>>>>>>> I think this is already reflected in the
> > > document
> > > >> >>>> since
> > > >> >>>>>>>>>>>>>>>>> "maturity"
> > > >> >>>>>>>>>>>>>>>>>>> is
> > > >> >>>>>>>>>>>>>>>>>>>> in
> > > >> >>>>>>>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer
> > the
> > > >> >>>> operator
> > > >> >>>>>>>>>>>>> to
> > > >> >>>>>>>>>>>>>>>>> the
> > > >> >>>>>>>>>>>>>>>>>>> core
> > > >> >>>>>>>>>>>>>>>>>>>> or
> > > >> >>>>>>>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib
> > to
> > > >> >>>>>>>>>>> incubating
> > > >> >>>>>>>>>>>>>>>>> will
> > > >> >>>>>>>>>>>>>>>>>>> confuse
> > > >> >>>>>>>>>>>>>>>>>>>>>>> the
> > > >> >>>>>>>>>>>>>>>>>>>>>>> user even more. For people who aren't as
> known
> > > >> >> with
> > > >> >>>>>>>>>>>>> Airflow it
> > > >> >>>>>>>>>>>>>>>>> isn't
> > > >> >>>>>>>>>>>>>>>>>>>> that
> > > >> >>>>>>>>>>>>>>>>>>>>>>> transparent which operator lives where,
> > > especially
> > > >> >>> if
> > > >> >>>> you
> > > >> >>>>>>>>>>>>>>>>> don't use
> > > >> >>>>>>>>>>>>>>>>>>> an
> > > >> >>>>>>>>>>>>>>>>>>>> IDE
> > > >> >>>>>>>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it
> > is
> > > >> >>> worth
> > > >> >>>>>>>>>>>>> changing
> > > >> >>>>>>>>>>>>>>>>> the
> > > >> >>>>>>>>>>>>>>>>>>>> public
> > > >> >>>>>>>>>>>>>>>>>>>>>>> API just for a different name.
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote
> > with
> > > >> >>> this.
> > > >> >>>> It
> > > >> >>>>>>>>>>>>> will
> > > >> >>>>>>>>>>>>>>>>> make
> > > >> >>>>>>>>>>>>>>>>>>>> easier
> > > >> >>>>>>>>>>>>>>>>>>>>>> as we will not have to define other rules as
> > > those
> > > >> >>>> that we
> > > >> >>>>>>>>>>>>>>>>> already
> > > >> >>>>>>>>>>>>>>>>>>> have.
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B
> > (let's
> > > >> >>> say a
> > > >> >>>>>>>>>>> +0)
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> I like to search on Github with t, and then
> > > search
> > > >> >>> for
> > > >> >>>>>>>>>>>>>>>>> BashOperator.
> > > >> >>>>>>>>>>>>>>>>>>>> After
> > > >> >>>>>>>>>>>>>>>>>>>>>>> the change, you should search for
> > Operator/Bash.
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this,
> > from
> > > >> >>> your
> > > >> >>>>>>>>>>> mail
> > > >> >>>>>>>>>>>>> you
> > > >> >>>>>>>>>>>>>>>>>>> state: -
> > > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 2: B:
> > > >> >>> Airflow.operators.foo_operator.FooOperator
> > > >> >>>>>>>>>>>>> could
> > > >> >>>>>>>>>>>>>>>>> become
> > > >> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the
> B
> > > >> >>>> solution is
> > > >> >>>>>>>>>>>>>>>>> keeping it
> > > >> >>>>>>>>>>>>>>>>>>>> the
> > > >> >>>>>>>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
> > > >> >>>>>>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator
> > > *would
> > > >> >>> stay
> > > >> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think
> there
> > > is a
> > > >> >>>> little
> > > >> >>>>>>>>>>>>>>>>> confusion
> > > >> >>>>>>>>>>>>>>>>>>>> here.
> > > >> >>>>>>>>>>>>>>>>>>>>>> This case is not for changing BashOperator
> into
> > > >> >> Bash
> > > >> >>>> but
> > > >> >>>>>>>>>>> for
> > > >> >>>>>>>>>>>>>>>>> changing
> > > >> >>>>>>>>>>>>>>>>>>>> the
> > > >> >>>>>>>>>>>>>>>>>>>>>> *module* name not the *class* name. So
> > > >> >>>>>>>>>>>>>>>>> bash_operator.BashOperator ->
> > > >> >>>>>>>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search
> > for
> > > >> >>>>>>>>>>>>> BashOperator
> > > >> >>>>>>>>>>>>>>>>> in this
> > > >> >>>>>>>>>>>>>>>>>>>> case.
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide
> > > when
> > > >> >> it
> > > >> >>>> gets
> > > >> >>>>>>>>>>>>> its
> > > >> >>>>>>>>>>>>>>>>> own
> > > >> >>>>>>>>>>>>>>>>>>>> package
> > > >> >>>>>>>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there
> are
> > > >> >>>> companies
> > > >> >>>>>>>>>>>>> like
> > > >> >>>>>>>>>>>>>>>>>>>> Databricks
> > > >> >>>>>>>>>>>>>>>>>>>>>>> and Qubole which might also end up with
> their
> > > own
> > > >> >>>> package
> > > >> >>>>>>>>>>>>>>>>> because
> > > >> >>>>>>>>>>>>>>>>>>> their
> > > >> >>>>>>>>>>>>>>>>>>>>>>> integration is getting richer over time.
> > Moving
> > > >> >> this
> > > >> >>>> is a
> > > >> >>>>>>>>>>>>> bit
> > > >> >>>>>>>>>>>>>>>>>>> painful
> > > >> >>>>>>>>>>>>>>>>>>>>>>> because we need to deprecate the old path
> and
> > > move
> > > >> >>>>>>>>>>>>> everything
> > > >> >>>>>>>>>>>>>>>>> which
> > > >> >>>>>>>>>>>>>>>>>>>>>>> introduces noise into version control etc.
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed.
> > It's
> > > >> >> not
> > > >> >>> as
> > > >> >>>>>>>>>>>>>>>>> difficult as
> > > >> >>>>>>>>>>>>>>>>>>> it
> > > >> >>>>>>>>>>>>>>>>>>>>>> seems to group operators in packages and it
> has
> > > >> >>>> numerous
> > > >> >>>>>>>>>>>>>>>>> advantages.
> > > >> >>>>>>>>>>>>>>>>>>> The
> > > >> >>>>>>>>>>>>>>>>>>>>>> nice thing about deprecations - we can do
> them
> > in
> > > >> >> the
> > > >> >>>>>>>>>>>>>>>>> __init__.py of
> > > >> >>>>>>>>>>>>>>>>>>> the
> > > >> >>>>>>>>>>>>>>>>>>>>>> packages which causes the noise fairly small
> > (Git
> > > >> >>> will
> > > >> >>>>>>>>>>>>> actually
> > > >> >>>>>>>>>>>>>>>>>>> realise
> > > >> >>>>>>>>>>>>>>>>>>>>>> that the file has been moved and you can
> follow
> > > >> >> that.
> > > >> >>>>>>>>>>> Maybe
> > > >> >>>>>>>>>>>>> not
> > > >> >>>>>>>>>>>>>>>>> as
> > > >> >>>>>>>>>>>>>>>>>>> super
> > > >> >>>>>>>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but
> it's
> > > >> >> not a
> > > >> >>>>>>>>>>> rocket
> > > >> >>>>>>>>>>>>>>>>> science
> > > >> >>>>>>>>>>>>>>>>>>>> either.
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>> Yes.
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> ---
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> Apart from all, great work!
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix
> > > >> >> Uellendall
> > > >> >>>>>>>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
> > > >> >>>>>>>>>>>>>>>>>>>>>>>> :
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed
> > > change
> > > >> >>> of
> > > >> >>>>>>>>>>> these
> > > >> >>>>>>>>>>>>>>>>> cases.
> > > >> >>>>>>>>>>>>>>>>>>> So I
> > > >> >>>>>>>>>>>>>>>>>>>>>>> go
> > > >> >>>>>>>>>>>>>>>>>>>>>>>> with +1 (non-binding).
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up
> again
> > > and
> > > >> >>> your
> > > >> >>>>>>>>>>>>> intense
> > > >> >>>>>>>>>>>>>>>>> work
> > > >> >>>>>>>>>>>>>>>>>>>>>>> towards
> > > >> >>>>>>>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil
> for
> > > even
> > > >> >>>>>>>>>>> creating
> > > >> >>>>>>>>>>>>>>>>> this
> > > >> >>>>>>>>>>>>>>>>>>>>>>> document. I
> > > >> >>>>>>>>>>>>>>>>>>>>>>>> like how the code is getting more and more
> > > >> >>> consistent
> > > >> >>>>>>>>>>> and
> > > >> >>>>>>>>>>>>>>>>> clean :)
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> > > >> >>>>>>>>>>>>>>>>>>>>>>>> Felix
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>> -------- Original Message --------
> > > >> >>>>>>>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> This email is calling a vote on the
> changes
> > in
> > > >> >>>> import
> > > >> >>>>>>>>>>>>> paths.
> > > >> >>>>>>>>>>>>>>>>> It's
> > > >> >>>>>>>>>>>>>>>>>>>> been
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> discussed in
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week
> (July
> > > >> >> 30th
> > > >> >>>> 6pm
> > > >> >>>>>>>>>>>>> CEST),
> > > >> >>>>>>>>>>>>>>>>> and
> > > >> >>>>>>>>>>>>>>>>>>> at
> > > >> >>>>>>>>>>>>>>>>>>>>>>> least
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the
> > document
> > > >> >> from
> > > >> >>>>>>>>>>> Kamil
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> <
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> The proposed solution is:
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all
> > that
> > > >> >> are
> > > >> >>>>>>>>>>> "well"
> > > >> >>>>>>>>>>>>>>>>> tested
> > > >> >>>>>>>>>>>>>>>>>>> and
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or
> similar.*
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 2: B:
> > > >> >>>>>>>>>>> Airflow.operators.foo_operator.FooOperator
> > > >> >>>>>>>>>>>>>>>>> could
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>
> > > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > > >> >>>>>>>>>>>>>>>>>>>> could
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> become
> > > >> >>>> airflow.gcp.operators.bigtable.BigTableOperator*
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and
> "Sensor")
> > > >> >>>> suffixes
> > > >> >>>>>>>>>>> on
> > > >> >>>>>>>>>>>>>>>>> class
> > > >> >>>>>>>>>>>>>>>>>>> names*
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> > > >> >>>> case-by-case
> > > >> >>>>>>>>>>>>> (and
> > > >> >>>>>>>>>>>>>>>>> my
> > > >> >>>>>>>>>>>>>>>>>>> team
> > > >> >>>>>>>>>>>>>>>>>>>>>>> can
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> J.
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> --
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> |
> > > Principal
> > > >> >>>>>>>>>>> Software
> > > >> >>>>>>>>>>>>>>>>> Engineer
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
> > > >> >>>>>>>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <
> https://www.polidea.com/>
> > > >> >>>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>> --
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> > > >> >>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> |
> Principal
> > > >> >>>> Software
> > > >> >>>>>>>>>>>>>>>>> Engineer
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > > >> >>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>> --
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> > > >> >>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> |
> Principal
> > > >> >>> Software
> > > >> >>>>>>>>>>>>>> Engineer
> > > >> >>>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > > >> >>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > > >> >>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>> --
> > > >> >>>>>>>>>>>>>>>>>> *Kaxil Naik*
> > > >> >>>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > > >> >>>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer |
> > *Certified*
> > > >> >>> Apache
> > > >> >>>>>>>>>>> Spark
> > > >> >>>>>>>>>>>>> &
> > > >> >>>>>>>>>>>>>>>>> Neo4j
> > > >> >>>>>>>>>>>>>>>>>> Developer
> > > >> >>>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > >> >>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>> --
> > > >> >>>>>>>>>>>>>>>>> *Kaxil Naik*
> > > >> >>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > > >> >>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer |
> *Certified*
> > > >> >>> Apache
> > > >> >>>>>>>> Spark
> > > >> >>>>>>>>>>> &
> > > >> >>>>>>>>>>>>>>>>> Neo4j
> > > >> >>>>>>>>>>>>>>>>> Developer
> > > >> >>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> --
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Jarek Potiuk
> > > >> >>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> > > >> >>> Engineer
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > > >> >>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>>>>> --
> > > >> >>>>>>>> *Kaxil Naik*
> > > >> >>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > > >> >>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> > > >> >> Spark &
> > > >> >>>> Neo4j
> > > >> >>>>>>>> Developer
> > > >> >>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > >> >>>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> --
> > > >> >>>>>>>
> > > >> >>>>>>> Jarek Potiuk
> > > >> >>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> > > Engineer
> > > >> >>>>>>>
> > > >> >>>>>>> M: +48 660 796 129 <+48660796129>
> > > >> >>>>>>> [image: Polidea] <https://www.polidea.com/>
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>
> > > >> >>>>>> --
> > > >> >>>>>>
> > > >> >>>>>> Jarek Potiuk
> > > >> >>>>>> Polidea <https://www.polidea.com/> | Principal Software
> > Engineer
> > > >> >>>>>>
> > > >> >>>>>> M: +48 660 796 129 <+48660796129>
> > > >> >>>>>> [image: Polidea] <https://www.polidea.com/>
> > > >> >>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > > >>
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
Re: [VOTE] Changes in import paths
Posted by Jarek Potiuk <Ja...@polidea.com>.
Hey Fokko,
Really sorry, to miss this . I see now that you have not voted because you
thought it will be transferred automatically -it was thought as not and
additional option but combined replacement for (3+4) cases.
This was after I realised that there was a veto for namespaces from Ash and
I realised from it that namespaces were a bit confusing and overlapping
with grouping per-cloud-provider (see the voting "chapter
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting>").
I re-fined the choice back then - whether to not group them (A) or whether
to group first by type of entity or first by cloud provider (B-E in various
ways). Maybe I was not very clear in my message - apologies.
I now re-added your vote and also Ben's vote who also voted on leaving the
prefix and not grouping operators (Ben please correct me if I was wrong).
Still the result is 4:2 (binding), 6:2 (including non-binding) in favour of
grouping per cloud provider rather than using prefixes. And no vetoes.
Re - the point of which operators should be grouped. From the discussions -
I do not think it's easy to do a general and always applicable rule. But I
believe we agreed that we group the "obvious" choices - like "gcp",
"azure", "aws". It's for pure "Action" type operators - but still we leave
the cross-platform "transfer" ones or non-obvious ones in the old place
(which we keep as bag of "everything else"). Personally - I am all for
grouping the qubole/databricks operators in packages same as other cloud
providers. But with the "softness" of that rule this might (and should) be
an individual decision of people who maintain it. And it is quite OK - not
everything has to be decided upfront. I don't think we have a "hard" rule
now that covers all the cases and binds us. It would be very complex one if
we have it (and we can defer introducing it for later if needed especially
if it requires more than just renaming/moving but also architectural
changes - as Ash proposed for the transfer operators). But at least we can
decrease an entropy by grouping the obvious cases, and we can move forward
leaving the Airflow world a bit better, one step at a time.
J.
On Mon, Aug 5, 2019 at 3:58 PM Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> Hi Jarek,
>
> Just wondering how the vote is unanimous. On case 3 I've voted for B. I
> still believe introducing namespaces is hard to maintain, as I explained
> earlier: Mostly because it isn't as trivial to decide when it gets its own
> package or not. Besides the cloud providers, there are companies like
> Databricks and Qubole which might also end up with their own package
> because their integration is getting richer over time. Moving this is a bit
> painful because we need to deprecate the old path and move everything which
> introduces noise into version control etc.
>
> Adding an additional option to the list, does not invalidate the older
> options, right?
>
> Cheers, Fokko
>
>
> Op ma 5 aug. 2019 om 15:43 schreef Jarek Potiuk <Jarek.Potiuk@polidea.com
> >:
>
> > Seems that after this months of discussion we got an unanimous voting
> > result. I summarised it in the AIP-21
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > >
> > and
> > changed its status to "Accepted". Thanks for all your feedback! We will
> > start soon moving GCP operators following those rules and we will resolve
> > any initial problems with those - then we might provide some additional
> > learnings and see if we can easily (probably semi-automatically) move all
> > other operators.
> >
> > Case 1
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#1airflow.contrib.{resources}
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%231airflow.contrib.%7Bresources%7D>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%231airflow.contrib.%7Bresources%7D
> >
> > >
> >
> > What to do with the "contrib" folder
> >
> > Case 2
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#2git*_{operator/sensor}{/s}.py
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%232git*_%7Boperator/sensor%7D%7B/s%7D.py>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%232git*_%7Boperator/sensor%7D%7B/s%7D.py
> >
> > >
> >
> > Drop modules *_operator suffix
> >
> > Case 3
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#3{aws/azure/gcp}_*
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%233%7Baws/azure/gcp%7D_*>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%233%7Baws/azure/gcp%7D_*
> >
> > >
> > + Case 4
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#4Separatenamespaceforresources
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%234Separatenamespaceforresources>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%234Separatenamespaceforresources
> >
> > >
> >
> > Grouping Cloud Providers operators/sensors/hooks
> >
> > Case 5
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#5*Operator
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%235*Operator>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%235*Operator
> >
> > >
> >
> > *Operator *Sensor *Hook in class name
> >
> > Case 6
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#6Otherisolatedcases
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%236Otherisolatedcases>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%236Otherisolatedcases
> >
> > >
> >
> > Isolated cases
> >
> > Case 7
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#7
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%237>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%237
> >
> > >
> >
> > Deprecation method
> >
> > *A.* Move everything "contrib" → "airflow"
> >
> > *A.* Drop the suffix.
> >
> > Example:
> >
> > - *airflow.operator.gcp_bigtable_operator.py*
> > becomes *airflow.operator.gcp_bigtable.py
> > <http://airflow.operator.gcp_bigtable.py>*.
> >
> > *D. * Group operators/sensors/hooks in
> > *airflow/<PROVIDER>*/operators(sensors,
> > hooks). Non-cloud provider ones are moved to
> > airflow/operators(sensors/hooks).
> > *Drop the prefix.*
> >
> > -
> > *airflow/contrib/operators/sns_publish_operator.py
> > becomes airflow/aws/operators/**sns_publish_operator.py*
> >
> >
> > - *airflow/contrib/operators/dataproc_operator.py*
> > becomes *airflow/gcp/operators/dataproc_operator.py*
> >
> >
> >
> > -
> > *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
> > *gcp/operators/bigtable_operator.py*
> >
> >
> >
> > -
> > *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > *operators/ssh_operator.py*
> >
> > *B.* No change - keep Operator/Sensor suffix in class name.
> >
> > *A.* Make individual decisions of renames for operators that do not
> follow
> > common conventions used for other operators.
> >
> > Consistency trumps compatibility.
> >
> > Examples:
> >
> > *DataProcHadoopOperator*
> >
> > renamed to:
> >
> > *DataprocHadoopOperator*
> > *A.* Native python method (with better IDE support and more flexible
> but a
> > bit more verbose)
> >
> > J.
> >
> > On Wed, Jul 31, 2019 at 10:30 AM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> > > Yep. Agree with Ash on it. There are a number of 'action' operators
> > > specific for cloud providers and these should be our target. The
> transfer
> > > ones require another AIP (A lot of that already discussed in AIP-8
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
> > > ).
> > >
> > > J.
> > >
> > > Principal Software Engineer
> > > Phone: +48660796129
> > >
> > > śr., 31 lip 2019, 10:01 użytkownik Ash Berlin-Taylor <as...@apache.org>
> > > napisał:
> > >
> > >> This is a good idea for now. I'm also not overly concerned about these
> > >> few non-cloud examples - FTPtoS3Operator can stay where it is and
> > doesn't
> > >> have to move under 'aws.' to my mind.
> > >>
> > >> Longer term I'd like to go back to making the
> "transfer/copy/transform"
> > >> operators "composable" so that we can have a single DataCopyOperator,
> > and
> > >> give it a source and destination, and it uses hooks to do the work.
> > This is
> > >> not a small undertaking though to do well and efficiently though.
> > >>
> > >> -ash
> > >>
> > >> > On 30 Jul 2019, at 20:54, Tomasz Urbaszek <
> > tomasz.urbaszek@polidea.com>
> > >> wrote:
> > >> >
> > >> > Maybe we can put all those AtoB operators under one name like
> > >> “transfer”,
> > >> > then it would be easier to look for such operator?
> > >> >
> > >> > Best,
> > >> > Tomek
> > >> >
> > >> > On Tue, 30 Jul 2019 at 21:39, Chen Tong <ci...@gmail.com> wrote:
> > >> >
> > >> >> Daniel mentioned a good point. Such composed operator may also
> > involves
> > >> >> both cloud and non-cloud provider saying FTPtoS3Operator. Should it
> > in
> > >> AWS
> > >> >> folder?
> > >> >>
> > >> >> Also, saying in the future, another cloud provider is growing large
> > >> enough.
> > >> >> Will we rename all plugins related to this provider? What's the
> > >> criteria we
> > >> >> say it's big enough? And option D gives an impression like these
> > tools
> > >> may
> > >> >> be maintained and supported by the Cloud provider. I am not sure if
> > it
> > >> will
> > >> >> be a truth or not.
> > >> >>
> > >> >> Best,
> > >> >>
> > >> >>
> > >> >> On Tue, Jul 30, 2019 at 11:18 AM Daniel Standish <
> > dpstandish@gmail.com
> > >> >
> > >> >> wrote:
> > >> >>
> > >> >>> One wrinkle with case 3+4 option D is inter-provider operators.
> > >> Mainly
> > >> >>> it's storage I think e.g. XToS3Operator or XToGCSOperator where
> the
> > X
> > >> is
> > >> >> a
> > >> >>> service some different provider.
> > >> >>>
> > >> >>> Maybe the rule should be to locate the operator according to the
> > first
> > >> >>> provider referenced. So e.g. s3_to_gcs_transfer_operator.py would
> > go
> > >> to
> > >> >>> aws.
> > >> >>>
> > >> >>>
> > >> >>> On Tue, Jul 30, 2019 at 6:21 AM Kamil Breguła <
> > >> kamil.bregula@polidea.com
> > >> >>>
> > >> >>> wrote:
> > >> >>>
> > >> >>>> Yes. All changes will be backwards compatible. In the case of
> using
> > >> >>>> the old path, a message containing a proposal for change will be
> > >> >>>> reported to the user.
> > >> >>>>
> > >> >>>> I prepared an example of how to change the name of a class in a
> > case
> > >> >>>> with the use of a native solution.
> > >> >>>> Source code:
> > >> >>>>
> > >> >>
> > >>
> > https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename
> > >> >>>> Preview from IDE: https://imgur.com/a/45NxS5W
> > >> >>>>
> > >> >>>> On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <
> ash@apache.org>
> > >> >>> wrote:
> > >> >>>>>
> > >> >>>>> Just cos I'm not sure it's _explicitly_ stated, but all of the
> > moves
> > >> >>>> will have a deprecation of the old name right?
> > >> >>>>>
> > >> >>>>> 3+4 case D gets my vote too.
> > >> >>>>>
> > >> >>>>> -a
> > >> >>>>>
> > >> >>>>>> On 30 Jul 2019, at 09:58, Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >
> > >> >>>> wrote:
> > >> >>>>>>
> > >> >>>>>> I went ahead and updated the page (just to speed it up) as I
> > think
> > >> >> it
> > >> >>>>>> really makes sense to join those two cases (and I do not see
> any
> > >> >>>> drawbacks
> > >> >>>>>> - I think the options we have cover all possible approaches)
> and
> > we
> > >> >>> can
> > >> >>>>>> always go back if we need to.
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > >> >>>>>>
> > >> >>>>>> My vote is *D*.
> > >> >>>>>>
> > >> >>>>>> - airflow/contrib/operators/gcp_bigtable_operator.py →
> > >> >>> airflow/gcp
> > >> >>>>>> /operators/bigtable_operator.py
> > >> >>>>>> - airflow/contrib/operators/ssh_operator.py ->
> > >> >>>>>> airflow/operators/ssh_operator.py
> > >> >>>>>>
> > >> >>>>>> I updated the page with my vote.
> > >> >>>>>> I propose that we vote till Friday on that one (all the rest is
> > >> >>> already
> > >> >>>>>> agreed).
> > >> >>>>>>
> > >> >>>>>> J.
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <
> > >> >>> Jarek.Potiuk@polidea.com
> > >> >>>>>
> > >> >>>>>> wrote:
> > >> >>>>>>
> > >> >>>>>>> I think almost everyone voted and we have almost perfect
> > >> >> consensus.
> > >> >>>> We all
> > >> >>>>>>> agree amongst other on moving all operators out of contrib
> > >> >> (Great!).
> > >> >>>>>>>
> > >> >>>>>>> The only doubts are for *Case 3* (Cloud provider prefix) and
> > *Case
> > >> >>> 4*
> > >> >>>>>>> (Using Namespaces).
> > >> >>>>>>> I think there was actually an overlap between those two. Also
> > >> >> Ash's
> > >> >>>>>>> comment/veto on that is quite clear.
> > >> >>>>>>> So I have final (I hope!) proposal to join those two cases and
> > >> >> have
> > >> >>>> one
> > >> >>>>>>> voting (till Friday) only on that.
> > >> >>>>>>>
> > >> >>>>>>> I will update the doc if you all agree with me that makes more
> > >> >> sense
> > >> >>>> as
> > >> >>>>>>> single case to vote on:
> > >> >>>>>>>
> > >> >>>>>>> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
> > >> >>>>>>>
> > >> >>>>>>> *Option A*. Keep operators/sensors/hooks in
> > >> >>> airflow/operators(sensors,
> > >> >>>>>>> hooks) and keep/add prefixes in file names.
> > >> >>>>>>>
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py becomes
> > >> >>>>>>> airflow/operators/**aws_sns_publish_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > >> >>>>>>> becomes *airflow/operators/gcp_dataproc_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes
> *airflow/*
> > >> >>>>>>> *hooks/gcp_bigtable_hook.py*
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> *airflow/*
> > >> >>>>>>> *operators/ssh_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> *Option B*. Group operators/sensors/hooks in
> > >> >>>> airflow/operators(sensors,
> > >> >>>>>>> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
> > >> >>>>>>> airflow/operators(sensors/hooks), Drop the prefix.
> > >> >>>>>>>
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> > >> >>>>>>> becomes airflow/operators/**aws/sns_publish_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > >> >>>>>>> becomes *airflow/operators/gcp/dataproc_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> > >> >>>> *airflow/*
> > >> >>>>>>> *operators/gcp/bigtable_operator.py*
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> *airflow/*
> > >> >>>>>>> *operators/ssh_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> *Option C*. Group operators/sensors/hooks in
> > >> >>>> airflow/operators(sensors,
> > >> >>>>>>> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
> > >> >>>>>>> airflow/operators(sensors/hooks), Keep the prefix.
> > >> >>>>>>>
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> > >> >>>>>>> becomes airflow/operators/**aws/aws_sns_publish_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > >> >>>>>>> becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> > >> >>>> *airflow/*
> > >> >>>>>>> *operators/gcp/gcp_bigtable_operator.py*
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> *airflow/*
> > >> >>>>>>> *operators/ssh_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> *Option D.* Group operators/sensors/hooks in
> > >> >>>> *airflow/<PROVIDER>*/operators(sensors,
> > >> >>>>>>> hooks). Non-cloud provider ones are moved to
> > >> >>>>>>> airflow/operators(sensors/hooks). Drop the prefix.
> > >> >>>>>>>
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> > >> >>>>>>> becomes airflow/aws/operators/**sns_publish_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > >> >>>>>>> becomes *airflow/gcp/operators/dataproc_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> > >> >>>> *airflow/*
> > >> >>>>>>> *gcp/operators/bigtable_operator.py*
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> *airflow/*
> > >> >>>>>>> *operators/ssh_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> *Option E.* Group operators/sensors/hooks in
> > >> >>>> *airflow/<PROVIDER>*/operators(sensors,
> > >> >>>>>>> hooks). Non-cloud provider ones are moved to
> > >> >>>>>>> airflow/operators(sensors/hooks).* Keep the prefix*.
> > >> >>>>>>>
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> > >> >>>>>>> becomes airflow/aws/operators/aws_**sns_publish_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> > >> >>>>>>> becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> > >> >>>> *airflow/*
> > >> >>>>>>> *gcp/operators/gcp_bigtable_operator.py*
> > >> >>>>>>> -
> > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes
> *airflow/*
> > >> >>>>>>> *operators/ssh_operator.py*
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> Shall I proceed with updating the page ?
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> J.
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <
> > kaxilnaik@gmail.com>
> > >> >>>> wrote:
> > >> >>>>>>>
> > >> >>>>>>>> Thanks Jarek, adding my vote.
> > >> >>>>>>>>
> > >> >>>>>>>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <
> > >> >> ash@apache.org>
> > >> >>>> wrote:
> > >> >>>>>>>>
> > >> >>>>>>>>> Thanks Jarek, much clearer. I have voted there too.
> > >> >>>>>>>>>
> > >> >>>>>>>>> -ash
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko
> > >> >> <fokko@driesprong.frl
> > >> >>>>
> > >> >>>>>>>>> wrote:
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Thanks Jarek, the Wiki works much better for me. I've added
> > my
> > >> >>> vote
> > >> >>>>>>>> there
> > >> >>>>>>>>>> as well.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> You've convinced me on the second case. I've changed my
> vote
> > on
> > >> >>>> Case 2
> > >> >>>>>>>>> from
> > >> >>>>>>>>>> A to B :-)
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Maybe we should leave the vote open a bit longer since it
> > >> >>> involves
> > >> >>>>>>>> quite
> > >> >>>>>>>>>> some reading, and I would like to get some proper feedback
> > and
> > >> >>>>>>>> opinions
> > >> >>>>>>>>> on
> > >> >>>>>>>>>> this. Plus I only see two votes right now. If we don't
> think
> > >> >> this
> > >> >>>>>>>>> through,
> > >> >>>>>>>>>> it might be that we're having this vote again in the near
> > >> >> future
> > >> >>>> :-)
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Cheers, Fokko
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
> > >> >>>>>>>>> Jarek.Potiuk@polidea.com>:
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>> I updated the AIP-21 proposal in the way that should
> answer
> > >> >> all
> > >> >>>> the
> > >> >>>>>>>>>>> concerns in this thread.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> NOTE! There is an action for all of those who are
> > interested:
> > >> >>>> Please
> > >> >>>>>>>>>>> fill-in your voting in the voting table till Tuesday 30th
> of
> > >> >>> July
> > >> >>>> 6pm
> > >> >>>>>>>>> CEST
> > >> >>>>>>>>>>> (5pm BST, 9 am PST).
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I created a summary of the proposal
> > >> >>>>>>>>>>> <
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>> in table form - which contains also examples for each
> case.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I added a voting table
> > >> >>>>>>>>>>> <
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>> where you should add your votes:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I propose this voting mechanism:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST
> > (5
> > >> >>> pm
> > >> >>>>>>>> BST,
> > >> >>>>>>>>> 9am
> > >> >>>>>>>>>>> PST) .
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> After the choice, the final consistent set of choices will
> > be
> > >> >>>>>>>> announced
> > >> >>>>>>>>>>> (taking into account majority of binding vote, also
> > including
> > >> >>>>>>>> potential
> > >> >>>>>>>>>>> vetos and consistency between the choices. Non-binding
> votes
> > >> >>> will
> > >> >>>> be
> > >> >>>>>>>>> taken
> > >> >>>>>>>>>>> into account in case there is a draw. The final set of
> > choices
> > >> >>>> will
> > >> >>>>>>>> be
> > >> >>>>>>>>>>> announced at devlist thread
> > >> >>>>>>>>>>> <
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>> after
> > >> >>>>>>>>>>> the voting completes.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Unless there is a veto raised to the final proposal, the
> > final
> > >> >>>>>>>> proposal
> > >> >>>>>>>>> is
> > >> >>>>>>>>>>> accepted by Lazy Consensus
> > >> >>>>>>>>>>> <
> https://community.apache.org/committers/lazyConsensus.html
> > >
> > >> >>> on
> > >> >>>>>>>>> *Friday*
> > >> >>>>>>>>>>> 02
> > >> >>>>>>>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Please let me know if what I proposed can be further
> > improved
> > >> >> to
> > >> >>>>>>>> avoid
> > >> >>>>>>>>>>> chaos.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> J.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
> > >> >>>>>>>> Jarek.Potiuk@polidea.com
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>> Ok. I see that indeed voting could have been organised a
> > bit
> > >> >>>> better
> > >> >>>>>>>> -
> > >> >>>>>>>>> dev
> > >> >>>>>>>>>>>> list does not seem to cope well with such a complex case
> > :).
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> As Kamil noticed - we already have a formal AIP-21
> > >> >>>>>>>>>>>> <
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> in the Wiki - which I should have mentioned before. I
> have
> > >> >> not
> > >> >>>> much
> > >> >>>>>>>>> time
> > >> >>>>>>>>>>>> today - but tomorrow I will try to summarise the options
> -
> > >> >>> based
> > >> >>>> on
> > >> >>>>>>>>> some
> > >> >>>>>>>>>>>> real code examples (to make it easier to digest). I think
> > the
> > >> >>>> best
> > >> >>>>>>>>>>> approach
> > >> >>>>>>>>>>>> will be to make a voting matrix in the AIP-21 Wiki page
> > where
> > >> >>>> people
> > >> >>>>>>>>> will
> > >> >>>>>>>>>>>> be able to state their preferences for each case (by
> > adding a
> > >> >>>> row to
> > >> >>>>>>>>> the
> > >> >>>>>>>>>>>> table). I think that might provide much better clarity
> and
> > a
> > >> >>>> single
> > >> >>>>>>>>> place
> > >> >>>>>>>>>>>> which will show the current aggregated state of voting.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> However - as Ash also stated - some of the options are
> > >> >>>>>>>> inter-dependent
> > >> >>>>>>>>> so
> > >> >>>>>>>>>>>> at the end we will have to adjust the results in case
> they
> > >> >> are
> > >> >>>> not
> > >> >>>>>>>>>>>> consistent - and make final voting on aggregated set of
> > >> >> cases
> > >> >>> -
> > >> >>>>>>>> but I
> > >> >>>>>>>>>>>> think in this case following "lasy consensus" - i,e. if
> > noone
> > >> >>>>>>>> objects
> > >> >>>>>>>>> to
> > >> >>>>>>>>>>>> final set of cases, we will proceed with it..
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> Do you think that will work better ?
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> J.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> Principal Software Engineer
> > >> >>>>>>>>>>>> Phone: +48660796129
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> > >> >>>>>>>>>>>> kamil.bregula@polidea.com> napisał:
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>> Document is also available as AIP on wiki:
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
> > >> >>>>>>>> <fokko@driesprong.frl
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> I agree with all, this is a bit chaotic. For the sake
> of
> > >> >>>> clarity,
> > >> >>>>>>>> I
> > >> >>>>>>>>>>>>> would
> > >> >>>>>>>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP.
> > >> >> Create a
> > >> >>>>>>>>>>> structural
> > >> >>>>>>>>>>>>>> code improvement AIP and then and add a matrix of
> > >> >> users/cases
> > >> >>>>>>>> where
> > >> >>>>>>>>>>>>>> everybody can add his vote per case. This gives also
> the
> > >> >>>>>>>> opportunity
> > >> >>>>>>>>>>> for
> > >> >>>>>>>>>>>>>> changing your vote on a specific case. WDYT?
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> Cheers, Fokko
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <
> > >> >>>> cixuuz@gmail.com>:
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> I agree with Ash. It is clear to vote on each case
> > rather
> > >> >>> than
> > >> >>>>>>>> all
> > >> >>>>>>>>>>>>>>> together.
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
> > >> >>>>>>>> ash@apache.org>
> > >> >>>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> A number of proposals overlap or add on to each
> other,
> > >> >> so I
> > >> >>>>>>>> think
> > >> >>>>>>>>>>>>> (?) a
> > >> >>>>>>>>>>>>>>>> single vote makes sense.
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly
> > what
> > >> >> we
> > >> >>>> are
> > >> >>>>>>>>>>>>> voting
> > >> >>>>>>>>>>>>>> on.
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <
> > >> >>>> kaxilnaik@gmail.com>
> > >> >>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>>>> I agree with Ash and instead of voting on all 7
> Cases
> > >> >>>> together,
> > >> >>>>>>>>>>> how
> > >> >>>>>>>>>>>>>>>>> about
> > >> >>>>>>>>>>>>>>>>> separate vote emails for all the cases? Each email
> can
> > >> >>> have
> > >> >>>> the
> > >> >>>>>>>>>>>>>>>>> description
> > >> >>>>>>>>>>>>>>>>> copied over from the doc.
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>> It would probably be a bit easier to track a
> decision
> > in
> > >> >>> the
> > >> >>>>>>>>>>> future
> > >> >>>>>>>>>>>>> as
> > >> >>>>>>>>>>>>>>>>> well.
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
> > >> >>>>>>>> Jarek.Potiuk@polidea.com>
> > >> >>>>>>>>>>>>>>>>> @Driesprong,
> > >> >>>>>>>>>>>>>>>>> Fokko <fo...@driesprong.frl>
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
> > >> >>>>>>>> kaxilnaik@gmail.com>
> > >> >>>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution
> A *
> > >> >>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py :
> *Solution
> > A
> > >> >> *
> > >> >>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
> > >> >>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>> *Case #4* Separate namespace for resources :
> > >> >>>>>>>>>>>>>>>>>> *Solution A*
> > >> >>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
> > >> >>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>> *Case #7* : *Solution A*
> > >> >>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> > >> >>>>>>>>>>>>>>>>> <dp...@gmail.com>
> > >> >>>>>>>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> I have tried to add some clarification to Jarek's
> > >> >>> summary,
> > >> >>>>>>>> but
> > >> >>>>>>>>>>> I
> > >> >>>>>>>>>>>>> am
> > >> >>>>>>>>>>>>>>>>> a
> > >> >>>>>>>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so
> > hopefully
> > >> >>>> Jarek
> > >> >>>>>>>>>>> can
> > >> >>>>>>>>>>>>>>>>> further
> > >> >>>>>>>>>>>>>>>>>>> clarify.
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> According to my reading, in this motion cases
> 4,5,6
> > >> >> are
> > >> >>>> all
> > >> >>>>>>>>>>>>> either
> > >> >>>>>>>>>>>>>>>>>>> proposal
> > >> >>>>>>>>>>>>>>>>>>> *rejections* or otherwise have no effect, so
> > attention
> > >> >>>> can be
> > >> >>>>>>>>>>>>>>>>> focused on
> > >> >>>>>>>>>>>>>>>>>>> cases 1,2,3 and 7.
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> *Motion is to adopt the following:*
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
> > >> >>>>>>>>>>>>>>>>>>> All objects will be moved out of contrib folder
> and
> > >> >> into
> > >> >>>>>>>>>>>>> respective
> > >> >>>>>>>>>>>>>>>>>>> non-contrib folder.
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix
> from
> > >> >>>> modules
> > >> >>>>>>>>>>>>>>>>>>> Example:
> > >> >>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
> > >> >>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator
> > >> >>>>>>>>>>>>>>>>>>> Example:
> > >> >>>>>>>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
> > >> >>>>>>>>>>>>> airflow.hooks.foo.FooHook
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc
> prefixes
> > >> >>>>>>>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
> > >> >>>>>>>>>>>>>>>>>>> Example case is
> > >> >>>>>>>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
> > >> >>>>>>>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can
> assume
> > >> >> it
> > >> >>> is
> > >> >>>>>>>> now
> > >> >>>>>>>>>>>>> named
> > >> >>>>>>>>>>>>>>>>> like
> > >> >>>>>>>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
> > >> >>>>>>>>>>>>>>>>>>> So what is proposed for this case?
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> Case 4 - Solution B - *no change*
> > >> >>>>>>>>>>>>>>>>>>> This proposal was about namespaces but the motion
> is
> > >> >> to
> > >> >>>>>>>> reject
> > >> >>>>>>>>>>>>> the
> > >> >>>>>>>>>>>>>>>>>>> proposal.
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> Case 5 - Solution B - *no change*
> > >> >>>>>>>>>>>>>>>>>>> The proposal was to remove class type suffix e.g.
> > >> >> remove
> > >> >>>> the
> > >> >>>>>>>>>>>>>>>>> Operator in
> > >> >>>>>>>>>>>>>>>>>>> FooOperator, but the motion is to reject this
> > proposal
> > >> >>> --
> > >> >>>>>>>> i.e.
> > >> >>>>>>>>>>>>>>>>> motion is
> > >> >>>>>>>>>>>>>>>>>>> no
> > >> >>>>>>>>>>>>>>>>>>> change on this case, i.e. we resolve to keep
> > >> >> "Operator"
> > >> >>>> (and
> > >> >>>>>>>>>>>>>>>>> "Sensor")
> > >> >>>>>>>>>>>>>>>>>>> suffixes on class names
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> Case 6 - *No change*
> > >> >>>>>>>>>>>>>>>>>>> This case concerns object naming inconsistencies,
> > but
> > >> >>>> motion
> > >> >>>>>>>>>>> does
> > >> >>>>>>>>>>>>>>>>> not
> > >> >>>>>>>>>>>>>>>>>>> enact
> > >> >>>>>>>>>>>>>>>>>>> any specific proposal.
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template
> for
> > >> >>>>>>>>>>> deprecation
> > >> >>>>>>>>>>>>>>>>> warnings
> > >> >>>>>>>>>>>>>>>>>>> (instead of zope)
> > >> >>>>>>>>>>>>>>>>>>> Example:
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> # in old_package/old_mod.py
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> import warnings
> > >> >>>>>>>>>>>>>>>>>>> from new_package.new_mod import *
> > >> >>>>>>>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
> > >> >>>>>>>>>>>>> new_package.new_mod.
> > >> >>>>>>>>>>>>>>>>>>> Import
> > >> >>>>>>>>>>>>>>>>>>> of "
> > >> >>>>>>>>>>>>>>>>>>> "old_package.old_mod will become unsupported in
> > >> >> version
> > >> >>>> 2",
> > >> >>>>>>>>>>>>>>>>>>> DeprecationWarning, 2)
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> Reference implementation here:
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> I am very happy that this motion now gets rid of
> > >> >>> contrib.
> > >> >>>>>>>>>>> Thank
> > >> >>>>>>>>>>>>> you
> > >> >>>>>>>>>>>>>>>>>>> Sergio
> > >> >>>>>>>>>>>>>>>>>>> for turning the tide with your effective
> > argumentation
> > >> >>> ;)
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor
> <
> > >> >>>>>>>>>>>>> ash@apache.org>
> > >> >>>>>>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>> Could someone summarise what the proposed name
> > >> >> changes
> > >> >>>> are,
> > >> >>>>>>>>>>>>> here,
> > >> >>>>>>>>>>>>>>>>> in
> > >> >>>>>>>>>>>>>>>>>>> this
> > >> >>>>>>>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving
> > >> >>> partial
> > >> >>>>>>>>>>>>> examples
> > >> >>>>>>>>>>>>>>>>> is 1)
> > >> >>>>>>>>>>>>>>>>>>> a
> > >> >>>>>>>>>>>>>>>>>>>> bit difficult to quickly work out what we are
> > voting
> > >> >>> on,
> > >> >>>> and
> > >> >>>>>>>>>>> 2)
> > >> >>>>>>>>>>>>>>>>> using a
> > >> >>>>>>>>>>>>>>>>>>>> google doc isn't great for a longer term record.
> > >> >>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>> Thanks,
> > >> >>>>>>>>>>>>>>>>>>>> Ash
> > >> >>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
> > >> >>>>>>>>>>>>>>>>> <Ja...@polidea.com>
> > >> >>>>>>>>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>> I would like to recast the vote. Let's start
> from
> > 0
> > >> >> :)
> > >> >>>> . I
> > >> >>>>>>>>>>>>> would
> > >> >>>>>>>>>>>>>>>>> like
> > >> >>>>>>>>>>>>>>>>>>> to
> > >> >>>>>>>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least
> > three
> > >> >>> +1
> > >> >>>>>>>>>>>>>>>>> (binding)
> > >> >>>>>>>>>>>>>>>>>>> votes
> > >> >>>>>>>>>>>>>>>>>>>> are
> > >> >>>>>>>>>>>>>>>>>>>>> needed to pass. I marked in red the
> modifications
> > to
> > >> >>> the
> > >> >>>>>>>>>>>>>>>>> previous
> > >> >>>>>>>>>>>>>>>>>>>> proposal.
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>> Here is the proposal (details here
> > >> >>>>>>>>>>>>>>>>>>>>> <
> > >> >>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>> )
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
> > >> >>>>>>>>>>>>>>>>>>>>> - *Case 2: A:
> > >> >>> Airflow.operators.foo_operator.FooOperator
> > >> >>>>>>>>>>>>> could
> > >> >>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > >> >>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>
> > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > >> >>>>>>>>>>>>>>>>>>> could
> > >> >>>>>>>>>>>>>>>>>>>>> become
> > >> >>> airflow.gcp.operators.bigtable.BigTableOperator*
> > >> >>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > >> >>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
> > >> >> suffixes
> > >> >>> on
> > >> >>>>>>>>>>>>> class
> > >> >>>>>>>>>>>>>>>>>>> names*
> > >> >>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> > >> >>> case-by-case
> > >> >>>>>>>>>>>>> (and
> > >> >>>>>>>>>>>>>>>>> my team
> > >> >>>>>>>>>>>>>>>>>>>> can
> > >> >>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > >> >>>>>>>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better
> IDE
> > >> >>>>>>>>>>>>>>>>> integration)*
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>> This is my binding (+1) vote.
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> > >> >>>>>>>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
> > >> >>>>>>>>>>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>> Hey Felix,
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native
> one
> > >> >> as
> > >> >>>> well
> > >> >>>>>>>>>>> (I
> > >> >>>>>>>>>>>>>>>>> use IDE
> > >> >>>>>>>>>>>>>>>>>>>> quite
> > >> >>>>>>>>>>>>>>>>>>>>>> often ;) )
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>> J.
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong,
> Fokko
> > >> >>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document
> together.
> > I
> > >> >>>> wasn't
> > >> >>>>>>>>>>>>> able
> > >> >>>>>>>>>>>>>>>>> to
> > >> >>>>>>>>>>>>>>>>>>>> respond
> > >> >>>>>>>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops
> > >> >>>> throughout
> > >> >>>>>>>>>>>>> Europe
> > >> >>>>>>>>>>>>>>>>> :-)
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from
> > >> >>> contrib
> > >> >>>>>>>>>>> into
> > >> >>>>>>>>>>>>> a
> > >> >>>>>>>>>>>>>>>>> single
> > >> >>>>>>>>>>>>>>>>>>>>>>> package.
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> To make it easier to the end-user, my
> preference
> > >> >>>> would be
> > >> >>>>>>>>>>>>> to
> > >> >>>>>>>>>>>>>>>>> get
> > >> >>>>>>>>>>>>>>>>>>> rid of
> > >> >>>>>>>>>>>>>>>>>>>>>>> the
> > >> >>>>>>>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib
> or
> > >> >> not
> > >> >>>> is a
> > >> >>>>>>>>>>>>>>>>> tricky
> > >> >>>>>>>>>>>>>>>>>>>> question,
> > >> >>>>>>>>>>>>>>>>>>>>>>> I think this is already reflected in the
> > document
> > >> >>>> since
> > >> >>>>>>>>>>>>>>>>> "maturity"
> > >> >>>>>>>>>>>>>>>>>>> is
> > >> >>>>>>>>>>>>>>>>>>>> in
> > >> >>>>>>>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer
> the
> > >> >>>> operator
> > >> >>>>>>>>>>>>> to
> > >> >>>>>>>>>>>>>>>>> the
> > >> >>>>>>>>>>>>>>>>>>> core
> > >> >>>>>>>>>>>>>>>>>>>> or
> > >> >>>>>>>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib
> to
> > >> >>>>>>>>>>> incubating
> > >> >>>>>>>>>>>>>>>>> will
> > >> >>>>>>>>>>>>>>>>>>> confuse
> > >> >>>>>>>>>>>>>>>>>>>>>>> the
> > >> >>>>>>>>>>>>>>>>>>>>>>> user even more. For people who aren't as known
> > >> >> with
> > >> >>>>>>>>>>>>> Airflow it
> > >> >>>>>>>>>>>>>>>>> isn't
> > >> >>>>>>>>>>>>>>>>>>>> that
> > >> >>>>>>>>>>>>>>>>>>>>>>> transparent which operator lives where,
> > especially
> > >> >>> if
> > >> >>>> you
> > >> >>>>>>>>>>>>>>>>> don't use
> > >> >>>>>>>>>>>>>>>>>>> an
> > >> >>>>>>>>>>>>>>>>>>>> IDE
> > >> >>>>>>>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it
> is
> > >> >>> worth
> > >> >>>>>>>>>>>>> changing
> > >> >>>>>>>>>>>>>>>>> the
> > >> >>>>>>>>>>>>>>>>>>>> public
> > >> >>>>>>>>>>>>>>>>>>>>>>> API just for a different name.
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote
> with
> > >> >>> this.
> > >> >>>> It
> > >> >>>>>>>>>>>>> will
> > >> >>>>>>>>>>>>>>>>> make
> > >> >>>>>>>>>>>>>>>>>>>> easier
> > >> >>>>>>>>>>>>>>>>>>>>>> as we will not have to define other rules as
> > those
> > >> >>>> that we
> > >> >>>>>>>>>>>>>>>>> already
> > >> >>>>>>>>>>>>>>>>>>> have.
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B
> (let's
> > >> >>> say a
> > >> >>>>>>>>>>> +0)
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> I like to search on Github with t, and then
> > search
> > >> >>> for
> > >> >>>>>>>>>>>>>>>>> BashOperator.
> > >> >>>>>>>>>>>>>>>>>>>> After
> > >> >>>>>>>>>>>>>>>>>>>>>>> the change, you should search for
> Operator/Bash.
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this,
> from
> > >> >>> your
> > >> >>>>>>>>>>> mail
> > >> >>>>>>>>>>>>> you
> > >> >>>>>>>>>>>>>>>>>>> state: -
> > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 2: B:
> > >> >>> Airflow.operators.foo_operator.FooOperator
> > >> >>>>>>>>>>>>> could
> > >> >>>>>>>>>>>>>>>>> become
> > >> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B
> > >> >>>> solution is
> > >> >>>>>>>>>>>>>>>>> keeping it
> > >> >>>>>>>>>>>>>>>>>>>> the
> > >> >>>>>>>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
> > >> >>>>>>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator
> > *would
> > >> >>> stay
> > >> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there
> > is a
> > >> >>>> little
> > >> >>>>>>>>>>>>>>>>> confusion
> > >> >>>>>>>>>>>>>>>>>>>> here.
> > >> >>>>>>>>>>>>>>>>>>>>>> This case is not for changing BashOperator into
> > >> >> Bash
> > >> >>>> but
> > >> >>>>>>>>>>> for
> > >> >>>>>>>>>>>>>>>>> changing
> > >> >>>>>>>>>>>>>>>>>>>> the
> > >> >>>>>>>>>>>>>>>>>>>>>> *module* name not the *class* name. So
> > >> >>>>>>>>>>>>>>>>> bash_operator.BashOperator ->
> > >> >>>>>>>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search
> for
> > >> >>>>>>>>>>>>> BashOperator
> > >> >>>>>>>>>>>>>>>>> in this
> > >> >>>>>>>>>>>>>>>>>>>> case.
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide
> > when
> > >> >> it
> > >> >>>> gets
> > >> >>>>>>>>>>>>> its
> > >> >>>>>>>>>>>>>>>>> own
> > >> >>>>>>>>>>>>>>>>>>>> package
> > >> >>>>>>>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are
> > >> >>>> companies
> > >> >>>>>>>>>>>>> like
> > >> >>>>>>>>>>>>>>>>>>>> Databricks
> > >> >>>>>>>>>>>>>>>>>>>>>>> and Qubole which might also end up with their
> > own
> > >> >>>> package
> > >> >>>>>>>>>>>>>>>>> because
> > >> >>>>>>>>>>>>>>>>>>> their
> > >> >>>>>>>>>>>>>>>>>>>>>>> integration is getting richer over time.
> Moving
> > >> >> this
> > >> >>>> is a
> > >> >>>>>>>>>>>>> bit
> > >> >>>>>>>>>>>>>>>>>>> painful
> > >> >>>>>>>>>>>>>>>>>>>>>>> because we need to deprecate the old path and
> > move
> > >> >>>>>>>>>>>>> everything
> > >> >>>>>>>>>>>>>>>>> which
> > >> >>>>>>>>>>>>>>>>>>>>>>> introduces noise into version control etc.
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed.
> It's
> > >> >> not
> > >> >>> as
> > >> >>>>>>>>>>>>>>>>> difficult as
> > >> >>>>>>>>>>>>>>>>>>> it
> > >> >>>>>>>>>>>>>>>>>>>>>> seems to group operators in packages and it has
> > >> >>>> numerous
> > >> >>>>>>>>>>>>>>>>> advantages.
> > >> >>>>>>>>>>>>>>>>>>> The
> > >> >>>>>>>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them
> in
> > >> >> the
> > >> >>>>>>>>>>>>>>>>> __init__.py of
> > >> >>>>>>>>>>>>>>>>>>> the
> > >> >>>>>>>>>>>>>>>>>>>>>> packages which causes the noise fairly small
> (Git
> > >> >>> will
> > >> >>>>>>>>>>>>> actually
> > >> >>>>>>>>>>>>>>>>>>> realise
> > >> >>>>>>>>>>>>>>>>>>>>>> that the file has been moved and you can follow
> > >> >> that.
> > >> >>>>>>>>>>> Maybe
> > >> >>>>>>>>>>>>> not
> > >> >>>>>>>>>>>>>>>>> as
> > >> >>>>>>>>>>>>>>>>>>> super
> > >> >>>>>>>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's
> > >> >> not a
> > >> >>>>>>>>>>> rocket
> > >> >>>>>>>>>>>>>>>>> science
> > >> >>>>>>>>>>>>>>>>>>>> either.
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>> Yes.
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> ---
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> Apart from all, great work!
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix
> > >> >> Uellendall
> > >> >>>>>>>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
> > >> >>>>>>>>>>>>>>>>>>>>>>>> :
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed
> > change
> > >> >>> of
> > >> >>>>>>>>>>> these
> > >> >>>>>>>>>>>>>>>>> cases.
> > >> >>>>>>>>>>>>>>>>>>> So I
> > >> >>>>>>>>>>>>>>>>>>>>>>> go
> > >> >>>>>>>>>>>>>>>>>>>>>>>> with +1 (non-binding).
> > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again
> > and
> > >> >>> your
> > >> >>>>>>>>>>>>> intense
> > >> >>>>>>>>>>>>>>>>> work
> > >> >>>>>>>>>>>>>>>>>>>>>>> towards
> > >> >>>>>>>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for
> > even
> > >> >>>>>>>>>>> creating
> > >> >>>>>>>>>>>>>>>>> this
> > >> >>>>>>>>>>>>>>>>>>>>>>> document. I
> > >> >>>>>>>>>>>>>>>>>>>>>>>> like how the code is getting more and more
> > >> >>> consistent
> > >> >>>>>>>>>>> and
> > >> >>>>>>>>>>>>>>>>> clean :)
> > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> > >> >>>>>>>>>>>>>>>>>>>>>>>> Felix
> > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
> > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>> -------- Original Message --------
> > >> >>>>>>>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes
> in
> > >> >>>> import
> > >> >>>>>>>>>>>>> paths.
> > >> >>>>>>>>>>>>>>>>> It's
> > >> >>>>>>>>>>>>>>>>>>>> been
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> discussed in
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July
> > >> >> 30th
> > >> >>>> 6pm
> > >> >>>>>>>>>>>>> CEST),
> > >> >>>>>>>>>>>>>>>>> and
> > >> >>>>>>>>>>>>>>>>>>> at
> > >> >>>>>>>>>>>>>>>>>>>>>>> least
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the
> document
> > >> >> from
> > >> >>>>>>>>>>> Kamil
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> <
> > >> >>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> The proposed solution is:
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all
> that
> > >> >> are
> > >> >>>>>>>>>>> "well"
> > >> >>>>>>>>>>>>>>>>> tested
> > >> >>>>>>>>>>>>>>>>>>> and
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 2: B:
> > >> >>>>>>>>>>> Airflow.operators.foo_operator.FooOperator
> > >> >>>>>>>>>>>>>>>>> could
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>
> > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > >> >>>>>>>>>>>>>>>>>>>> could
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> become
> > >> >>>> airflow.gcp.operators.bigtable.BigTableOperator*
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
> > >> >>>> suffixes
> > >> >>>>>>>>>>> on
> > >> >>>>>>>>>>>>>>>>> class
> > >> >>>>>>>>>>>>>>>>>>> names*
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> > >> >>>> case-by-case
> > >> >>>>>>>>>>>>> (and
> > >> >>>>>>>>>>>>>>>>> my
> > >> >>>>>>>>>>>>>>>>>>> team
> > >> >>>>>>>>>>>>>>>>>>>>>>> can
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> J.
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> --
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> |
> > Principal
> > >> >>>>>>>>>>> Software
> > >> >>>>>>>>>>>>>>>>> Engineer
> > >> >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
> > >> >>>>>>>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
> > >> >>>>>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >> >>>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>> --
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> > >> >>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> > >> >>>> Software
> > >> >>>>>>>>>>>>>>>>> Engineer
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > >> >>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>> --
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> > >> >>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> > >> >>> Software
> > >> >>>>>>>>>>>>>> Engineer
> > >> >>>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > >> >>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >> >>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>> --
> > >> >>>>>>>>>>>>>>>>>> *Kaxil Naik*
> > >> >>>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > >> >>>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer |
> *Certified*
> > >> >>> Apache
> > >> >>>>>>>>>>> Spark
> > >> >>>>>>>>>>>>> &
> > >> >>>>>>>>>>>>>>>>> Neo4j
> > >> >>>>>>>>>>>>>>>>>> Developer
> > >> >>>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >> >>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>> --
> > >> >>>>>>>>>>>>>>>>> *Kaxil Naik*
> > >> >>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > >> >>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
> > >> >>> Apache
> > >> >>>>>>>> Spark
> > >> >>>>>>>>>>> &
> > >> >>>>>>>>>>>>>>>>> Neo4j
> > >> >>>>>>>>>>>>>>>>> Developer
> > >> >>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> --
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Jarek Potiuk
> > >> >>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> > >> >>> Engineer
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > >> >>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> --
> > >> >>>>>>>> *Kaxil Naik*
> > >> >>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > >> >>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> > >> >> Spark &
> > >> >>>> Neo4j
> > >> >>>>>>>> Developer
> > >> >>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >> >>>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> --
> > >> >>>>>>>
> > >> >>>>>>> Jarek Potiuk
> > >> >>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> > Engineer
> > >> >>>>>>>
> > >> >>>>>>> M: +48 660 796 129 <+48660796129>
> > >> >>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>
> > >> >>>>>> --
> > >> >>>>>>
> > >> >>>>>> Jarek Potiuk
> > >> >>>>>> Polidea <https://www.polidea.com/> | Principal Software
> Engineer
> > >> >>>>>>
> > >> >>>>>> M: +48 660 796 129 <+48660796129>
> > >> >>>>>> [image: Polidea] <https://www.polidea.com/>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> > >>
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Changes in import paths
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi Jarek,
Just wondering how the vote is unanimous. On case 3 I've voted for B. I
still believe introducing namespaces is hard to maintain, as I explained
earlier: Mostly because it isn't as trivial to decide when it gets its own
package or not. Besides the cloud providers, there are companies like
Databricks and Qubole which might also end up with their own package
because their integration is getting richer over time. Moving this is a bit
painful because we need to deprecate the old path and move everything which
introduces noise into version control etc.
Adding an additional option to the list, does not invalidate the older
options, right?
Cheers, Fokko
Op ma 5 aug. 2019 om 15:43 schreef Jarek Potiuk <Ja...@polidea.com>:
> Seems that after this months of discussion we got an unanimous voting
> result. I summarised it in the AIP-21
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >
> and
> changed its status to "Accepted". Thanks for all your feedback! We will
> start soon moving GCP operators following those rules and we will resolve
> any initial problems with those - then we might provide some additional
> learnings and see if we can easily (probably semi-automatically) move all
> other operators.
>
> Case 1
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#1airflow.contrib.{resources}
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%231airflow.contrib.%7Bresources%7D>
> >
>
> What to do with the "contrib" folder
>
> Case 2
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#2git*_{operator/sensor}{/s}.py
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%232git*_%7Boperator/sensor%7D%7B/s%7D.py>
> >
>
> Drop modules *_operator suffix
>
> Case 3
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#3{aws/azure/gcp}_*
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%233%7Baws/azure/gcp%7D_*>
> >
> + Case 4
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#4Separatenamespaceforresources
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%234Separatenamespaceforresources>
> >
>
> Grouping Cloud Providers operators/sensors/hooks
>
> Case 5
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#5*Operator
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%235*Operator>
> >
>
> *Operator *Sensor *Hook in class name
>
> Case 6
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#6Otherisolatedcases
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%236Otherisolatedcases>
> >
>
> Isolated cases
>
> Case 7
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#7
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%237>
> >
>
> Deprecation method
>
> *A.* Move everything "contrib" → "airflow"
>
> *A.* Drop the suffix.
>
> Example:
>
> - *airflow.operator.gcp_bigtable_operator.py*
> becomes *airflow.operator.gcp_bigtable.py
> <http://airflow.operator.gcp_bigtable.py>*.
>
> *D. * Group operators/sensors/hooks in
> *airflow/<PROVIDER>*/operators(sensors,
> hooks). Non-cloud provider ones are moved to
> airflow/operators(sensors/hooks).
> *Drop the prefix.*
>
> -
> *airflow/contrib/operators/sns_publish_operator.py
> becomes airflow/aws/operators/**sns_publish_operator.py*
>
>
> - *airflow/contrib/operators/dataproc_operator.py*
> becomes *airflow/gcp/operators/dataproc_operator.py*
>
>
>
> -
> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
> *gcp/operators/bigtable_operator.py*
>
>
>
> -
> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> *operators/ssh_operator.py*
>
> *B.* No change - keep Operator/Sensor suffix in class name.
>
> *A.* Make individual decisions of renames for operators that do not follow
> common conventions used for other operators.
>
> Consistency trumps compatibility.
>
> Examples:
>
> *DataProcHadoopOperator*
>
> renamed to:
>
> *DataprocHadoopOperator*
> *A.* Native python method (with better IDE support and more flexible but a
> bit more verbose)
>
> J.
>
> On Wed, Jul 31, 2019 at 10:30 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > Yep. Agree with Ash on it. There are a number of 'action' operators
> > specific for cloud providers and these should be our target. The transfer
> > ones require another AIP (A lot of that already discussed in AIP-8
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
> > ).
> >
> > J.
> >
> > Principal Software Engineer
> > Phone: +48660796129
> >
> > śr., 31 lip 2019, 10:01 użytkownik Ash Berlin-Taylor <as...@apache.org>
> > napisał:
> >
> >> This is a good idea for now. I'm also not overly concerned about these
> >> few non-cloud examples - FTPtoS3Operator can stay where it is and
> doesn't
> >> have to move under 'aws.' to my mind.
> >>
> >> Longer term I'd like to go back to making the "transfer/copy/transform"
> >> operators "composable" so that we can have a single DataCopyOperator,
> and
> >> give it a source and destination, and it uses hooks to do the work.
> This is
> >> not a small undertaking though to do well and efficiently though.
> >>
> >> -ash
> >>
> >> > On 30 Jul 2019, at 20:54, Tomasz Urbaszek <
> tomasz.urbaszek@polidea.com>
> >> wrote:
> >> >
> >> > Maybe we can put all those AtoB operators under one name like
> >> “transfer”,
> >> > then it would be easier to look for such operator?
> >> >
> >> > Best,
> >> > Tomek
> >> >
> >> > On Tue, 30 Jul 2019 at 21:39, Chen Tong <ci...@gmail.com> wrote:
> >> >
> >> >> Daniel mentioned a good point. Such composed operator may also
> involves
> >> >> both cloud and non-cloud provider saying FTPtoS3Operator. Should it
> in
> >> AWS
> >> >> folder?
> >> >>
> >> >> Also, saying in the future, another cloud provider is growing large
> >> enough.
> >> >> Will we rename all plugins related to this provider? What's the
> >> criteria we
> >> >> say it's big enough? And option D gives an impression like these
> tools
> >> may
> >> >> be maintained and supported by the Cloud provider. I am not sure if
> it
> >> will
> >> >> be a truth or not.
> >> >>
> >> >> Best,
> >> >>
> >> >>
> >> >> On Tue, Jul 30, 2019 at 11:18 AM Daniel Standish <
> dpstandish@gmail.com
> >> >
> >> >> wrote:
> >> >>
> >> >>> One wrinkle with case 3+4 option D is inter-provider operators.
> >> Mainly
> >> >>> it's storage I think e.g. XToS3Operator or XToGCSOperator where the
> X
> >> is
> >> >> a
> >> >>> service some different provider.
> >> >>>
> >> >>> Maybe the rule should be to locate the operator according to the
> first
> >> >>> provider referenced. So e.g. s3_to_gcs_transfer_operator.py would
> go
> >> to
> >> >>> aws.
> >> >>>
> >> >>>
> >> >>> On Tue, Jul 30, 2019 at 6:21 AM Kamil Breguła <
> >> kamil.bregula@polidea.com
> >> >>>
> >> >>> wrote:
> >> >>>
> >> >>>> Yes. All changes will be backwards compatible. In the case of using
> >> >>>> the old path, a message containing a proposal for change will be
> >> >>>> reported to the user.
> >> >>>>
> >> >>>> I prepared an example of how to change the name of a class in a
> case
> >> >>>> with the use of a native solution.
> >> >>>> Source code:
> >> >>>>
> >> >>
> >>
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename
> >> >>>> Preview from IDE: https://imgur.com/a/45NxS5W
> >> >>>>
> >> >>>> On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <as...@apache.org>
> >> >>> wrote:
> >> >>>>>
> >> >>>>> Just cos I'm not sure it's _explicitly_ stated, but all of the
> moves
> >> >>>> will have a deprecation of the old name right?
> >> >>>>>
> >> >>>>> 3+4 case D gets my vote too.
> >> >>>>>
> >> >>>>> -a
> >> >>>>>
> >> >>>>>> On 30 Jul 2019, at 09:58, Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> >> >>>> wrote:
> >> >>>>>>
> >> >>>>>> I went ahead and updated the page (just to speed it up) as I
> think
> >> >> it
> >> >>>>>> really makes sense to join those two cases (and I do not see any
> >> >>>> drawbacks
> >> >>>>>> - I think the options we have cover all possible approaches) and
> we
> >> >>> can
> >> >>>>>> always go back if we need to.
> >> >>>>>>
> >> >>>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> >> >>>>>>
> >> >>>>>> My vote is *D*.
> >> >>>>>>
> >> >>>>>> - airflow/contrib/operators/gcp_bigtable_operator.py →
> >> >>> airflow/gcp
> >> >>>>>> /operators/bigtable_operator.py
> >> >>>>>> - airflow/contrib/operators/ssh_operator.py ->
> >> >>>>>> airflow/operators/ssh_operator.py
> >> >>>>>>
> >> >>>>>> I updated the page with my vote.
> >> >>>>>> I propose that we vote till Friday on that one (all the rest is
> >> >>> already
> >> >>>>>> agreed).
> >> >>>>>>
> >> >>>>>> J.
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <
> >> >>> Jarek.Potiuk@polidea.com
> >> >>>>>
> >> >>>>>> wrote:
> >> >>>>>>
> >> >>>>>>> I think almost everyone voted and we have almost perfect
> >> >> consensus.
> >> >>>> We all
> >> >>>>>>> agree amongst other on moving all operators out of contrib
> >> >> (Great!).
> >> >>>>>>>
> >> >>>>>>> The only doubts are for *Case 3* (Cloud provider prefix) and
> *Case
> >> >>> 4*
> >> >>>>>>> (Using Namespaces).
> >> >>>>>>> I think there was actually an overlap between those two. Also
> >> >> Ash's
> >> >>>>>>> comment/veto on that is quite clear.
> >> >>>>>>> So I have final (I hope!) proposal to join those two cases and
> >> >> have
> >> >>>> one
> >> >>>>>>> voting (till Friday) only on that.
> >> >>>>>>>
> >> >>>>>>> I will update the doc if you all agree with me that makes more
> >> >> sense
> >> >>>> as
> >> >>>>>>> single case to vote on:
> >> >>>>>>>
> >> >>>>>>> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
> >> >>>>>>>
> >> >>>>>>> *Option A*. Keep operators/sensors/hooks in
> >> >>> airflow/operators(sensors,
> >> >>>>>>> hooks) and keep/add prefixes in file names.
> >> >>>>>>>
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py becomes
> >> >>>>>>> airflow/operators/**aws_sns_publish_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> >> >>>>>>> becomes *airflow/operators/gcp_dataproc_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/*
> >> >>>>>>> *hooks/gcp_bigtable_hook.py*
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >> >>>>>>> *operators/ssh_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> *Option B*. Group operators/sensors/hooks in
> >> >>>> airflow/operators(sensors,
> >> >>>>>>> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
> >> >>>>>>> airflow/operators(sensors/hooks), Drop the prefix.
> >> >>>>>>>
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> >> >>>>>>> becomes airflow/operators/**aws/sns_publish_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> >> >>>>>>> becomes *airflow/operators/gcp/dataproc_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> >> >>>> *airflow/*
> >> >>>>>>> *operators/gcp/bigtable_operator.py*
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >> >>>>>>> *operators/ssh_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> *Option C*. Group operators/sensors/hooks in
> >> >>>> airflow/operators(sensors,
> >> >>>>>>> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
> >> >>>>>>> airflow/operators(sensors/hooks), Keep the prefix.
> >> >>>>>>>
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> >> >>>>>>> becomes airflow/operators/**aws/aws_sns_publish_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> >> >>>>>>> becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> >> >>>> *airflow/*
> >> >>>>>>> *operators/gcp/gcp_bigtable_operator.py*
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >> >>>>>>> *operators/ssh_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> *Option D.* Group operators/sensors/hooks in
> >> >>>> *airflow/<PROVIDER>*/operators(sensors,
> >> >>>>>>> hooks). Non-cloud provider ones are moved to
> >> >>>>>>> airflow/operators(sensors/hooks). Drop the prefix.
> >> >>>>>>>
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> >> >>>>>>> becomes airflow/aws/operators/**sns_publish_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> >> >>>>>>> becomes *airflow/gcp/operators/dataproc_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> >> >>>> *airflow/*
> >> >>>>>>> *gcp/operators/bigtable_operator.py*
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >> >>>>>>> *operators/ssh_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> *Option E.* Group operators/sensors/hooks in
> >> >>>> *airflow/<PROVIDER>*/operators(sensors,
> >> >>>>>>> hooks). Non-cloud provider ones are moved to
> >> >>>>>>> airflow/operators(sensors/hooks).* Keep the prefix*.
> >> >>>>>>>
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> >> >>>>>>> becomes airflow/aws/operators/aws_**sns_publish_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> >> >>>>>>> becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> >> >>>> *airflow/*
> >> >>>>>>> *gcp/operators/gcp_bigtable_operator.py*
> >> >>>>>>> -
> >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >> >>>>>>> *operators/ssh_operator.py*
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> Shall I proceed with updating the page ?
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> J.
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <
> kaxilnaik@gmail.com>
> >> >>>> wrote:
> >> >>>>>>>
> >> >>>>>>>> Thanks Jarek, adding my vote.
> >> >>>>>>>>
> >> >>>>>>>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <
> >> >> ash@apache.org>
> >> >>>> wrote:
> >> >>>>>>>>
> >> >>>>>>>>> Thanks Jarek, much clearer. I have voted there too.
> >> >>>>>>>>>
> >> >>>>>>>>> -ash
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko
> >> >> <fokko@driesprong.frl
> >> >>>>
> >> >>>>>>>>> wrote:
> >> >>>>>>>>>>
> >> >>>>>>>>>> Thanks Jarek, the Wiki works much better for me. I've added
> my
> >> >>> vote
> >> >>>>>>>> there
> >> >>>>>>>>>> as well.
> >> >>>>>>>>>>
> >> >>>>>>>>>> You've convinced me on the second case. I've changed my vote
> on
> >> >>>> Case 2
> >> >>>>>>>>> from
> >> >>>>>>>>>> A to B :-)
> >> >>>>>>>>>>
> >> >>>>>>>>>> Maybe we should leave the vote open a bit longer since it
> >> >>> involves
> >> >>>>>>>> quite
> >> >>>>>>>>>> some reading, and I would like to get some proper feedback
> and
> >> >>>>>>>> opinions
> >> >>>>>>>>> on
> >> >>>>>>>>>> this. Plus I only see two votes right now. If we don't think
> >> >> this
> >> >>>>>>>>> through,
> >> >>>>>>>>>> it might be that we're having this vote again in the near
> >> >> future
> >> >>>> :-)
> >> >>>>>>>>>>
> >> >>>>>>>>>> Cheers, Fokko
> >> >>>>>>>>>>
> >> >>>>>>>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
> >> >>>>>>>>> Jarek.Potiuk@polidea.com>:
> >> >>>>>>>>>>
> >> >>>>>>>>>>> I updated the AIP-21 proposal in the way that should answer
> >> >> all
> >> >>>> the
> >> >>>>>>>>>>> concerns in this thread.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> NOTE! There is an action for all of those who are
> interested:
> >> >>>> Please
> >> >>>>>>>>>>> fill-in your voting in the voting table till Tuesday 30th of
> >> >>> July
> >> >>>> 6pm
> >> >>>>>>>>> CEST
> >> >>>>>>>>>>> (5pm BST, 9 am PST).
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I created a summary of the proposal
> >> >>>>>>>>>>> <
> >> >>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>> in table form - which contains also examples for each case.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I added a voting table
> >> >>>>>>>>>>> <
> >> >>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>> where you should add your votes:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I propose this voting mechanism:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST
> (5
> >> >>> pm
> >> >>>>>>>> BST,
> >> >>>>>>>>> 9am
> >> >>>>>>>>>>> PST) .
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> After the choice, the final consistent set of choices will
> be
> >> >>>>>>>> announced
> >> >>>>>>>>>>> (taking into account majority of binding vote, also
> including
> >> >>>>>>>> potential
> >> >>>>>>>>>>> vetos and consistency between the choices. Non-binding votes
> >> >>> will
> >> >>>> be
> >> >>>>>>>>> taken
> >> >>>>>>>>>>> into account in case there is a draw. The final set of
> choices
> >> >>>> will
> >> >>>>>>>> be
> >> >>>>>>>>>>> announced at devlist thread
> >> >>>>>>>>>>> <
> >> >>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>> after
> >> >>>>>>>>>>> the voting completes.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Unless there is a veto raised to the final proposal, the
> final
> >> >>>>>>>> proposal
> >> >>>>>>>>> is
> >> >>>>>>>>>>> accepted by Lazy Consensus
> >> >>>>>>>>>>> <https://community.apache.org/committers/lazyConsensus.html
> >
> >> >>> on
> >> >>>>>>>>> *Friday*
> >> >>>>>>>>>>> 02
> >> >>>>>>>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Please let me know if what I proposed can be further
> improved
> >> >> to
> >> >>>>>>>> avoid
> >> >>>>>>>>>>> chaos.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> J.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
> >> >>>>>>>> Jarek.Potiuk@polidea.com
> >> >>>>>>>>>>
> >> >>>>>>>>>>> wrote:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>> Ok. I see that indeed voting could have been organised a
> bit
> >> >>>> better
> >> >>>>>>>> -
> >> >>>>>>>>> dev
> >> >>>>>>>>>>>> list does not seem to cope well with such a complex case
> :).
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> As Kamil noticed - we already have a formal AIP-21
> >> >>>>>>>>>>>> <
> >> >>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> in the Wiki - which I should have mentioned before. I have
> >> >> not
> >> >>>> much
> >> >>>>>>>>> time
> >> >>>>>>>>>>>> today - but tomorrow I will try to summarise the options -
> >> >>> based
> >> >>>> on
> >> >>>>>>>>> some
> >> >>>>>>>>>>>> real code examples (to make it easier to digest). I think
> the
> >> >>>> best
> >> >>>>>>>>>>> approach
> >> >>>>>>>>>>>> will be to make a voting matrix in the AIP-21 Wiki page
> where
> >> >>>> people
> >> >>>>>>>>> will
> >> >>>>>>>>>>>> be able to state their preferences for each case (by
> adding a
> >> >>>> row to
> >> >>>>>>>>> the
> >> >>>>>>>>>>>> table). I think that might provide much better clarity and
> a
> >> >>>> single
> >> >>>>>>>>> place
> >> >>>>>>>>>>>> which will show the current aggregated state of voting.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> However - as Ash also stated - some of the options are
> >> >>>>>>>> inter-dependent
> >> >>>>>>>>> so
> >> >>>>>>>>>>>> at the end we will have to adjust the results in case they
> >> >> are
> >> >>>> not
> >> >>>>>>>>>>>> consistent - and make final voting on aggregated set of
> >> >> cases
> >> >>> -
> >> >>>>>>>> but I
> >> >>>>>>>>>>>> think in this case following "lasy consensus" - i,e. if
> noone
> >> >>>>>>>> objects
> >> >>>>>>>>> to
> >> >>>>>>>>>>>> final set of cases, we will proceed with it..
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Do you think that will work better ?
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> J.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Principal Software Engineer
> >> >>>>>>>>>>>> Phone: +48660796129
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> >> >>>>>>>>>>>> kamil.bregula@polidea.com> napisał:
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>> Document is also available as AIP on wiki:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
> >> >>>>>>>> <fokko@driesprong.frl
> >> >>>>>>>>>>
> >> >>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> I agree with all, this is a bit chaotic. For the sake of
> >> >>>> clarity,
> >> >>>>>>>> I
> >> >>>>>>>>>>>>> would
> >> >>>>>>>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP.
> >> >> Create a
> >> >>>>>>>>>>> structural
> >> >>>>>>>>>>>>>> code improvement AIP and then and add a matrix of
> >> >> users/cases
> >> >>>>>>>> where
> >> >>>>>>>>>>>>>> everybody can add his vote per case. This gives also the
> >> >>>>>>>> opportunity
> >> >>>>>>>>>>> for
> >> >>>>>>>>>>>>>> changing your vote on a specific case. WDYT?
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Cheers, Fokko
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <
> >> >>>> cixuuz@gmail.com>:
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> I agree with Ash. It is clear to vote on each case
> rather
> >> >>> than
> >> >>>>>>>> all
> >> >>>>>>>>>>>>>>> together.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
> >> >>>>>>>> ash@apache.org>
> >> >>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> A number of proposals overlap or add on to each other,
> >> >> so I
> >> >>>>>>>> think
> >> >>>>>>>>>>>>> (?) a
> >> >>>>>>>>>>>>>>>> single vote makes sense.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly
> what
> >> >> we
> >> >>>> are
> >> >>>>>>>>>>>>> voting
> >> >>>>>>>>>>>>>> on.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <
> >> >>>> kaxilnaik@gmail.com>
> >> >>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases
> >> >>>> together,
> >> >>>>>>>>>>> how
> >> >>>>>>>>>>>>>>>>> about
> >> >>>>>>>>>>>>>>>>> separate vote emails for all the cases? Each email can
> >> >>> have
> >> >>>> the
> >> >>>>>>>>>>>>>>>>> description
> >> >>>>>>>>>>>>>>>>> copied over from the doc.
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> It would probably be a bit easier to track a decision
> in
> >> >>> the
> >> >>>>>>>>>>> future
> >> >>>>>>>>>>>>> as
> >> >>>>>>>>>>>>>>>>> well.
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
> >> >>>>>>>> Jarek.Potiuk@polidea.com>
> >> >>>>>>>>>>>>>>>>> @Driesprong,
> >> >>>>>>>>>>>>>>>>> Fokko <fo...@driesprong.frl>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
> >> >>>>>>>> kaxilnaik@gmail.com>
> >> >>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution
> A
> >> >> *
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> *Case #4* Separate namespace for resources :
> >> >>>>>>>>>>>>>>>>>> *Solution A*
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> *Case #7* : *Solution A*
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> >> >>>>>>>>>>>>>>>>> <dp...@gmail.com>
> >> >>>>>>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> I have tried to add some clarification to Jarek's
> >> >>> summary,
> >> >>>>>>>> but
> >> >>>>>>>>>>> I
> >> >>>>>>>>>>>>> am
> >> >>>>>>>>>>>>>>>>> a
> >> >>>>>>>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so
> hopefully
> >> >>>> Jarek
> >> >>>>>>>>>>> can
> >> >>>>>>>>>>>>>>>>> further
> >> >>>>>>>>>>>>>>>>>>> clarify.
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6
> >> >> are
> >> >>>> all
> >> >>>>>>>>>>>>> either
> >> >>>>>>>>>>>>>>>>>>> proposal
> >> >>>>>>>>>>>>>>>>>>> *rejections* or otherwise have no effect, so
> attention
> >> >>>> can be
> >> >>>>>>>>>>>>>>>>> focused on
> >> >>>>>>>>>>>>>>>>>>> cases 1,2,3 and 7.
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> *Motion is to adopt the following:*
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
> >> >>>>>>>>>>>>>>>>>>> All objects will be moved out of contrib folder and
> >> >> into
> >> >>>>>>>>>>>>> respective
> >> >>>>>>>>>>>>>>>>>>> non-contrib folder.
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from
> >> >>>> modules
> >> >>>>>>>>>>>>>>>>>>> Example:
> >> >>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
> >> >>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator
> >> >>>>>>>>>>>>>>>>>>> Example:
> >> >>>>>>>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
> >> >>>>>>>>>>>>> airflow.hooks.foo.FooHook
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
> >> >>>>>>>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
> >> >>>>>>>>>>>>>>>>>>> Example case is
> >> >>>>>>>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
> >> >>>>>>>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume
> >> >> it
> >> >>> is
> >> >>>>>>>> now
> >> >>>>>>>>>>>>> named
> >> >>>>>>>>>>>>>>>>> like
> >> >>>>>>>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
> >> >>>>>>>>>>>>>>>>>>> So what is proposed for this case?
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Case 4 - Solution B - *no change*
> >> >>>>>>>>>>>>>>>>>>> This proposal was about namespaces but the motion is
> >> >> to
> >> >>>>>>>> reject
> >> >>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>> proposal.
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Case 5 - Solution B - *no change*
> >> >>>>>>>>>>>>>>>>>>> The proposal was to remove class type suffix e.g.
> >> >> remove
> >> >>>> the
> >> >>>>>>>>>>>>>>>>> Operator in
> >> >>>>>>>>>>>>>>>>>>> FooOperator, but the motion is to reject this
> proposal
> >> >>> --
> >> >>>>>>>> i.e.
> >> >>>>>>>>>>>>>>>>> motion is
> >> >>>>>>>>>>>>>>>>>>> no
> >> >>>>>>>>>>>>>>>>>>> change on this case, i.e. we resolve to keep
> >> >> "Operator"
> >> >>>> (and
> >> >>>>>>>>>>>>>>>>> "Sensor")
> >> >>>>>>>>>>>>>>>>>>> suffixes on class names
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Case 6 - *No change*
> >> >>>>>>>>>>>>>>>>>>> This case concerns object naming inconsistencies,
> but
> >> >>>> motion
> >> >>>>>>>>>>> does
> >> >>>>>>>>>>>>>>>>> not
> >> >>>>>>>>>>>>>>>>>>> enact
> >> >>>>>>>>>>>>>>>>>>> any specific proposal.
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
> >> >>>>>>>>>>> deprecation
> >> >>>>>>>>>>>>>>>>> warnings
> >> >>>>>>>>>>>>>>>>>>> (instead of zope)
> >> >>>>>>>>>>>>>>>>>>> Example:
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> # in old_package/old_mod.py
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> import warnings
> >> >>>>>>>>>>>>>>>>>>> from new_package.new_mod import *
> >> >>>>>>>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
> >> >>>>>>>>>>>>> new_package.new_mod.
> >> >>>>>>>>>>>>>>>>>>> Import
> >> >>>>>>>>>>>>>>>>>>> of "
> >> >>>>>>>>>>>>>>>>>>> "old_package.old_mod will become unsupported in
> >> >> version
> >> >>>> 2",
> >> >>>>>>>>>>>>>>>>>>> DeprecationWarning, 2)
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Reference implementation here:
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> I am very happy that this motion now gets rid of
> >> >>> contrib.
> >> >>>>>>>>>>> Thank
> >> >>>>>>>>>>>>> you
> >> >>>>>>>>>>>>>>>>>>> Sergio
> >> >>>>>>>>>>>>>>>>>>> for turning the tide with your effective
> argumentation
> >> >>> ;)
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
> >> >>>>>>>>>>>>> ash@apache.org>
> >> >>>>>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Could someone summarise what the proposed name
> >> >> changes
> >> >>>> are,
> >> >>>>>>>>>>>>> here,
> >> >>>>>>>>>>>>>>>>> in
> >> >>>>>>>>>>>>>>>>>>> this
> >> >>>>>>>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving
> >> >>> partial
> >> >>>>>>>>>>>>> examples
> >> >>>>>>>>>>>>>>>>> is 1)
> >> >>>>>>>>>>>>>>>>>>> a
> >> >>>>>>>>>>>>>>>>>>>> bit difficult to quickly work out what we are
> voting
> >> >>> on,
> >> >>>> and
> >> >>>>>>>>>>> 2)
> >> >>>>>>>>>>>>>>>>> using a
> >> >>>>>>>>>>>>>>>>>>>> google doc isn't great for a longer term record.
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Thanks,
> >> >>>>>>>>>>>>>>>>>>>> Ash
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
> >> >>>>>>>>>>>>>>>>> <Ja...@polidea.com>
> >> >>>>>>>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from
> 0
> >> >> :)
> >> >>>> . I
> >> >>>>>>>>>>>>> would
> >> >>>>>>>>>>>>>>>>> like
> >> >>>>>>>>>>>>>>>>>>> to
> >> >>>>>>>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least
> three
> >> >>> +1
> >> >>>>>>>>>>>>>>>>> (binding)
> >> >>>>>>>>>>>>>>>>>>> votes
> >> >>>>>>>>>>>>>>>>>>>> are
> >> >>>>>>>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications
> to
> >> >>> the
> >> >>>>>>>>>>>>>>>>> previous
> >> >>>>>>>>>>>>>>>>>>>> proposal.
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> Here is the proposal (details here
> >> >>>>>>>>>>>>>>>>>>>>> <
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> )
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
> >> >>>>>>>>>>>>>>>>>>>>> - *Case 2: A:
> >> >>> Airflow.operators.foo_operator.FooOperator
> >> >>>>>>>>>>>>> could
> >> >>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> >> >>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>
> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >> >>>>>>>>>>>>>>>>>>> could
> >> >>>>>>>>>>>>>>>>>>>>> become
> >> >>> airflow.gcp.operators.bigtable.BigTableOperator*
> >> >>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> >> >>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
> >> >> suffixes
> >> >>> on
> >> >>>>>>>>>>>>> class
> >> >>>>>>>>>>>>>>>>>>> names*
> >> >>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> >> >>> case-by-case
> >> >>>>>>>>>>>>> (and
> >> >>>>>>>>>>>>>>>>> my team
> >> >>>>>>>>>>>>>>>>>>>> can
> >> >>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> >> >>>>>>>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
> >> >>>>>>>>>>>>>>>>> integration)*
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> This is my binding (+1) vote.
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> >> >>>>>>>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
> >> >>>>>>>>>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> Hey Felix,
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one
> >> >> as
> >> >>>> well
> >> >>>>>>>>>>> (I
> >> >>>>>>>>>>>>>>>>> use IDE
> >> >>>>>>>>>>>>>>>>>>>> quite
> >> >>>>>>>>>>>>>>>>>>>>>> often ;) )
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> J.
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> >> >>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together.
> I
> >> >>>> wasn't
> >> >>>>>>>>>>>>> able
> >> >>>>>>>>>>>>>>>>> to
> >> >>>>>>>>>>>>>>>>>>>> respond
> >> >>>>>>>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops
> >> >>>> throughout
> >> >>>>>>>>>>>>> Europe
> >> >>>>>>>>>>>>>>>>> :-)
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from
> >> >>> contrib
> >> >>>>>>>>>>> into
> >> >>>>>>>>>>>>> a
> >> >>>>>>>>>>>>>>>>> single
> >> >>>>>>>>>>>>>>>>>>>>>>> package.
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference
> >> >>>> would be
> >> >>>>>>>>>>>>> to
> >> >>>>>>>>>>>>>>>>> get
> >> >>>>>>>>>>>>>>>>>>> rid of
> >> >>>>>>>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or
> >> >> not
> >> >>>> is a
> >> >>>>>>>>>>>>>>>>> tricky
> >> >>>>>>>>>>>>>>>>>>>> question,
> >> >>>>>>>>>>>>>>>>>>>>>>> I think this is already reflected in the
> document
> >> >>>> since
> >> >>>>>>>>>>>>>>>>> "maturity"
> >> >>>>>>>>>>>>>>>>>>> is
> >> >>>>>>>>>>>>>>>>>>>> in
> >> >>>>>>>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the
> >> >>>> operator
> >> >>>>>>>>>>>>> to
> >> >>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>> core
> >> >>>>>>>>>>>>>>>>>>>> or
> >> >>>>>>>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
> >> >>>>>>>>>>> incubating
> >> >>>>>>>>>>>>>>>>> will
> >> >>>>>>>>>>>>>>>>>>> confuse
> >> >>>>>>>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>>>>> user even more. For people who aren't as known
> >> >> with
> >> >>>>>>>>>>>>> Airflow it
> >> >>>>>>>>>>>>>>>>> isn't
> >> >>>>>>>>>>>>>>>>>>>> that
> >> >>>>>>>>>>>>>>>>>>>>>>> transparent which operator lives where,
> especially
> >> >>> if
> >> >>>> you
> >> >>>>>>>>>>>>>>>>> don't use
> >> >>>>>>>>>>>>>>>>>>> an
> >> >>>>>>>>>>>>>>>>>>>> IDE
> >> >>>>>>>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is
> >> >>> worth
> >> >>>>>>>>>>>>> changing
> >> >>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>> public
> >> >>>>>>>>>>>>>>>>>>>>>>> API just for a different name.
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with
> >> >>> this.
> >> >>>> It
> >> >>>>>>>>>>>>> will
> >> >>>>>>>>>>>>>>>>> make
> >> >>>>>>>>>>>>>>>>>>>> easier
> >> >>>>>>>>>>>>>>>>>>>>>> as we will not have to define other rules as
> those
> >> >>>> that we
> >> >>>>>>>>>>>>>>>>> already
> >> >>>>>>>>>>>>>>>>>>> have.
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's
> >> >>> say a
> >> >>>>>>>>>>> +0)
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> I like to search on Github with t, and then
> search
> >> >>> for
> >> >>>>>>>>>>>>>>>>> BashOperator.
> >> >>>>>>>>>>>>>>>>>>>> After
> >> >>>>>>>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash.
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from
> >> >>> your
> >> >>>>>>>>>>> mail
> >> >>>>>>>>>>>>> you
> >> >>>>>>>>>>>>>>>>>>> state: -
> >> >>>>>>>>>>>>>>>>>>>>>>> *Case 2: B:
> >> >>> Airflow.operators.foo_operator.FooOperator
> >> >>>>>>>>>>>>> could
> >> >>>>>>>>>>>>>>>>> become
> >> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B
> >> >>>> solution is
> >> >>>>>>>>>>>>>>>>> keeping it
> >> >>>>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
> >> >>>>>>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator
> *would
> >> >>> stay
> >> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there
> is a
> >> >>>> little
> >> >>>>>>>>>>>>>>>>> confusion
> >> >>>>>>>>>>>>>>>>>>>> here.
> >> >>>>>>>>>>>>>>>>>>>>>> This case is not for changing BashOperator into
> >> >> Bash
> >> >>>> but
> >> >>>>>>>>>>> for
> >> >>>>>>>>>>>>>>>>> changing
> >> >>>>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>>>> *module* name not the *class* name. So
> >> >>>>>>>>>>>>>>>>> bash_operator.BashOperator ->
> >> >>>>>>>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for
> >> >>>>>>>>>>>>> BashOperator
> >> >>>>>>>>>>>>>>>>> in this
> >> >>>>>>>>>>>>>>>>>>>> case.
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide
> when
> >> >> it
> >> >>>> gets
> >> >>>>>>>>>>>>> its
> >> >>>>>>>>>>>>>>>>> own
> >> >>>>>>>>>>>>>>>>>>>> package
> >> >>>>>>>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are
> >> >>>> companies
> >> >>>>>>>>>>>>> like
> >> >>>>>>>>>>>>>>>>>>>> Databricks
> >> >>>>>>>>>>>>>>>>>>>>>>> and Qubole which might also end up with their
> own
> >> >>>> package
> >> >>>>>>>>>>>>>>>>> because
> >> >>>>>>>>>>>>>>>>>>> their
> >> >>>>>>>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving
> >> >> this
> >> >>>> is a
> >> >>>>>>>>>>>>> bit
> >> >>>>>>>>>>>>>>>>>>> painful
> >> >>>>>>>>>>>>>>>>>>>>>>> because we need to deprecate the old path and
> move
> >> >>>>>>>>>>>>> everything
> >> >>>>>>>>>>>>>>>>> which
> >> >>>>>>>>>>>>>>>>>>>>>>> introduces noise into version control etc.
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's
> >> >> not
> >> >>> as
> >> >>>>>>>>>>>>>>>>> difficult as
> >> >>>>>>>>>>>>>>>>>>> it
> >> >>>>>>>>>>>>>>>>>>>>>> seems to group operators in packages and it has
> >> >>>> numerous
> >> >>>>>>>>>>>>>>>>> advantages.
> >> >>>>>>>>>>>>>>>>>>> The
> >> >>>>>>>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in
> >> >> the
> >> >>>>>>>>>>>>>>>>> __init__.py of
> >> >>>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git
> >> >>> will
> >> >>>>>>>>>>>>> actually
> >> >>>>>>>>>>>>>>>>>>> realise
> >> >>>>>>>>>>>>>>>>>>>>>> that the file has been moved and you can follow
> >> >> that.
> >> >>>>>>>>>>> Maybe
> >> >>>>>>>>>>>>> not
> >> >>>>>>>>>>>>>>>>> as
> >> >>>>>>>>>>>>>>>>>>> super
> >> >>>>>>>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's
> >> >> not a
> >> >>>>>>>>>>> rocket
> >> >>>>>>>>>>>>>>>>> science
> >> >>>>>>>>>>>>>>>>>>>> either.
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> Yes.
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> ---
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> Apart from all, great work!
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix
> >> >> Uellendall
> >> >>>>>>>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
> >> >>>>>>>>>>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed
> change
> >> >>> of
> >> >>>>>>>>>>> these
> >> >>>>>>>>>>>>>>>>> cases.
> >> >>>>>>>>>>>>>>>>>>> So I
> >> >>>>>>>>>>>>>>>>>>>>>>> go
> >> >>>>>>>>>>>>>>>>>>>>>>>> with +1 (non-binding).
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again
> and
> >> >>> your
> >> >>>>>>>>>>>>> intense
> >> >>>>>>>>>>>>>>>>> work
> >> >>>>>>>>>>>>>>>>>>>>>>> towards
> >> >>>>>>>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for
> even
> >> >>>>>>>>>>> creating
> >> >>>>>>>>>>>>>>>>> this
> >> >>>>>>>>>>>>>>>>>>>>>>> document. I
> >> >>>>>>>>>>>>>>>>>>>>>>>> like how the code is getting more and more
> >> >>> consistent
> >> >>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>>> clean :)
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> >> >>>>>>>>>>>>>>>>>>>>>>>> Felix
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> -------- Original Message --------
> >> >>>>>>>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in
> >> >>>> import
> >> >>>>>>>>>>>>> paths.
> >> >>>>>>>>>>>>>>>>> It's
> >> >>>>>>>>>>>>>>>>>>>> been
> >> >>>>>>>>>>>>>>>>>>>>>>>>> discussed in
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July
> >> >> 30th
> >> >>>> 6pm
> >> >>>>>>>>>>>>> CEST),
> >> >>>>>>>>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>>>>> at
> >> >>>>>>>>>>>>>>>>>>>>>>> least
> >> >>>>>>>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document
> >> >> from
> >> >>>>>>>>>>> Kamil
> >> >>>>>>>>>>>>>>>>>>>>>>>>> <
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>> The proposed solution is:
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that
> >> >> are
> >> >>>>>>>>>>> "well"
> >> >>>>>>>>>>>>>>>>> tested
> >> >>>>>>>>>>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
> >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 2: B:
> >> >>>>>>>>>>> Airflow.operators.foo_operator.FooOperator
> >> >>>>>>>>>>>>>>>>> could
> >> >>>>>>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>
> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >> >>>>>>>>>>>>>>>>>>>> could
> >> >>>>>>>>>>>>>>>>>>>>>>>>> become
> >> >>>> airflow.gcp.operators.bigtable.BigTableOperator*
> >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
> >> >>>> suffixes
> >> >>>>>>>>>>> on
> >> >>>>>>>>>>>>>>>>> class
> >> >>>>>>>>>>>>>>>>>>> names*
> >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> >> >>>> case-by-case
> >> >>>>>>>>>>>>> (and
> >> >>>>>>>>>>>>>>>>> my
> >> >>>>>>>>>>>>>>>>>>> team
> >> >>>>>>>>>>>>>>>>>>>>>>> can
> >> >>>>>>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>> J.
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>> --
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> >> >>>>>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> |
> Principal
> >> >>>>>>>>>>> Software
> >> >>>>>>>>>>>>>>>>> Engineer
> >> >>>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
> >> >>>>>>>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
> >> >>>>>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> --
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> >> >>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> >> >>>> Software
> >> >>>>>>>>>>>>>>>>> Engineer
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> >> >>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> --
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> >> >>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> >> >>> Software
> >> >>>>>>>>>>>>>> Engineer
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> >> >>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> --
> >> >>>>>>>>>>>>>>>>>> *Kaxil Naik*
> >> >>>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> >> >>>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
> >> >>> Apache
> >> >>>>>>>>>>> Spark
> >> >>>>>>>>>>>>> &
> >> >>>>>>>>>>>>>>>>> Neo4j
> >> >>>>>>>>>>>>>>>>>> Developer
> >> >>>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> --
> >> >>>>>>>>>>>>>>>>> *Kaxil Naik*
> >> >>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> >> >>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
> >> >>> Apache
> >> >>>>>>>> Spark
> >> >>>>>>>>>>> &
> >> >>>>>>>>>>>>>>>>> Neo4j
> >> >>>>>>>>>>>>>>>>> Developer
> >> >>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> --
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Jarek Potiuk
> >> >>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> >> >>> Engineer
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> >> >>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >> >>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> --
> >> >>>>>>>> *Kaxil Naik*
> >> >>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> >> >>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> >> >> Spark &
> >> >>>> Neo4j
> >> >>>>>>>> Developer
> >> >>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >> >>>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> --
> >> >>>>>>>
> >> >>>>>>> Jarek Potiuk
> >> >>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> Engineer
> >> >>>>>>>
> >> >>>>>>> M: +48 660 796 129 <+48660796129>
> >> >>>>>>> [image: Polidea] <https://www.polidea.com/>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>> --
> >> >>>>>>
> >> >>>>>> Jarek Potiuk
> >> >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >> >>>>>>
> >> >>>>>> M: +48 660 796 129 <+48660796129>
> >> >>>>>> [image: Polidea] <https://www.polidea.com/>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> >>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
Re: [VOTE] Changes in import paths
Posted by Jarek Potiuk <Ja...@polidea.com>.
Seems that after this months of discussion we got an unanimous voting
result. I summarised it in the AIP-21
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths>
and
changed its status to "Accepted". Thanks for all your feedback! We will
start soon moving GCP operators following those rules and we will resolve
any initial problems with those - then we might provide some additional
learnings and see if we can easily (probably semi-automatically) move all
other operators.
Case 1
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#1airflow.contrib.{resources}>
What to do with the "contrib" folder
Case 2
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#2git*_{operator/sensor}{/s}.py>
Drop modules *_operator suffix
Case 3
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#3{aws/azure/gcp}_*>
+ Case 4
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#4Separatenamespaceforresources>
Grouping Cloud Providers operators/sensors/hooks
Case 5
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#5*Operator>
*Operator *Sensor *Hook in class name
Case 6
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#6Otherisolatedcases>
Isolated cases
Case 7
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#7>
Deprecation method
*A.* Move everything "contrib" → "airflow"
*A.* Drop the suffix.
Example:
- *airflow.operator.gcp_bigtable_operator.py*
becomes *airflow.operator.gcp_bigtable.py
<http://airflow.operator.gcp_bigtable.py>*.
*D. * Group operators/sensors/hooks in *airflow/<PROVIDER>*/operators(sensors,
hooks). Non-cloud provider ones are moved to
airflow/operators(sensors/hooks).
*Drop the prefix.*
-
*airflow/contrib/operators/sns_publish_operator.py
becomes airflow/aws/operators/**sns_publish_operator.py*
- *airflow/contrib/operators/dataproc_operator.py*
becomes *airflow/gcp/operators/dataproc_operator.py*
-
*airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
*gcp/operators/bigtable_operator.py*
-
*airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
*operators/ssh_operator.py*
*B.* No change - keep Operator/Sensor suffix in class name.
*A.* Make individual decisions of renames for operators that do not follow
common conventions used for other operators.
Consistency trumps compatibility.
Examples:
*DataProcHadoopOperator*
renamed to:
*DataprocHadoopOperator*
*A.* Native python method (with better IDE support and more flexible but a
bit more verbose)
J.
On Wed, Jul 31, 2019 at 10:30 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Yep. Agree with Ash on it. There are a number of 'action' operators
> specific for cloud providers and these should be our target. The transfer
> ones require another AIP (A lot of that already discussed in AIP-8
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
> ).
>
> J.
>
> Principal Software Engineer
> Phone: +48660796129
>
> śr., 31 lip 2019, 10:01 użytkownik Ash Berlin-Taylor <as...@apache.org>
> napisał:
>
>> This is a good idea for now. I'm also not overly concerned about these
>> few non-cloud examples - FTPtoS3Operator can stay where it is and doesn't
>> have to move under 'aws.' to my mind.
>>
>> Longer term I'd like to go back to making the "transfer/copy/transform"
>> operators "composable" so that we can have a single DataCopyOperator, and
>> give it a source and destination, and it uses hooks to do the work. This is
>> not a small undertaking though to do well and efficiently though.
>>
>> -ash
>>
>> > On 30 Jul 2019, at 20:54, Tomasz Urbaszek <to...@polidea.com>
>> wrote:
>> >
>> > Maybe we can put all those AtoB operators under one name like
>> “transfer”,
>> > then it would be easier to look for such operator?
>> >
>> > Best,
>> > Tomek
>> >
>> > On Tue, 30 Jul 2019 at 21:39, Chen Tong <ci...@gmail.com> wrote:
>> >
>> >> Daniel mentioned a good point. Such composed operator may also involves
>> >> both cloud and non-cloud provider saying FTPtoS3Operator. Should it in
>> AWS
>> >> folder?
>> >>
>> >> Also, saying in the future, another cloud provider is growing large
>> enough.
>> >> Will we rename all plugins related to this provider? What's the
>> criteria we
>> >> say it's big enough? And option D gives an impression like these tools
>> may
>> >> be maintained and supported by the Cloud provider. I am not sure if it
>> will
>> >> be a truth or not.
>> >>
>> >> Best,
>> >>
>> >>
>> >> On Tue, Jul 30, 2019 at 11:18 AM Daniel Standish <dpstandish@gmail.com
>> >
>> >> wrote:
>> >>
>> >>> One wrinkle with case 3+4 option D is inter-provider operators.
>> Mainly
>> >>> it's storage I think e.g. XToS3Operator or XToGCSOperator where the X
>> is
>> >> a
>> >>> service some different provider.
>> >>>
>> >>> Maybe the rule should be to locate the operator according to the first
>> >>> provider referenced. So e.g. s3_to_gcs_transfer_operator.py would go
>> to
>> >>> aws.
>> >>>
>> >>>
>> >>> On Tue, Jul 30, 2019 at 6:21 AM Kamil Breguła <
>> kamil.bregula@polidea.com
>> >>>
>> >>> wrote:
>> >>>
>> >>>> Yes. All changes will be backwards compatible. In the case of using
>> >>>> the old path, a message containing a proposal for change will be
>> >>>> reported to the user.
>> >>>>
>> >>>> I prepared an example of how to change the name of a class in a case
>> >>>> with the use of a native solution.
>> >>>> Source code:
>> >>>>
>> >>
>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename
>> >>>> Preview from IDE: https://imgur.com/a/45NxS5W
>> >>>>
>> >>>> On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <as...@apache.org>
>> >>> wrote:
>> >>>>>
>> >>>>> Just cos I'm not sure it's _explicitly_ stated, but all of the moves
>> >>>> will have a deprecation of the old name right?
>> >>>>>
>> >>>>> 3+4 case D gets my vote too.
>> >>>>>
>> >>>>> -a
>> >>>>>
>> >>>>>> On 30 Jul 2019, at 09:58, Jarek Potiuk <Ja...@polidea.com>
>> >>>> wrote:
>> >>>>>>
>> >>>>>> I went ahead and updated the page (just to speed it up) as I think
>> >> it
>> >>>>>> really makes sense to join those two cases (and I do not see any
>> >>>> drawbacks
>> >>>>>> - I think the options we have cover all possible approaches) and we
>> >>> can
>> >>>>>> always go back if we need to.
>> >>>>>>
>> >>>>>>
>> >>>>
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
>> >>>>>>
>> >>>>>> My vote is *D*.
>> >>>>>>
>> >>>>>> - airflow/contrib/operators/gcp_bigtable_operator.py →
>> >>> airflow/gcp
>> >>>>>> /operators/bigtable_operator.py
>> >>>>>> - airflow/contrib/operators/ssh_operator.py ->
>> >>>>>> airflow/operators/ssh_operator.py
>> >>>>>>
>> >>>>>> I updated the page with my vote.
>> >>>>>> I propose that we vote till Friday on that one (all the rest is
>> >>> already
>> >>>>>> agreed).
>> >>>>>>
>> >>>>>> J.
>> >>>>>>
>> >>>>>>
>> >>>>>> On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <
>> >>> Jarek.Potiuk@polidea.com
>> >>>>>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> I think almost everyone voted and we have almost perfect
>> >> consensus.
>> >>>> We all
>> >>>>>>> agree amongst other on moving all operators out of contrib
>> >> (Great!).
>> >>>>>>>
>> >>>>>>> The only doubts are for *Case 3* (Cloud provider prefix) and *Case
>> >>> 4*
>> >>>>>>> (Using Namespaces).
>> >>>>>>> I think there was actually an overlap between those two. Also
>> >> Ash's
>> >>>>>>> comment/veto on that is quite clear.
>> >>>>>>> So I have final (I hope!) proposal to join those two cases and
>> >> have
>> >>>> one
>> >>>>>>> voting (till Friday) only on that.
>> >>>>>>>
>> >>>>>>> I will update the doc if you all agree with me that makes more
>> >> sense
>> >>>> as
>> >>>>>>> single case to vote on:
>> >>>>>>>
>> >>>>>>> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
>> >>>>>>>
>> >>>>>>> *Option A*. Keep operators/sensors/hooks in
>> >>> airflow/operators(sensors,
>> >>>>>>> hooks) and keep/add prefixes in file names.
>> >>>>>>>
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py becomes
>> >>>>>>> airflow/operators/**aws_sns_publish_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
>> >>>>>>> becomes *airflow/operators/gcp_dataproc_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/*
>> >>>>>>> *hooks/gcp_bigtable_hook.py*
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>> >>>>>>> *operators/ssh_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> *Option B*. Group operators/sensors/hooks in
>> >>>> airflow/operators(sensors,
>> >>>>>>> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
>> >>>>>>> airflow/operators(sensors/hooks), Drop the prefix.
>> >>>>>>>
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
>> >>>>>>> becomes airflow/operators/**aws/sns_publish_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
>> >>>>>>> becomes *airflow/operators/gcp/dataproc_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
>> >>>> *airflow/*
>> >>>>>>> *operators/gcp/bigtable_operator.py*
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>> >>>>>>> *operators/ssh_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> *Option C*. Group operators/sensors/hooks in
>> >>>> airflow/operators(sensors,
>> >>>>>>> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
>> >>>>>>> airflow/operators(sensors/hooks), Keep the prefix.
>> >>>>>>>
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
>> >>>>>>> becomes airflow/operators/**aws/aws_sns_publish_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
>> >>>>>>> becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
>> >>>> *airflow/*
>> >>>>>>> *operators/gcp/gcp_bigtable_operator.py*
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>> >>>>>>> *operators/ssh_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> *Option D.* Group operators/sensors/hooks in
>> >>>> *airflow/<PROVIDER>*/operators(sensors,
>> >>>>>>> hooks). Non-cloud provider ones are moved to
>> >>>>>>> airflow/operators(sensors/hooks). Drop the prefix.
>> >>>>>>>
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
>> >>>>>>> becomes airflow/aws/operators/**sns_publish_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
>> >>>>>>> becomes *airflow/gcp/operators/dataproc_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
>> >>>> *airflow/*
>> >>>>>>> *gcp/operators/bigtable_operator.py*
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>> >>>>>>> *operators/ssh_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> *Option E.* Group operators/sensors/hooks in
>> >>>> *airflow/<PROVIDER>*/operators(sensors,
>> >>>>>>> hooks). Non-cloud provider ones are moved to
>> >>>>>>> airflow/operators(sensors/hooks).* Keep the prefix*.
>> >>>>>>>
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
>> >>>>>>> becomes airflow/aws/operators/aws_**sns_publish_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
>> >>>>>>> becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
>> >>>> *airflow/*
>> >>>>>>> *gcp/operators/gcp_bigtable_operator.py*
>> >>>>>>> -
>> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>> >>>>>>> *operators/ssh_operator.py*
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Shall I proceed with updating the page ?
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> J.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <ka...@gmail.com>
>> >>>> wrote:
>> >>>>>>>
>> >>>>>>>> Thanks Jarek, adding my vote.
>> >>>>>>>>
>> >>>>>>>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <
>> >> ash@apache.org>
>> >>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Thanks Jarek, much clearer. I have voted there too.
>> >>>>>>>>>
>> >>>>>>>>> -ash
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko
>> >> <fokko@driesprong.frl
>> >>>>
>> >>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks Jarek, the Wiki works much better for me. I've added my
>> >>> vote
>> >>>>>>>> there
>> >>>>>>>>>> as well.
>> >>>>>>>>>>
>> >>>>>>>>>> You've convinced me on the second case. I've changed my vote on
>> >>>> Case 2
>> >>>>>>>>> from
>> >>>>>>>>>> A to B :-)
>> >>>>>>>>>>
>> >>>>>>>>>> Maybe we should leave the vote open a bit longer since it
>> >>> involves
>> >>>>>>>> quite
>> >>>>>>>>>> some reading, and I would like to get some proper feedback and
>> >>>>>>>> opinions
>> >>>>>>>>> on
>> >>>>>>>>>> this. Plus I only see two votes right now. If we don't think
>> >> this
>> >>>>>>>>> through,
>> >>>>>>>>>> it might be that we're having this vote again in the near
>> >> future
>> >>>> :-)
>> >>>>>>>>>>
>> >>>>>>>>>> Cheers, Fokko
>> >>>>>>>>>>
>> >>>>>>>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
>> >>>>>>>>> Jarek.Potiuk@polidea.com>:
>> >>>>>>>>>>
>> >>>>>>>>>>> I updated the AIP-21 proposal in the way that should answer
>> >> all
>> >>>> the
>> >>>>>>>>>>> concerns in this thread.
>> >>>>>>>>>>>
>> >>>>>>>>>>> NOTE! There is an action for all of those who are interested:
>> >>>> Please
>> >>>>>>>>>>> fill-in your voting in the voting table till Tuesday 30th of
>> >>> July
>> >>>> 6pm
>> >>>>>>>>> CEST
>> >>>>>>>>>>> (5pm BST, 9 am PST).
>> >>>>>>>>>>>
>> >>>>>>>>>>> I created a summary of the proposal
>> >>>>>>>>>>> <
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
>> >>>>>>>>>>>>
>> >>>>>>>>>>> in table form - which contains also examples for each case.
>> >>>>>>>>>>>
>> >>>>>>>>>>> I added a voting table
>> >>>>>>>>>>> <
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
>> >>>>>>>>>>>>
>> >>>>>>>>>>> where you should add your votes:
>> >>>>>>>>>>>
>> >>>>>>>>>>> I propose this voting mechanism:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5
>> >>> pm
>> >>>>>>>> BST,
>> >>>>>>>>> 9am
>> >>>>>>>>>>> PST) .
>> >>>>>>>>>>>
>> >>>>>>>>>>> After the choice, the final consistent set of choices will be
>> >>>>>>>> announced
>> >>>>>>>>>>> (taking into account majority of binding vote, also including
>> >>>>>>>> potential
>> >>>>>>>>>>> vetos and consistency between the choices. Non-binding votes
>> >>> will
>> >>>> be
>> >>>>>>>>> taken
>> >>>>>>>>>>> into account in case there is a draw. The final set of choices
>> >>>> will
>> >>>>>>>> be
>> >>>>>>>>>>> announced at devlist thread
>> >>>>>>>>>>> <
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>
>> >>>
>> >>
>> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
>> >>>>>>>>>>>>
>> >>>>>>>>>>> after
>> >>>>>>>>>>> the voting completes.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Unless there is a veto raised to the final proposal, the final
>> >>>>>>>> proposal
>> >>>>>>>>> is
>> >>>>>>>>>>> accepted by Lazy Consensus
>> >>>>>>>>>>> <https://community.apache.org/committers/lazyConsensus.html>
>> >>> on
>> >>>>>>>>> *Friday*
>> >>>>>>>>>>> 02
>> >>>>>>>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
>> >>>>>>>>>>>
>> >>>>>>>>>>> Please let me know if what I proposed can be further improved
>> >> to
>> >>>>>>>> avoid
>> >>>>>>>>>>> chaos.
>> >>>>>>>>>>>
>> >>>>>>>>>>> J.
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
>> >>>>>>>> Jarek.Potiuk@polidea.com
>> >>>>>>>>>>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Ok. I see that indeed voting could have been organised a bit
>> >>>> better
>> >>>>>>>> -
>> >>>>>>>>> dev
>> >>>>>>>>>>>> list does not seem to cope well with such a complex case :).
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> As Kamil noticed - we already have a formal AIP-21
>> >>>>>>>>>>>> <
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> in the Wiki - which I should have mentioned before. I have
>> >> not
>> >>>> much
>> >>>>>>>>> time
>> >>>>>>>>>>>> today - but tomorrow I will try to summarise the options -
>> >>> based
>> >>>> on
>> >>>>>>>>> some
>> >>>>>>>>>>>> real code examples (to make it easier to digest). I think the
>> >>>> best
>> >>>>>>>>>>> approach
>> >>>>>>>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where
>> >>>> people
>> >>>>>>>>> will
>> >>>>>>>>>>>> be able to state their preferences for each case (by adding a
>> >>>> row to
>> >>>>>>>>> the
>> >>>>>>>>>>>> table). I think that might provide much better clarity and a
>> >>>> single
>> >>>>>>>>> place
>> >>>>>>>>>>>> which will show the current aggregated state of voting.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> However - as Ash also stated - some of the options are
>> >>>>>>>> inter-dependent
>> >>>>>>>>> so
>> >>>>>>>>>>>> at the end we will have to adjust the results in case they
>> >> are
>> >>>> not
>> >>>>>>>>>>>> consistent - and make final voting on aggregated set of
>> >> cases
>> >>> -
>> >>>>>>>> but I
>> >>>>>>>>>>>> think in this case following "lasy consensus" - i,e. if noone
>> >>>>>>>> objects
>> >>>>>>>>> to
>> >>>>>>>>>>>> final set of cases, we will proceed with it..
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Do you think that will work better ?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> J.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Principal Software Engineer
>> >>>>>>>>>>>> Phone: +48660796129
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
>> >>>>>>>>>>>> kamil.bregula@polidea.com> napisał:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Document is also available as AIP on wiki:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
>> >>>>>>>> <fokko@driesprong.frl
>> >>>>>>>>>>
>> >>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I agree with all, this is a bit chaotic. For the sake of
>> >>>> clarity,
>> >>>>>>>> I
>> >>>>>>>>>>>>> would
>> >>>>>>>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP.
>> >> Create a
>> >>>>>>>>>>> structural
>> >>>>>>>>>>>>>> code improvement AIP and then and add a matrix of
>> >> users/cases
>> >>>>>>>> where
>> >>>>>>>>>>>>>> everybody can add his vote per case. This gives also the
>> >>>>>>>> opportunity
>> >>>>>>>>>>> for
>> >>>>>>>>>>>>>> changing your vote on a specific case. WDYT?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Cheers, Fokko
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <
>> >>>> cixuuz@gmail.com>:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I agree with Ash. It is clear to vote on each case rather
>> >>> than
>> >>>>>>>> all
>> >>>>>>>>>>>>>>> together.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
>> >>>>>>>> ash@apache.org>
>> >>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> A number of proposals overlap or add on to each other,
>> >> so I
>> >>>>>>>> think
>> >>>>>>>>>>>>> (?) a
>> >>>>>>>>>>>>>>>> single vote makes sense.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what
>> >> we
>> >>>> are
>> >>>>>>>>>>>>> voting
>> >>>>>>>>>>>>>> on.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <
>> >>>> kaxilnaik@gmail.com>
>> >>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases
>> >>>> together,
>> >>>>>>>>>>> how
>> >>>>>>>>>>>>>>>>> about
>> >>>>>>>>>>>>>>>>> separate vote emails for all the cases? Each email can
>> >>> have
>> >>>> the
>> >>>>>>>>>>>>>>>>> description
>> >>>>>>>>>>>>>>>>> copied over from the doc.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> It would probably be a bit easier to track a decision in
>> >>> the
>> >>>>>>>>>>> future
>> >>>>>>>>>>>>> as
>> >>>>>>>>>>>>>>>>> well.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
>> >>>>>>>> Jarek.Potiuk@polidea.com>
>> >>>>>>>>>>>>>>>>> @Driesprong,
>> >>>>>>>>>>>>>>>>> Fokko <fo...@driesprong.frl>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
>> >>>>>>>> kaxilnaik@gmail.com>
>> >>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A
>> >> *
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> *Case #4* Separate namespace for resources :
>> >>>>>>>>>>>>>>>>>> *Solution A*
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> *Case #7* : *Solution A*
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
>> >>>>>>>>>>>>>>>>> <dp...@gmail.com>
>> >>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> I have tried to add some clarification to Jarek's
>> >>> summary,
>> >>>>>>>> but
>> >>>>>>>>>>> I
>> >>>>>>>>>>>>> am
>> >>>>>>>>>>>>>>>>> a
>> >>>>>>>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully
>> >>>> Jarek
>> >>>>>>>>>>> can
>> >>>>>>>>>>>>>>>>> further
>> >>>>>>>>>>>>>>>>>>> clarify.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6
>> >> are
>> >>>> all
>> >>>>>>>>>>>>> either
>> >>>>>>>>>>>>>>>>>>> proposal
>> >>>>>>>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention
>> >>>> can be
>> >>>>>>>>>>>>>>>>> focused on
>> >>>>>>>>>>>>>>>>>>> cases 1,2,3 and 7.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> *Motion is to adopt the following:*
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
>> >>>>>>>>>>>>>>>>>>> All objects will be moved out of contrib folder and
>> >> into
>> >>>>>>>>>>>>> respective
>> >>>>>>>>>>>>>>>>>>> non-contrib folder.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from
>> >>>> modules
>> >>>>>>>>>>>>>>>>>>> Example:
>> >>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
>> >>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator
>> >>>>>>>>>>>>>>>>>>> Example:
>> >>>>>>>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
>> >>>>>>>>>>>>> airflow.hooks.foo.FooHook
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
>> >>>>>>>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
>> >>>>>>>>>>>>>>>>>>> Example case is
>> >>>>>>>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
>> >>>>>>>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume
>> >> it
>> >>> is
>> >>>>>>>> now
>> >>>>>>>>>>>>> named
>> >>>>>>>>>>>>>>>>> like
>> >>>>>>>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
>> >>>>>>>>>>>>>>>>>>> So what is proposed for this case?
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Case 4 - Solution B - *no change*
>> >>>>>>>>>>>>>>>>>>> This proposal was about namespaces but the motion is
>> >> to
>> >>>>>>>> reject
>> >>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>> proposal.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Case 5 - Solution B - *no change*
>> >>>>>>>>>>>>>>>>>>> The proposal was to remove class type suffix e.g.
>> >> remove
>> >>>> the
>> >>>>>>>>>>>>>>>>> Operator in
>> >>>>>>>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal
>> >>> --
>> >>>>>>>> i.e.
>> >>>>>>>>>>>>>>>>> motion is
>> >>>>>>>>>>>>>>>>>>> no
>> >>>>>>>>>>>>>>>>>>> change on this case, i.e. we resolve to keep
>> >> "Operator"
>> >>>> (and
>> >>>>>>>>>>>>>>>>> "Sensor")
>> >>>>>>>>>>>>>>>>>>> suffixes on class names
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Case 6 - *No change*
>> >>>>>>>>>>>>>>>>>>> This case concerns object naming inconsistencies, but
>> >>>> motion
>> >>>>>>>>>>> does
>> >>>>>>>>>>>>>>>>> not
>> >>>>>>>>>>>>>>>>>>> enact
>> >>>>>>>>>>>>>>>>>>> any specific proposal.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
>> >>>>>>>>>>> deprecation
>> >>>>>>>>>>>>>>>>> warnings
>> >>>>>>>>>>>>>>>>>>> (instead of zope)
>> >>>>>>>>>>>>>>>>>>> Example:
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> # in old_package/old_mod.py
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> import warnings
>> >>>>>>>>>>>>>>>>>>> from new_package.new_mod import *
>> >>>>>>>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
>> >>>>>>>>>>>>> new_package.new_mod.
>> >>>>>>>>>>>>>>>>>>> Import
>> >>>>>>>>>>>>>>>>>>> of "
>> >>>>>>>>>>>>>>>>>>> "old_package.old_mod will become unsupported in
>> >> version
>> >>>> 2",
>> >>>>>>>>>>>>>>>>>>> DeprecationWarning, 2)
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Reference implementation here:
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>
>> >>>
>> >>
>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> I am very happy that this motion now gets rid of
>> >>> contrib.
>> >>>>>>>>>>> Thank
>> >>>>>>>>>>>>> you
>> >>>>>>>>>>>>>>>>>>> Sergio
>> >>>>>>>>>>>>>>>>>>> for turning the tide with your effective argumentation
>> >>> ;)
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
>> >>>>>>>>>>>>> ash@apache.org>
>> >>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Could someone summarise what the proposed name
>> >> changes
>> >>>> are,
>> >>>>>>>>>>>>> here,
>> >>>>>>>>>>>>>>>>> in
>> >>>>>>>>>>>>>>>>>>> this
>> >>>>>>>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving
>> >>> partial
>> >>>>>>>>>>>>> examples
>> >>>>>>>>>>>>>>>>> is 1)
>> >>>>>>>>>>>>>>>>>>> a
>> >>>>>>>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting
>> >>> on,
>> >>>> and
>> >>>>>>>>>>> 2)
>> >>>>>>>>>>>>>>>>> using a
>> >>>>>>>>>>>>>>>>>>>> google doc isn't great for a longer term record.
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>>>>>>>> Ash
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
>> >>>>>>>>>>>>>>>>> <Ja...@polidea.com>
>> >>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0
>> >> :)
>> >>>> . I
>> >>>>>>>>>>>>> would
>> >>>>>>>>>>>>>>>>> like
>> >>>>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three
>> >>> +1
>> >>>>>>>>>>>>>>>>> (binding)
>> >>>>>>>>>>>>>>>>>>> votes
>> >>>>>>>>>>>>>>>>>>>> are
>> >>>>>>>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to
>> >>> the
>> >>>>>>>>>>>>>>>>> previous
>> >>>>>>>>>>>>>>>>>>>> proposal.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Here is the proposal (details here
>> >>>>>>>>>>>>>>>>>>>>> <
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>
>> >>>
>> >>
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> )
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
>> >>>>>>>>>>>>>>>>>>>>> - *Case 2: A:
>> >>> Airflow.operators.foo_operator.FooOperator
>> >>>>>>>>>>>>> could
>> >>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
>> >>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>> >>>>>>>>>>>>>>>>>>> could
>> >>>>>>>>>>>>>>>>>>>>> become
>> >>> airflow.gcp.operators.bigtable.BigTableOperator*
>> >>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
>> >>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
>> >> suffixes
>> >>> on
>> >>>>>>>>>>>>> class
>> >>>>>>>>>>>>>>>>>>> names*
>> >>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
>> >>> case-by-case
>> >>>>>>>>>>>>> (and
>> >>>>>>>>>>>>>>>>> my team
>> >>>>>>>>>>>>>>>>>>>> can
>> >>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
>> >>>>>>>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
>> >>>>>>>>>>>>>>>>> integration)*
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> This is my binding (+1) vote.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
>> >>>>>>>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
>> >>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Hey Felix,
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one
>> >> as
>> >>>> well
>> >>>>>>>>>>> (I
>> >>>>>>>>>>>>>>>>> use IDE
>> >>>>>>>>>>>>>>>>>>>> quite
>> >>>>>>>>>>>>>>>>>>>>>> often ;) )
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> J.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
>> >>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I
>> >>>> wasn't
>> >>>>>>>>>>>>> able
>> >>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>> respond
>> >>>>>>>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops
>> >>>> throughout
>> >>>>>>>>>>>>> Europe
>> >>>>>>>>>>>>>>>>> :-)
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from
>> >>> contrib
>> >>>>>>>>>>> into
>> >>>>>>>>>>>>> a
>> >>>>>>>>>>>>>>>>> single
>> >>>>>>>>>>>>>>>>>>>>>>> package.
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference
>> >>>> would be
>> >>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>> get
>> >>>>>>>>>>>>>>>>>>> rid of
>> >>>>>>>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or
>> >> not
>> >>>> is a
>> >>>>>>>>>>>>>>>>> tricky
>> >>>>>>>>>>>>>>>>>>>> question,
>> >>>>>>>>>>>>>>>>>>>>>>> I think this is already reflected in the document
>> >>>> since
>> >>>>>>>>>>>>>>>>> "maturity"
>> >>>>>>>>>>>>>>>>>>> is
>> >>>>>>>>>>>>>>>>>>>> in
>> >>>>>>>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the
>> >>>> operator
>> >>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>> core
>> >>>>>>>>>>>>>>>>>>>> or
>> >>>>>>>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
>> >>>>>>>>>>> incubating
>> >>>>>>>>>>>>>>>>> will
>> >>>>>>>>>>>>>>>>>>> confuse
>> >>>>>>>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>> user even more. For people who aren't as known
>> >> with
>> >>>>>>>>>>>>> Airflow it
>> >>>>>>>>>>>>>>>>> isn't
>> >>>>>>>>>>>>>>>>>>>> that
>> >>>>>>>>>>>>>>>>>>>>>>> transparent which operator lives where, especially
>> >>> if
>> >>>> you
>> >>>>>>>>>>>>>>>>> don't use
>> >>>>>>>>>>>>>>>>>>> an
>> >>>>>>>>>>>>>>>>>>>> IDE
>> >>>>>>>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is
>> >>> worth
>> >>>>>>>>>>>>> changing
>> >>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>> public
>> >>>>>>>>>>>>>>>>>>>>>>> API just for a different name.
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with
>> >>> this.
>> >>>> It
>> >>>>>>>>>>>>> will
>> >>>>>>>>>>>>>>>>> make
>> >>>>>>>>>>>>>>>>>>>> easier
>> >>>>>>>>>>>>>>>>>>>>>> as we will not have to define other rules as those
>> >>>> that we
>> >>>>>>>>>>>>>>>>> already
>> >>>>>>>>>>>>>>>>>>> have.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's
>> >>> say a
>> >>>>>>>>>>> +0)
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search
>> >>> for
>> >>>>>>>>>>>>>>>>> BashOperator.
>> >>>>>>>>>>>>>>>>>>>> After
>> >>>>>>>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash.
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from
>> >>> your
>> >>>>>>>>>>> mail
>> >>>>>>>>>>>>> you
>> >>>>>>>>>>>>>>>>>>> state: -
>> >>>>>>>>>>>>>>>>>>>>>>> *Case 2: B:
>> >>> Airflow.operators.foo_operator.FooOperator
>> >>>>>>>>>>>>> could
>> >>>>>>>>>>>>>>>>> become
>> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B
>> >>>> solution is
>> >>>>>>>>>>>>>>>>> keeping it
>> >>>>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
>> >>>>>>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would
>> >>> stay
>> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a
>> >>>> little
>> >>>>>>>>>>>>>>>>> confusion
>> >>>>>>>>>>>>>>>>>>>> here.
>> >>>>>>>>>>>>>>>>>>>>>> This case is not for changing BashOperator into
>> >> Bash
>> >>>> but
>> >>>>>>>>>>> for
>> >>>>>>>>>>>>>>>>> changing
>> >>>>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>> *module* name not the *class* name. So
>> >>>>>>>>>>>>>>>>> bash_operator.BashOperator ->
>> >>>>>>>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for
>> >>>>>>>>>>>>> BashOperator
>> >>>>>>>>>>>>>>>>> in this
>> >>>>>>>>>>>>>>>>>>>> case.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when
>> >> it
>> >>>> gets
>> >>>>>>>>>>>>> its
>> >>>>>>>>>>>>>>>>> own
>> >>>>>>>>>>>>>>>>>>>> package
>> >>>>>>>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are
>> >>>> companies
>> >>>>>>>>>>>>> like
>> >>>>>>>>>>>>>>>>>>>> Databricks
>> >>>>>>>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own
>> >>>> package
>> >>>>>>>>>>>>>>>>> because
>> >>>>>>>>>>>>>>>>>>> their
>> >>>>>>>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving
>> >> this
>> >>>> is a
>> >>>>>>>>>>>>> bit
>> >>>>>>>>>>>>>>>>>>> painful
>> >>>>>>>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move
>> >>>>>>>>>>>>> everything
>> >>>>>>>>>>>>>>>>> which
>> >>>>>>>>>>>>>>>>>>>>>>> introduces noise into version control etc.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's
>> >> not
>> >>> as
>> >>>>>>>>>>>>>>>>> difficult as
>> >>>>>>>>>>>>>>>>>>> it
>> >>>>>>>>>>>>>>>>>>>>>> seems to group operators in packages and it has
>> >>>> numerous
>> >>>>>>>>>>>>>>>>> advantages.
>> >>>>>>>>>>>>>>>>>>> The
>> >>>>>>>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in
>> >> the
>> >>>>>>>>>>>>>>>>> __init__.py of
>> >>>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git
>> >>> will
>> >>>>>>>>>>>>> actually
>> >>>>>>>>>>>>>>>>>>> realise
>> >>>>>>>>>>>>>>>>>>>>>> that the file has been moved and you can follow
>> >> that.
>> >>>>>>>>>>> Maybe
>> >>>>>>>>>>>>> not
>> >>>>>>>>>>>>>>>>> as
>> >>>>>>>>>>>>>>>>>>> super
>> >>>>>>>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's
>> >> not a
>> >>>>>>>>>>> rocket
>> >>>>>>>>>>>>>>>>> science
>> >>>>>>>>>>>>>>>>>>>> either.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Yes.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> Apart from all, great work!
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix
>> >> Uellendall
>> >>>>>>>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
>> >>>>>>>>>>>>>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change
>> >>> of
>> >>>>>>>>>>> these
>> >>>>>>>>>>>>>>>>> cases.
>> >>>>>>>>>>>>>>>>>>> So I
>> >>>>>>>>>>>>>>>>>>>>>>> go
>> >>>>>>>>>>>>>>>>>>>>>>>> with +1 (non-binding).
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and
>> >>> your
>> >>>>>>>>>>>>> intense
>> >>>>>>>>>>>>>>>>> work
>> >>>>>>>>>>>>>>>>>>>>>>> towards
>> >>>>>>>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
>> >>>>>>>>>>> creating
>> >>>>>>>>>>>>>>>>> this
>> >>>>>>>>>>>>>>>>>>>>>>> document. I
>> >>>>>>>>>>>>>>>>>>>>>>>> like how the code is getting more and more
>> >>> consistent
>> >>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>> clean :)
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>> >>>>>>>>>>>>>>>>>>>>>>>> Felix
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> -------- Original Message --------
>> >>>>>>>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in
>> >>>> import
>> >>>>>>>>>>>>> paths.
>> >>>>>>>>>>>>>>>>> It's
>> >>>>>>>>>>>>>>>>>>>> been
>> >>>>>>>>>>>>>>>>>>>>>>>>> discussed in
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>
>> >>>
>> >>
>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July
>> >> 30th
>> >>>> 6pm
>> >>>>>>>>>>>>> CEST),
>> >>>>>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>>> at
>> >>>>>>>>>>>>>>>>>>>>>>> least
>> >>>>>>>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document
>> >> from
>> >>>>>>>>>>> Kamil
>> >>>>>>>>>>>>>>>>>>>>>>>>> <
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>
>> >>>
>> >>
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> The proposed solution is:
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that
>> >> are
>> >>>>>>>>>>> "well"
>> >>>>>>>>>>>>>>>>> tested
>> >>>>>>>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
>> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 2: B:
>> >>>>>>>>>>> Airflow.operators.foo_operator.FooOperator
>> >>>>>>>>>>>>>>>>> could
>> >>>>>>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
>> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>> >>>>>>>>>>>>>>>>>>>> could
>> >>>>>>>>>>>>>>>>>>>>>>>>> become
>> >>>> airflow.gcp.operators.bigtable.BigTableOperator*
>> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
>> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
>> >>>> suffixes
>> >>>>>>>>>>> on
>> >>>>>>>>>>>>>>>>> class
>> >>>>>>>>>>>>>>>>>>> names*
>> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
>> >>>> case-by-case
>> >>>>>>>>>>>>> (and
>> >>>>>>>>>>>>>>>>> my
>> >>>>>>>>>>>>>>>>>>> team
>> >>>>>>>>>>>>>>>>>>>>>>> can
>> >>>>>>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> J.
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
>> >>>>>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
>> >>>>>>>>>>> Software
>> >>>>>>>>>>>>>>>>> Engineer
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
>> >>>>>>>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
>> >>>>>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
>> >>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
>> >>>> Software
>> >>>>>>>>>>>>>>>>> Engineer
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>> >>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
>> >>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
>> >>> Software
>> >>>>>>>>>>>>>> Engineer
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>> >>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>>>>>> *Kaxil Naik*
>> >>>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>> >>>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
>> >>> Apache
>> >>>>>>>>>>> Spark
>> >>>>>>>>>>>>> &
>> >>>>>>>>>>>>>>>>> Neo4j
>> >>>>>>>>>>>>>>>>>> Developer
>> >>>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>>>>> *Kaxil Naik*
>> >>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>> >>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
>> >>> Apache
>> >>>>>>>> Spark
>> >>>>>>>>>>> &
>> >>>>>>>>>>>>>>>>> Neo4j
>> >>>>>>>>>>>>>>>>> Developer
>> >>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> --
>> >>>>>>>>>>>
>> >>>>>>>>>>> Jarek Potiuk
>> >>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
>> >>> Engineer
>> >>>>>>>>>>>
>> >>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>> >>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> *Kaxil Naik*
>> >>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>> >>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
>> >> Spark &
>> >>>> Neo4j
>> >>>>>>>> Developer
>> >>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>>
>> >>>>>>> Jarek Potiuk
>> >>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>> >>>>>>>
>> >>>>>>> M: +48 660 796 129 <+48660796129>
>> >>>>>>> [image: Polidea] <https://www.polidea.com/>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Jarek Potiuk
>> >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>> >>>>>>
>> >>>>>> M: +48 660 796 129 <+48660796129>
>> >>>>>> [image: Polidea] <https://www.polidea.com/>
>> >>>>>
>> >>>>
>> >>>
>> >>
>>
>>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Changes in import paths
Posted by Jarek Potiuk <Ja...@polidea.com>.
Yep. Agree with Ash on it. There are a number of 'action' operators
specific for cloud providers and these should be our target. The transfer
ones require another AIP (A lot of that already discussed in AIP-8
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303).
J.
Principal Software Engineer
Phone: +48660796129
śr., 31 lip 2019, 10:01 użytkownik Ash Berlin-Taylor <as...@apache.org>
napisał:
> This is a good idea for now. I'm also not overly concerned about these few
> non-cloud examples - FTPtoS3Operator can stay where it is and doesn't have
> to move under 'aws.' to my mind.
>
> Longer term I'd like to go back to making the "transfer/copy/transform"
> operators "composable" so that we can have a single DataCopyOperator, and
> give it a source and destination, and it uses hooks to do the work. This is
> not a small undertaking though to do well and efficiently though.
>
> -ash
>
> > On 30 Jul 2019, at 20:54, Tomasz Urbaszek <to...@polidea.com>
> wrote:
> >
> > Maybe we can put all those AtoB operators under one name like “transfer”,
> > then it would be easier to look for such operator?
> >
> > Best,
> > Tomek
> >
> > On Tue, 30 Jul 2019 at 21:39, Chen Tong <ci...@gmail.com> wrote:
> >
> >> Daniel mentioned a good point. Such composed operator may also involves
> >> both cloud and non-cloud provider saying FTPtoS3Operator. Should it in
> AWS
> >> folder?
> >>
> >> Also, saying in the future, another cloud provider is growing large
> enough.
> >> Will we rename all plugins related to this provider? What's the
> criteria we
> >> say it's big enough? And option D gives an impression like these tools
> may
> >> be maintained and supported by the Cloud provider. I am not sure if it
> will
> >> be a truth or not.
> >>
> >> Best,
> >>
> >>
> >> On Tue, Jul 30, 2019 at 11:18 AM Daniel Standish <dp...@gmail.com>
> >> wrote:
> >>
> >>> One wrinkle with case 3+4 option D is inter-provider operators. Mainly
> >>> it's storage I think e.g. XToS3Operator or XToGCSOperator where the X
> is
> >> a
> >>> service some different provider.
> >>>
> >>> Maybe the rule should be to locate the operator according to the first
> >>> provider referenced. So e.g. s3_to_gcs_transfer_operator.py would go
> to
> >>> aws.
> >>>
> >>>
> >>> On Tue, Jul 30, 2019 at 6:21 AM Kamil Breguła <
> kamil.bregula@polidea.com
> >>>
> >>> wrote:
> >>>
> >>>> Yes. All changes will be backwards compatible. In the case of using
> >>>> the old path, a message containing a proposal for change will be
> >>>> reported to the user.
> >>>>
> >>>> I prepared an example of how to change the name of a class in a case
> >>>> with the use of a native solution.
> >>>> Source code:
> >>>>
> >>
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename
> >>>> Preview from IDE: https://imgur.com/a/45NxS5W
> >>>>
> >>>> On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <as...@apache.org>
> >>> wrote:
> >>>>>
> >>>>> Just cos I'm not sure it's _explicitly_ stated, but all of the moves
> >>>> will have a deprecation of the old name right?
> >>>>>
> >>>>> 3+4 case D gets my vote too.
> >>>>>
> >>>>> -a
> >>>>>
> >>>>>> On 30 Jul 2019, at 09:58, Jarek Potiuk <Ja...@polidea.com>
> >>>> wrote:
> >>>>>>
> >>>>>> I went ahead and updated the page (just to speed it up) as I think
> >> it
> >>>>>> really makes sense to join those two cases (and I do not see any
> >>>> drawbacks
> >>>>>> - I think the options we have cover all possible approaches) and we
> >>> can
> >>>>>> always go back if we need to.
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> >>>>>>
> >>>>>> My vote is *D*.
> >>>>>>
> >>>>>> - airflow/contrib/operators/gcp_bigtable_operator.py →
> >>> airflow/gcp
> >>>>>> /operators/bigtable_operator.py
> >>>>>> - airflow/contrib/operators/ssh_operator.py ->
> >>>>>> airflow/operators/ssh_operator.py
> >>>>>>
> >>>>>> I updated the page with my vote.
> >>>>>> I propose that we vote till Friday on that one (all the rest is
> >>> already
> >>>>>> agreed).
> >>>>>>
> >>>>>> J.
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <
> >>> Jarek.Potiuk@polidea.com
> >>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I think almost everyone voted and we have almost perfect
> >> consensus.
> >>>> We all
> >>>>>>> agree amongst other on moving all operators out of contrib
> >> (Great!).
> >>>>>>>
> >>>>>>> The only doubts are for *Case 3* (Cloud provider prefix) and *Case
> >>> 4*
> >>>>>>> (Using Namespaces).
> >>>>>>> I think there was actually an overlap between those two. Also
> >> Ash's
> >>>>>>> comment/veto on that is quite clear.
> >>>>>>> So I have final (I hope!) proposal to join those two cases and
> >> have
> >>>> one
> >>>>>>> voting (till Friday) only on that.
> >>>>>>>
> >>>>>>> I will update the doc if you all agree with me that makes more
> >> sense
> >>>> as
> >>>>>>> single case to vote on:
> >>>>>>>
> >>>>>>> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
> >>>>>>>
> >>>>>>> *Option A*. Keep operators/sensors/hooks in
> >>> airflow/operators(sensors,
> >>>>>>> hooks) and keep/add prefixes in file names.
> >>>>>>>
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py becomes
> >>>>>>> airflow/operators/**aws_sns_publish_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> >>>>>>> becomes *airflow/operators/gcp_dataproc_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> -
> >>>>>>> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/*
> >>>>>>> *hooks/gcp_bigtable_hook.py*
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >>>>>>> *operators/ssh_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> *Option B*. Group operators/sensors/hooks in
> >>>> airflow/operators(sensors,
> >>>>>>> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
> >>>>>>> airflow/operators(sensors/hooks), Drop the prefix.
> >>>>>>>
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> >>>>>>> becomes airflow/operators/**aws/sns_publish_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> >>>>>>> becomes *airflow/operators/gcp/dataproc_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> >>>> *airflow/*
> >>>>>>> *operators/gcp/bigtable_operator.py*
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >>>>>>> *operators/ssh_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> *Option C*. Group operators/sensors/hooks in
> >>>> airflow/operators(sensors,
> >>>>>>> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
> >>>>>>> airflow/operators(sensors/hooks), Keep the prefix.
> >>>>>>>
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> >>>>>>> becomes airflow/operators/**aws/aws_sns_publish_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> >>>>>>> becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> >>>> *airflow/*
> >>>>>>> *operators/gcp/gcp_bigtable_operator.py*
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >>>>>>> *operators/ssh_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> *Option D.* Group operators/sensors/hooks in
> >>>> *airflow/<PROVIDER>*/operators(sensors,
> >>>>>>> hooks). Non-cloud provider ones are moved to
> >>>>>>> airflow/operators(sensors/hooks). Drop the prefix.
> >>>>>>>
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> >>>>>>> becomes airflow/aws/operators/**sns_publish_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> >>>>>>> becomes *airflow/gcp/operators/dataproc_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> >>>> *airflow/*
> >>>>>>> *gcp/operators/bigtable_operator.py*
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >>>>>>> *operators/ssh_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> *Option E.* Group operators/sensors/hooks in
> >>>> *airflow/<PROVIDER>*/operators(sensors,
> >>>>>>> hooks). Non-cloud provider ones are moved to
> >>>>>>> airflow/operators(sensors/hooks).* Keep the prefix*.
> >>>>>>>
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py
> >>>>>>> becomes airflow/aws/operators/aws_**sns_publish_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
> >>>>>>> becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> >>>> *airflow/*
> >>>>>>> *gcp/operators/gcp_bigtable_operator.py*
> >>>>>>> -
> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >>>>>>> *operators/ssh_operator.py*
> >>>>>>>
> >>>>>>>
> >>>>>>> Shall I proceed with updating the page ?
> >>>>>>>
> >>>>>>>
> >>>>>>> J.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <ka...@gmail.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> Thanks Jarek, adding my vote.
> >>>>>>>>
> >>>>>>>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <
> >> ash@apache.org>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks Jarek, much clearer. I have voted there too.
> >>>>>>>>>
> >>>>>>>>> -ash
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko
> >> <fokko@driesprong.frl
> >>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Thanks Jarek, the Wiki works much better for me. I've added my
> >>> vote
> >>>>>>>> there
> >>>>>>>>>> as well.
> >>>>>>>>>>
> >>>>>>>>>> You've convinced me on the second case. I've changed my vote on
> >>>> Case 2
> >>>>>>>>> from
> >>>>>>>>>> A to B :-)
> >>>>>>>>>>
> >>>>>>>>>> Maybe we should leave the vote open a bit longer since it
> >>> involves
> >>>>>>>> quite
> >>>>>>>>>> some reading, and I would like to get some proper feedback and
> >>>>>>>> opinions
> >>>>>>>>> on
> >>>>>>>>>> this. Plus I only see two votes right now. If we don't think
> >> this
> >>>>>>>>> through,
> >>>>>>>>>> it might be that we're having this vote again in the near
> >> future
> >>>> :-)
> >>>>>>>>>>
> >>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>
> >>>>>>>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
> >>>>>>>>> Jarek.Potiuk@polidea.com>:
> >>>>>>>>>>
> >>>>>>>>>>> I updated the AIP-21 proposal in the way that should answer
> >> all
> >>>> the
> >>>>>>>>>>> concerns in this thread.
> >>>>>>>>>>>
> >>>>>>>>>>> NOTE! There is an action for all of those who are interested:
> >>>> Please
> >>>>>>>>>>> fill-in your voting in the voting table till Tuesday 30th of
> >>> July
> >>>> 6pm
> >>>>>>>>> CEST
> >>>>>>>>>>> (5pm BST, 9 am PST).
> >>>>>>>>>>>
> >>>>>>>>>>> I created a summary of the proposal
> >>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> >>>>>>>>>>>>
> >>>>>>>>>>> in table form - which contains also examples for each case.
> >>>>>>>>>>>
> >>>>>>>>>>> I added a voting table
> >>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> >>>>>>>>>>>>
> >>>>>>>>>>> where you should add your votes:
> >>>>>>>>>>>
> >>>>>>>>>>> I propose this voting mechanism:
> >>>>>>>>>>>
> >>>>>>>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5
> >>> pm
> >>>>>>>> BST,
> >>>>>>>>> 9am
> >>>>>>>>>>> PST) .
> >>>>>>>>>>>
> >>>>>>>>>>> After the choice, the final consistent set of choices will be
> >>>>>>>> announced
> >>>>>>>>>>> (taking into account majority of binding vote, also including
> >>>>>>>> potential
> >>>>>>>>>>> vetos and consistency between the choices. Non-binding votes
> >>> will
> >>>> be
> >>>>>>>>> taken
> >>>>>>>>>>> into account in case there is a draw. The final set of choices
> >>>> will
> >>>>>>>> be
> >>>>>>>>>>> announced at devlist thread
> >>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>
> >>>
> >>
> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> >>>>>>>>>>>>
> >>>>>>>>>>> after
> >>>>>>>>>>> the voting completes.
> >>>>>>>>>>>
> >>>>>>>>>>> Unless there is a veto raised to the final proposal, the final
> >>>>>>>> proposal
> >>>>>>>>> is
> >>>>>>>>>>> accepted by Lazy Consensus
> >>>>>>>>>>> <https://community.apache.org/committers/lazyConsensus.html>
> >>> on
> >>>>>>>>> *Friday*
> >>>>>>>>>>> 02
> >>>>>>>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
> >>>>>>>>>>>
> >>>>>>>>>>> Please let me know if what I proposed can be further improved
> >> to
> >>>>>>>> avoid
> >>>>>>>>>>> chaos.
> >>>>>>>>>>>
> >>>>>>>>>>> J.
> >>>>>>>>>>>
> >>>>>>>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
> >>>>>>>> Jarek.Potiuk@polidea.com
> >>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Ok. I see that indeed voting could have been organised a bit
> >>>> better
> >>>>>>>> -
> >>>>>>>>> dev
> >>>>>>>>>>>> list does not seem to cope well with such a complex case :).
> >>>>>>>>>>>>
> >>>>>>>>>>>> As Kamil noticed - we already have a formal AIP-21
> >>>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >>>>>>>>>>>>
> >>>>>>>>>>>> in the Wiki - which I should have mentioned before. I have
> >> not
> >>>> much
> >>>>>>>>> time
> >>>>>>>>>>>> today - but tomorrow I will try to summarise the options -
> >>> based
> >>>> on
> >>>>>>>>> some
> >>>>>>>>>>>> real code examples (to make it easier to digest). I think the
> >>>> best
> >>>>>>>>>>> approach
> >>>>>>>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where
> >>>> people
> >>>>>>>>> will
> >>>>>>>>>>>> be able to state their preferences for each case (by adding a
> >>>> row to
> >>>>>>>>> the
> >>>>>>>>>>>> table). I think that might provide much better clarity and a
> >>>> single
> >>>>>>>>> place
> >>>>>>>>>>>> which will show the current aggregated state of voting.
> >>>>>>>>>>>>
> >>>>>>>>>>>> However - as Ash also stated - some of the options are
> >>>>>>>> inter-dependent
> >>>>>>>>> so
> >>>>>>>>>>>> at the end we will have to adjust the results in case they
> >> are
> >>>> not
> >>>>>>>>>>>> consistent - and make final voting on aggregated set of
> >> cases
> >>> -
> >>>>>>>> but I
> >>>>>>>>>>>> think in this case following "lasy consensus" - i,e. if noone
> >>>>>>>> objects
> >>>>>>>>> to
> >>>>>>>>>>>> final set of cases, we will proceed with it..
> >>>>>>>>>>>>
> >>>>>>>>>>>> Do you think that will work better ?
> >>>>>>>>>>>>
> >>>>>>>>>>>> J.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Principal Software Engineer
> >>>>>>>>>>>> Phone: +48660796129
> >>>>>>>>>>>>
> >>>>>>>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> >>>>>>>>>>>> kamil.bregula@polidea.com> napisał:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Document is also available as AIP on wiki:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
> >>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I agree with all, this is a bit chaotic. For the sake of
> >>>> clarity,
> >>>>>>>> I
> >>>>>>>>>>>>> would
> >>>>>>>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP.
> >> Create a
> >>>>>>>>>>> structural
> >>>>>>>>>>>>>> code improvement AIP and then and add a matrix of
> >> users/cases
> >>>>>>>> where
> >>>>>>>>>>>>>> everybody can add his vote per case. This gives also the
> >>>>>>>> opportunity
> >>>>>>>>>>> for
> >>>>>>>>>>>>>> changing your vote on a specific case. WDYT?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <
> >>>> cixuuz@gmail.com>:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I agree with Ash. It is clear to vote on each case rather
> >>> than
> >>>>>>>> all
> >>>>>>>>>>>>>>> together.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
> >>>>>>>> ash@apache.org>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> A number of proposals overlap or add on to each other,
> >> so I
> >>>>>>>> think
> >>>>>>>>>>>>> (?) a
> >>>>>>>>>>>>>>>> single vote makes sense.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what
> >> we
> >>>> are
> >>>>>>>>>>>>> voting
> >>>>>>>>>>>>>> on.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <
> >>>> kaxilnaik@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases
> >>>> together,
> >>>>>>>>>>> how
> >>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>> separate vote emails for all the cases? Each email can
> >>> have
> >>>> the
> >>>>>>>>>>>>>>>>> description
> >>>>>>>>>>>>>>>>> copied over from the doc.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It would probably be a bit easier to track a decision in
> >>> the
> >>>>>>>>>>> future
> >>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>> well.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
> >>>>>>>> Jarek.Potiuk@polidea.com>
> >>>>>>>>>>>>>>>>> @Driesprong,
> >>>>>>>>>>>>>>>>> Fokko <fo...@driesprong.frl>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
> >>>>>>>> kaxilnaik@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A
> >> *
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case #4* Separate namespace for resources :
> >>>>>>>>>>>>>>>>>> *Solution A*
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case #7* : *Solution A*
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> >>>>>>>>>>>>>>>>> <dp...@gmail.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I have tried to add some clarification to Jarek's
> >>> summary,
> >>>>>>>> but
> >>>>>>>>>>> I
> >>>>>>>>>>>>> am
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully
> >>>> Jarek
> >>>>>>>>>>> can
> >>>>>>>>>>>>>>>>> further
> >>>>>>>>>>>>>>>>>>> clarify.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6
> >> are
> >>>> all
> >>>>>>>>>>>>> either
> >>>>>>>>>>>>>>>>>>> proposal
> >>>>>>>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention
> >>>> can be
> >>>>>>>>>>>>>>>>> focused on
> >>>>>>>>>>>>>>>>>>> cases 1,2,3 and 7.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> *Motion is to adopt the following:*
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
> >>>>>>>>>>>>>>>>>>> All objects will be moved out of contrib folder and
> >> into
> >>>>>>>>>>>>> respective
> >>>>>>>>>>>>>>>>>>> non-contrib folder.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from
> >>>> modules
> >>>>>>>>>>>>>>>>>>> Example:
> >>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
> >>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator
> >>>>>>>>>>>>>>>>>>> Example:
> >>>>>>>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
> >>>>>>>>>>>>> airflow.hooks.foo.FooHook
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
> >>>>>>>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
> >>>>>>>>>>>>>>>>>>> Example case is
> >>>>>>>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
> >>>>>>>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume
> >> it
> >>> is
> >>>>>>>> now
> >>>>>>>>>>>>> named
> >>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
> >>>>>>>>>>>>>>>>>>> So what is proposed for this case?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Case 4 - Solution B - *no change*
> >>>>>>>>>>>>>>>>>>> This proposal was about namespaces but the motion is
> >> to
> >>>>>>>> reject
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> proposal.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Case 5 - Solution B - *no change*
> >>>>>>>>>>>>>>>>>>> The proposal was to remove class type suffix e.g.
> >> remove
> >>>> the
> >>>>>>>>>>>>>>>>> Operator in
> >>>>>>>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal
> >>> --
> >>>>>>>> i.e.
> >>>>>>>>>>>>>>>>> motion is
> >>>>>>>>>>>>>>>>>>> no
> >>>>>>>>>>>>>>>>>>> change on this case, i.e. we resolve to keep
> >> "Operator"
> >>>> (and
> >>>>>>>>>>>>>>>>> "Sensor")
> >>>>>>>>>>>>>>>>>>> suffixes on class names
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Case 6 - *No change*
> >>>>>>>>>>>>>>>>>>> This case concerns object naming inconsistencies, but
> >>>> motion
> >>>>>>>>>>> does
> >>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>> enact
> >>>>>>>>>>>>>>>>>>> any specific proposal.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
> >>>>>>>>>>> deprecation
> >>>>>>>>>>>>>>>>> warnings
> >>>>>>>>>>>>>>>>>>> (instead of zope)
> >>>>>>>>>>>>>>>>>>> Example:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> # in old_package/old_mod.py
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> import warnings
> >>>>>>>>>>>>>>>>>>> from new_package.new_mod import *
> >>>>>>>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
> >>>>>>>>>>>>> new_package.new_mod.
> >>>>>>>>>>>>>>>>>>> Import
> >>>>>>>>>>>>>>>>>>> of "
> >>>>>>>>>>>>>>>>>>> "old_package.old_mod will become unsupported in
> >> version
> >>>> 2",
> >>>>>>>>>>>>>>>>>>> DeprecationWarning, 2)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Reference implementation here:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>
> >>>
> >>
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I am very happy that this motion now gets rid of
> >>> contrib.
> >>>>>>>>>>> Thank
> >>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>> Sergio
> >>>>>>>>>>>>>>>>>>> for turning the tide with your effective argumentation
> >>> ;)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
> >>>>>>>>>>>>> ash@apache.org>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Could someone summarise what the proposed name
> >> changes
> >>>> are,
> >>>>>>>>>>>>> here,
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving
> >>> partial
> >>>>>>>>>>>>> examples
> >>>>>>>>>>>>>>>>> is 1)
> >>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting
> >>> on,
> >>>> and
> >>>>>>>>>>> 2)
> >>>>>>>>>>>>>>>>> using a
> >>>>>>>>>>>>>>>>>>>> google doc isn't great for a longer term record.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>> Ash
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
> >>>>>>>>>>>>>>>>> <Ja...@polidea.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0
> >> :)
> >>>> . I
> >>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three
> >>> +1
> >>>>>>>>>>>>>>>>> (binding)
> >>>>>>>>>>>>>>>>>>> votes
> >>>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to
> >>> the
> >>>>>>>>>>>>>>>>> previous
> >>>>>>>>>>>>>>>>>>>> proposal.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Here is the proposal (details here
> >>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>
> >>>
> >>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
> >>>>>>>>>>>>>>>>>>>>> - *Case 2: A:
> >>> Airflow.operators.foo_operator.FooOperator
> >>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> >>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >>>>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>> become
> >>> airflow.gcp.operators.bigtable.BigTableOperator*
> >>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> >>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
> >> suffixes
> >>> on
> >>>>>>>>>>>>> class
> >>>>>>>>>>>>>>>>>>> names*
> >>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> >>> case-by-case
> >>>>>>>>>>>>> (and
> >>>>>>>>>>>>>>>>> my team
> >>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> >>>>>>>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
> >>>>>>>>>>>>>>>>> integration)*
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> This is my binding (+1) vote.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> >>>>>>>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hey Felix,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one
> >> as
> >>>> well
> >>>>>>>>>>> (I
> >>>>>>>>>>>>>>>>> use IDE
> >>>>>>>>>>>>>>>>>>>> quite
> >>>>>>>>>>>>>>>>>>>>>> often ;) )
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> J.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> >>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I
> >>>> wasn't
> >>>>>>>>>>>>> able
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> respond
> >>>>>>>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops
> >>>> throughout
> >>>>>>>>>>>>> Europe
> >>>>>>>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from
> >>> contrib
> >>>>>>>>>>> into
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> single
> >>>>>>>>>>>>>>>>>>>>>>> package.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference
> >>>> would be
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> get
> >>>>>>>>>>>>>>>>>>> rid of
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or
> >> not
> >>>> is a
> >>>>>>>>>>>>>>>>> tricky
> >>>>>>>>>>>>>>>>>>>> question,
> >>>>>>>>>>>>>>>>>>>>>>> I think this is already reflected in the document
> >>>> since
> >>>>>>>>>>>>>>>>> "maturity"
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the
> >>>> operator
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> core
> >>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
> >>>>>>>>>>> incubating
> >>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>> confuse
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> user even more. For people who aren't as known
> >> with
> >>>>>>>>>>>>> Airflow it
> >>>>>>>>>>>>>>>>> isn't
> >>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>> transparent which operator lives where, especially
> >>> if
> >>>> you
> >>>>>>>>>>>>>>>>> don't use
> >>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>> IDE
> >>>>>>>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is
> >>> worth
> >>>>>>>>>>>>> changing
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> public
> >>>>>>>>>>>>>>>>>>>>>>> API just for a different name.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with
> >>> this.
> >>>> It
> >>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> make
> >>>>>>>>>>>>>>>>>>>> easier
> >>>>>>>>>>>>>>>>>>>>>> as we will not have to define other rules as those
> >>>> that we
> >>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>> have.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's
> >>> say a
> >>>>>>>>>>> +0)
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search
> >>> for
> >>>>>>>>>>>>>>>>> BashOperator.
> >>>>>>>>>>>>>>>>>>>> After
> >>>>>>>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from
> >>> your
> >>>>>>>>>>> mail
> >>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>> state: -
> >>>>>>>>>>>>>>>>>>>>>>> *Case 2: B:
> >>> Airflow.operators.foo_operator.FooOperator
> >>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>> become
> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B
> >>>> solution is
> >>>>>>>>>>>>>>>>> keeping it
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
> >>>>>>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would
> >>> stay
> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a
> >>>> little
> >>>>>>>>>>>>>>>>> confusion
> >>>>>>>>>>>>>>>>>>>> here.
> >>>>>>>>>>>>>>>>>>>>>> This case is not for changing BashOperator into
> >> Bash
> >>>> but
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> changing
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> *module* name not the *class* name. So
> >>>>>>>>>>>>>>>>> bash_operator.BashOperator ->
> >>>>>>>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for
> >>>>>>>>>>>>> BashOperator
> >>>>>>>>>>>>>>>>> in this
> >>>>>>>>>>>>>>>>>>>> case.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when
> >> it
> >>>> gets
> >>>>>>>>>>>>> its
> >>>>>>>>>>>>>>>>> own
> >>>>>>>>>>>>>>>>>>>> package
> >>>>>>>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are
> >>>> companies
> >>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>> Databricks
> >>>>>>>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own
> >>>> package
> >>>>>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>>>> their
> >>>>>>>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving
> >> this
> >>>> is a
> >>>>>>>>>>>>> bit
> >>>>>>>>>>>>>>>>>>> painful
> >>>>>>>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move
> >>>>>>>>>>>>> everything
> >>>>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>>>>>> introduces noise into version control etc.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's
> >> not
> >>> as
> >>>>>>>>>>>>>>>>> difficult as
> >>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>> seems to group operators in packages and it has
> >>>> numerous
> >>>>>>>>>>>>>>>>> advantages.
> >>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in
> >> the
> >>>>>>>>>>>>>>>>> __init__.py of
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git
> >>> will
> >>>>>>>>>>>>> actually
> >>>>>>>>>>>>>>>>>>> realise
> >>>>>>>>>>>>>>>>>>>>>> that the file has been moved and you can follow
> >> that.
> >>>>>>>>>>> Maybe
> >>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>> super
> >>>>>>>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's
> >> not a
> >>>>>>>>>>> rocket
> >>>>>>>>>>>>>>>>> science
> >>>>>>>>>>>>>>>>>>>> either.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Yes.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Apart from all, great work!
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix
> >> Uellendall
> >>>>>>>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
> >>>>>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change
> >>> of
> >>>>>>>>>>> these
> >>>>>>>>>>>>>>>>> cases.
> >>>>>>>>>>>>>>>>>>> So I
> >>>>>>>>>>>>>>>>>>>>>>> go
> >>>>>>>>>>>>>>>>>>>>>>>> with +1 (non-binding).
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and
> >>> your
> >>>>>>>>>>>>> intense
> >>>>>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>>>>>>>> towards
> >>>>>>>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
> >>>>>>>>>>> creating
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>> document. I
> >>>>>>>>>>>>>>>>>>>>>>>> like how the code is getting more and more
> >>> consistent
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> clean :)
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> >>>>>>>>>>>>>>>>>>>>>>>> Felix
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> -------- Original Message --------
> >>>>>>>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in
> >>>> import
> >>>>>>>>>>>>> paths.
> >>>>>>>>>>>>>>>>> It's
> >>>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>>>> discussed in
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>
> >>>
> >>
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July
> >> 30th
> >>>> 6pm
> >>>>>>>>>>>>> CEST),
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>> least
> >>>>>>>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document
> >> from
> >>>>>>>>>>> Kamil
> >>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>
> >>>
> >>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The proposed solution is:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that
> >> are
> >>>>>>>>>>> "well"
> >>>>>>>>>>>>>>>>> tested
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 2: B:
> >>>>>>>>>>> Airflow.operators.foo_operator.FooOperator
> >>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >>>>>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>>>>>> become
> >>>> airflow.gcp.operators.bigtable.BigTableOperator*
> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
> >>>> suffixes
> >>>>>>>>>>> on
> >>>>>>>>>>>>>>>>> class
> >>>>>>>>>>>>>>>>>>> names*
> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> >>>> case-by-case
> >>>>>>>>>>>>> (and
> >>>>>>>>>>>>>>>>> my
> >>>>>>>>>>>>>>>>>>> team
> >>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> J.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> >>>>>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> >>>>>>>>>>> Software
> >>>>>>>>>>>>>>>>> Engineer
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
> >>>>>>>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
> >>>>>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> >>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> >>>> Software
> >>>>>>>>>>>>>>>>> Engineer
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> >>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> >>> Software
> >>>>>>>>>>>>>> Engineer
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>> *Kaxil Naik*
> >>>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> >>>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
> >>> Apache
> >>>>>>>>>>> Spark
> >>>>>>>>>>>>> &
> >>>>>>>>>>>>>>>>> Neo4j
> >>>>>>>>>>>>>>>>>> Developer
> >>>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>> *Kaxil Naik*
> >>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> >>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
> >>> Apache
> >>>>>>>> Spark
> >>>>>>>>>>> &
> >>>>>>>>>>>>>>>>> Neo4j
> >>>>>>>>>>>>>>>>> Developer
> >>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>>
> >>>>>>>>>>> Jarek Potiuk
> >>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> >>> Engineer
> >>>>>>>>>>>
> >>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> *Kaxil Naik*
> >>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> >>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> >> Spark &
> >>>> Neo4j
> >>>>>>>> Developer
> >>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>>
> >>>>>>> Jarek Potiuk
> >>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>>>>>
> >>>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Jarek Potiuk
> >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>>>>
> >>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: [VOTE] Changes in import paths
Posted by Ash Berlin-Taylor <as...@apache.org>.
This is a good idea for now. I'm also not overly concerned about these few non-cloud examples - FTPtoS3Operator can stay where it is and doesn't have to move under 'aws.' to my mind.
Longer term I'd like to go back to making the "transfer/copy/transform" operators "composable" so that we can have a single DataCopyOperator, and give it a source and destination, and it uses hooks to do the work. This is not a small undertaking though to do well and efficiently though.
-ash
> On 30 Jul 2019, at 20:54, Tomasz Urbaszek <to...@polidea.com> wrote:
>
> Maybe we can put all those AtoB operators under one name like “transfer”,
> then it would be easier to look for such operator?
>
> Best,
> Tomek
>
> On Tue, 30 Jul 2019 at 21:39, Chen Tong <ci...@gmail.com> wrote:
>
>> Daniel mentioned a good point. Such composed operator may also involves
>> both cloud and non-cloud provider saying FTPtoS3Operator. Should it in AWS
>> folder?
>>
>> Also, saying in the future, another cloud provider is growing large enough.
>> Will we rename all plugins related to this provider? What's the criteria we
>> say it's big enough? And option D gives an impression like these tools may
>> be maintained and supported by the Cloud provider. I am not sure if it will
>> be a truth or not.
>>
>> Best,
>>
>>
>> On Tue, Jul 30, 2019 at 11:18 AM Daniel Standish <dp...@gmail.com>
>> wrote:
>>
>>> One wrinkle with case 3+4 option D is inter-provider operators. Mainly
>>> it's storage I think e.g. XToS3Operator or XToGCSOperator where the X is
>> a
>>> service some different provider.
>>>
>>> Maybe the rule should be to locate the operator according to the first
>>> provider referenced. So e.g. s3_to_gcs_transfer_operator.py would go to
>>> aws.
>>>
>>>
>>> On Tue, Jul 30, 2019 at 6:21 AM Kamil Breguła <kamil.bregula@polidea.com
>>>
>>> wrote:
>>>
>>>> Yes. All changes will be backwards compatible. In the case of using
>>>> the old path, a message containing a proposal for change will be
>>>> reported to the user.
>>>>
>>>> I prepared an example of how to change the name of a class in a case
>>>> with the use of a native solution.
>>>> Source code:
>>>>
>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename
>>>> Preview from IDE: https://imgur.com/a/45NxS5W
>>>>
>>>> On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <as...@apache.org>
>>> wrote:
>>>>>
>>>>> Just cos I'm not sure it's _explicitly_ stated, but all of the moves
>>>> will have a deprecation of the old name right?
>>>>>
>>>>> 3+4 case D gets my vote too.
>>>>>
>>>>> -a
>>>>>
>>>>>> On 30 Jul 2019, at 09:58, Jarek Potiuk <Ja...@polidea.com>
>>>> wrote:
>>>>>>
>>>>>> I went ahead and updated the page (just to speed it up) as I think
>> it
>>>>>> really makes sense to join those two cases (and I do not see any
>>>> drawbacks
>>>>>> - I think the options we have cover all possible approaches) and we
>>> can
>>>>>> always go back if we need to.
>>>>>>
>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
>>>>>>
>>>>>> My vote is *D*.
>>>>>>
>>>>>> - airflow/contrib/operators/gcp_bigtable_operator.py →
>>> airflow/gcp
>>>>>> /operators/bigtable_operator.py
>>>>>> - airflow/contrib/operators/ssh_operator.py ->
>>>>>> airflow/operators/ssh_operator.py
>>>>>>
>>>>>> I updated the page with my vote.
>>>>>> I propose that we vote till Friday on that one (all the rest is
>>> already
>>>>>> agreed).
>>>>>>
>>>>>> J.
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <
>>> Jarek.Potiuk@polidea.com
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>> I think almost everyone voted and we have almost perfect
>> consensus.
>>>> We all
>>>>>>> agree amongst other on moving all operators out of contrib
>> (Great!).
>>>>>>>
>>>>>>> The only doubts are for *Case 3* (Cloud provider prefix) and *Case
>>> 4*
>>>>>>> (Using Namespaces).
>>>>>>> I think there was actually an overlap between those two. Also
>> Ash's
>>>>>>> comment/veto on that is quite clear.
>>>>>>> So I have final (I hope!) proposal to join those two cases and
>> have
>>>> one
>>>>>>> voting (till Friday) only on that.
>>>>>>>
>>>>>>> I will update the doc if you all agree with me that makes more
>> sense
>>>> as
>>>>>>> single case to vote on:
>>>>>>>
>>>>>>> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
>>>>>>>
>>>>>>> *Option A*. Keep operators/sensors/hooks in
>>> airflow/operators(sensors,
>>>>>>> hooks) and keep/add prefixes in file names.
>>>>>>>
>>>>>>> -
>>>>>>> *airflow/contrib/operators/sns_publish_operator.py becomes
>>>>>>> airflow/operators/**aws_sns_publish_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
>>>>>>> becomes *airflow/operators/gcp_dataproc_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> -
>>>>>>> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/*
>>>>>>> *hooks/gcp_bigtable_hook.py*
>>>>>>> -
>>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>>>>>>> *operators/ssh_operator.py*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Option B*. Group operators/sensors/hooks in
>>>> airflow/operators(sensors,
>>>>>>> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
>>>>>>> airflow/operators(sensors/hooks), Drop the prefix.
>>>>>>>
>>>>>>> -
>>>>>>> *airflow/contrib/operators/sns_publish_operator.py
>>>>>>> becomes airflow/operators/**aws/sns_publish_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
>>>>>>> becomes *airflow/operators/gcp/dataproc_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> -
>>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
>>>> *airflow/*
>>>>>>> *operators/gcp/bigtable_operator.py*
>>>>>>> -
>>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>>>>>>> *operators/ssh_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> *Option C*. Group operators/sensors/hooks in
>>>> airflow/operators(sensors,
>>>>>>> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
>>>>>>> airflow/operators(sensors/hooks), Keep the prefix.
>>>>>>>
>>>>>>> -
>>>>>>> *airflow/contrib/operators/sns_publish_operator.py
>>>>>>> becomes airflow/operators/**aws/aws_sns_publish_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
>>>>>>> becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> -
>>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
>>>> *airflow/*
>>>>>>> *operators/gcp/gcp_bigtable_operator.py*
>>>>>>> -
>>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>>>>>>> *operators/ssh_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> *Option D.* Group operators/sensors/hooks in
>>>> *airflow/<PROVIDER>*/operators(sensors,
>>>>>>> hooks). Non-cloud provider ones are moved to
>>>>>>> airflow/operators(sensors/hooks). Drop the prefix.
>>>>>>>
>>>>>>> -
>>>>>>> *airflow/contrib/operators/sns_publish_operator.py
>>>>>>> becomes airflow/aws/operators/**sns_publish_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
>>>>>>> becomes *airflow/gcp/operators/dataproc_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> -
>>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
>>>> *airflow/*
>>>>>>> *gcp/operators/bigtable_operator.py*
>>>>>>> -
>>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>>>>>>> *operators/ssh_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> *Option E.* Group operators/sensors/hooks in
>>>> *airflow/<PROVIDER>*/operators(sensors,
>>>>>>> hooks). Non-cloud provider ones are moved to
>>>>>>> airflow/operators(sensors/hooks).* Keep the prefix*.
>>>>>>>
>>>>>>> -
>>>>>>> *airflow/contrib/operators/sns_publish_operator.py
>>>>>>> becomes airflow/aws/operators/aws_**sns_publish_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> - *airflow/contrib/operators/dataproc_operator.py*
>>>>>>> becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> -
>>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
>>>> *airflow/*
>>>>>>> *gcp/operators/gcp_bigtable_operator.py*
>>>>>>> -
>>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>>>>>>> *operators/ssh_operator.py*
>>>>>>>
>>>>>>>
>>>>>>> Shall I proceed with updating the page ?
>>>>>>>
>>>>>>>
>>>>>>> J.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <ka...@gmail.com>
>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Jarek, adding my vote.
>>>>>>>>
>>>>>>>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <
>> ash@apache.org>
>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks Jarek, much clearer. I have voted there too.
>>>>>>>>>
>>>>>>>>> -ash
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko
>> <fokko@driesprong.frl
>>>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Thanks Jarek, the Wiki works much better for me. I've added my
>>> vote
>>>>>>>> there
>>>>>>>>>> as well.
>>>>>>>>>>
>>>>>>>>>> You've convinced me on the second case. I've changed my vote on
>>>> Case 2
>>>>>>>>> from
>>>>>>>>>> A to B :-)
>>>>>>>>>>
>>>>>>>>>> Maybe we should leave the vote open a bit longer since it
>>> involves
>>>>>>>> quite
>>>>>>>>>> some reading, and I would like to get some proper feedback and
>>>>>>>> opinions
>>>>>>>>> on
>>>>>>>>>> this. Plus I only see two votes right now. If we don't think
>> this
>>>>>>>>> through,
>>>>>>>>>> it might be that we're having this vote again in the near
>> future
>>>> :-)
>>>>>>>>>>
>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>
>>>>>>>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
>>>>>>>>> Jarek.Potiuk@polidea.com>:
>>>>>>>>>>
>>>>>>>>>>> I updated the AIP-21 proposal in the way that should answer
>> all
>>>> the
>>>>>>>>>>> concerns in this thread.
>>>>>>>>>>>
>>>>>>>>>>> NOTE! There is an action for all of those who are interested:
>>>> Please
>>>>>>>>>>> fill-in your voting in the voting table till Tuesday 30th of
>>> July
>>>> 6pm
>>>>>>>>> CEST
>>>>>>>>>>> (5pm BST, 9 am PST).
>>>>>>>>>>>
>>>>>>>>>>> I created a summary of the proposal
>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
>>>>>>>>>>>>
>>>>>>>>>>> in table form - which contains also examples for each case.
>>>>>>>>>>>
>>>>>>>>>>> I added a voting table
>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
>>>>>>>>>>>>
>>>>>>>>>>> where you should add your votes:
>>>>>>>>>>>
>>>>>>>>>>> I propose this voting mechanism:
>>>>>>>>>>>
>>>>>>>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5
>>> pm
>>>>>>>> BST,
>>>>>>>>> 9am
>>>>>>>>>>> PST) .
>>>>>>>>>>>
>>>>>>>>>>> After the choice, the final consistent set of choices will be
>>>>>>>> announced
>>>>>>>>>>> (taking into account majority of binding vote, also including
>>>>>>>> potential
>>>>>>>>>>> vetos and consistency between the choices. Non-binding votes
>>> will
>>>> be
>>>>>>>>> taken
>>>>>>>>>>> into account in case there is a draw. The final set of choices
>>>> will
>>>>>>>> be
>>>>>>>>>>> announced at devlist thread
>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>
>>>
>> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
>>>>>>>>>>>>
>>>>>>>>>>> after
>>>>>>>>>>> the voting completes.
>>>>>>>>>>>
>>>>>>>>>>> Unless there is a veto raised to the final proposal, the final
>>>>>>>> proposal
>>>>>>>>> is
>>>>>>>>>>> accepted by Lazy Consensus
>>>>>>>>>>> <https://community.apache.org/committers/lazyConsensus.html>
>>> on
>>>>>>>>> *Friday*
>>>>>>>>>>> 02
>>>>>>>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
>>>>>>>>>>>
>>>>>>>>>>> Please let me know if what I proposed can be further improved
>> to
>>>>>>>> avoid
>>>>>>>>>>> chaos.
>>>>>>>>>>>
>>>>>>>>>>> J.
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
>>>>>>>> Jarek.Potiuk@polidea.com
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Ok. I see that indeed voting could have been organised a bit
>>>> better
>>>>>>>> -
>>>>>>>>> dev
>>>>>>>>>>>> list does not seem to cope well with such a complex case :).
>>>>>>>>>>>>
>>>>>>>>>>>> As Kamil noticed - we already have a formal AIP-21
>>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>>>>>>>>>>>>
>>>>>>>>>>>> in the Wiki - which I should have mentioned before. I have
>> not
>>>> much
>>>>>>>>> time
>>>>>>>>>>>> today - but tomorrow I will try to summarise the options -
>>> based
>>>> on
>>>>>>>>> some
>>>>>>>>>>>> real code examples (to make it easier to digest). I think the
>>>> best
>>>>>>>>>>> approach
>>>>>>>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where
>>>> people
>>>>>>>>> will
>>>>>>>>>>>> be able to state their preferences for each case (by adding a
>>>> row to
>>>>>>>>> the
>>>>>>>>>>>> table). I think that might provide much better clarity and a
>>>> single
>>>>>>>>> place
>>>>>>>>>>>> which will show the current aggregated state of voting.
>>>>>>>>>>>>
>>>>>>>>>>>> However - as Ash also stated - some of the options are
>>>>>>>> inter-dependent
>>>>>>>>> so
>>>>>>>>>>>> at the end we will have to adjust the results in case they
>> are
>>>> not
>>>>>>>>>>>> consistent - and make final voting on aggregated set of
>> cases
>>> -
>>>>>>>> but I
>>>>>>>>>>>> think in this case following "lasy consensus" - i,e. if noone
>>>>>>>> objects
>>>>>>>>> to
>>>>>>>>>>>> final set of cases, we will proceed with it..
>>>>>>>>>>>>
>>>>>>>>>>>> Do you think that will work better ?
>>>>>>>>>>>>
>>>>>>>>>>>> J.
>>>>>>>>>>>>
>>>>>>>>>>>> Principal Software Engineer
>>>>>>>>>>>> Phone: +48660796129
>>>>>>>>>>>>
>>>>>>>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
>>>>>>>>>>>> kamil.bregula@polidea.com> napisał:
>>>>>>>>>>>>
>>>>>>>>>>>>> Document is also available as AIP on wiki:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I agree with all, this is a bit chaotic. For the sake of
>>>> clarity,
>>>>>>>> I
>>>>>>>>>>>>> would
>>>>>>>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP.
>> Create a
>>>>>>>>>>> structural
>>>>>>>>>>>>>> code improvement AIP and then and add a matrix of
>> users/cases
>>>>>>>> where
>>>>>>>>>>>>>> everybody can add his vote per case. This gives also the
>>>>>>>> opportunity
>>>>>>>>>>> for
>>>>>>>>>>>>>> changing your vote on a specific case. WDYT?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <
>>>> cixuuz@gmail.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I agree with Ash. It is clear to vote on each case rather
>>> than
>>>>>>>> all
>>>>>>>>>>>>>>> together.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
>>>>>>>> ash@apache.org>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> A number of proposals overlap or add on to each other,
>> so I
>>>>>>>> think
>>>>>>>>>>>>> (?) a
>>>>>>>>>>>>>>>> single vote makes sense.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what
>> we
>>>> are
>>>>>>>>>>>>> voting
>>>>>>>>>>>>>> on.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <
>>>> kaxilnaik@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases
>>>> together,
>>>>>>>>>>> how
>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>> separate vote emails for all the cases? Each email can
>>> have
>>>> the
>>>>>>>>>>>>>>>>> description
>>>>>>>>>>>>>>>>> copied over from the doc.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It would probably be a bit easier to track a decision in
>>> the
>>>>>>>>>>> future
>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> well.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
>>>>>>>> Jarek.Potiuk@polidea.com>
>>>>>>>>>>>>>>>>> @Driesprong,
>>>>>>>>>>>>>>>>> Fokko <fo...@driesprong.frl>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
>>>>>>>> kaxilnaik@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A
>> *
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Case #4* Separate namespace for resources :
>>>>>>>>>>>>>>>>>> *Solution A*
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Case #7* : *Solution A*
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
>>>>>>>>>>>>>>>>> <dp...@gmail.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I have tried to add some clarification to Jarek's
>>> summary,
>>>>>>>> but
>>>>>>>>>>> I
>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully
>>>> Jarek
>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> further
>>>>>>>>>>>>>>>>>>> clarify.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6
>> are
>>>> all
>>>>>>>>>>>>> either
>>>>>>>>>>>>>>>>>>> proposal
>>>>>>>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention
>>>> can be
>>>>>>>>>>>>>>>>> focused on
>>>>>>>>>>>>>>>>>>> cases 1,2,3 and 7.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> *Motion is to adopt the following:*
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
>>>>>>>>>>>>>>>>>>> All objects will be moved out of contrib folder and
>> into
>>>>>>>>>>>>> respective
>>>>>>>>>>>>>>>>>>> non-contrib folder.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from
>>>> modules
>>>>>>>>>>>>>>>>>>> Example:
>>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
>>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator
>>>>>>>>>>>>>>>>>>> Example:
>>>>>>>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
>>>>>>>>>>>>> airflow.hooks.foo.FooHook
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
>>>>>>>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
>>>>>>>>>>>>>>>>>>> Example case is
>>>>>>>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
>>>>>>>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume
>> it
>>> is
>>>>>>>> now
>>>>>>>>>>>>> named
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
>>>>>>>>>>>>>>>>>>> So what is proposed for this case?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Case 4 - Solution B - *no change*
>>>>>>>>>>>>>>>>>>> This proposal was about namespaces but the motion is
>> to
>>>>>>>> reject
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Case 5 - Solution B - *no change*
>>>>>>>>>>>>>>>>>>> The proposal was to remove class type suffix e.g.
>> remove
>>>> the
>>>>>>>>>>>>>>>>> Operator in
>>>>>>>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal
>>> --
>>>>>>>> i.e.
>>>>>>>>>>>>>>>>> motion is
>>>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>> change on this case, i.e. we resolve to keep
>> "Operator"
>>>> (and
>>>>>>>>>>>>>>>>> "Sensor")
>>>>>>>>>>>>>>>>>>> suffixes on class names
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Case 6 - *No change*
>>>>>>>>>>>>>>>>>>> This case concerns object naming inconsistencies, but
>>>> motion
>>>>>>>>>>> does
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> enact
>>>>>>>>>>>>>>>>>>> any specific proposal.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
>>>>>>>>>>> deprecation
>>>>>>>>>>>>>>>>> warnings
>>>>>>>>>>>>>>>>>>> (instead of zope)
>>>>>>>>>>>>>>>>>>> Example:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> # in old_package/old_mod.py
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> import warnings
>>>>>>>>>>>>>>>>>>> from new_package.new_mod import *
>>>>>>>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
>>>>>>>>>>>>> new_package.new_mod.
>>>>>>>>>>>>>>>>>>> Import
>>>>>>>>>>>>>>>>>>> of "
>>>>>>>>>>>>>>>>>>> "old_package.old_mod will become unsupported in
>> version
>>>> 2",
>>>>>>>>>>>>>>>>>>> DeprecationWarning, 2)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Reference implementation here:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>
>>>
>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I am very happy that this motion now gets rid of
>>> contrib.
>>>>>>>>>>> Thank
>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>> Sergio
>>>>>>>>>>>>>>>>>>> for turning the tide with your effective argumentation
>>> ;)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
>>>>>>>>>>>>> ash@apache.org>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Could someone summarise what the proposed name
>> changes
>>>> are,
>>>>>>>>>>>>> here,
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving
>>> partial
>>>>>>>>>>>>> examples
>>>>>>>>>>>>>>>>> is 1)
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting
>>> on,
>>>> and
>>>>>>>>>>> 2)
>>>>>>>>>>>>>>>>> using a
>>>>>>>>>>>>>>>>>>>> google doc isn't great for a longer term record.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Ash
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
>>>>>>>>>>>>>>>>> <Ja...@polidea.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0
>> :)
>>>> . I
>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three
>>> +1
>>>>>>>>>>>>>>>>> (binding)
>>>>>>>>>>>>>>>>>>> votes
>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to
>>> the
>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Here is the proposal (details here
>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>
>>>
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
>>>>>>>>>>>>>>>>>>>>> - *Case 2: A:
>>> Airflow.operators.foo_operator.FooOperator
>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
>>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>> become
>>> airflow.gcp.operators.bigtable.BigTableOperator*
>>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
>>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
>> suffixes
>>> on
>>>>>>>>>>>>> class
>>>>>>>>>>>>>>>>>>> names*
>>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
>>> case-by-case
>>>>>>>>>>>>> (and
>>>>>>>>>>>>>>>>> my team
>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
>>>>>>>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
>>>>>>>>>>>>>>>>> integration)*
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This is my binding (+1) vote.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
>>>>>>>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hey Felix,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one
>> as
>>>> well
>>>>>>>>>>> (I
>>>>>>>>>>>>>>>>> use IDE
>>>>>>>>>>>>>>>>>>>> quite
>>>>>>>>>>>>>>>>>>>>>> often ;) )
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
>>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I
>>>> wasn't
>>>>>>>>>>>>> able
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> respond
>>>>>>>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops
>>>> throughout
>>>>>>>>>>>>> Europe
>>>>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from
>>> contrib
>>>>>>>>>>> into
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> single
>>>>>>>>>>>>>>>>>>>>>>> package.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference
>>>> would be
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>> rid of
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or
>> not
>>>> is a
>>>>>>>>>>>>>>>>> tricky
>>>>>>>>>>>>>>>>>>>> question,
>>>>>>>>>>>>>>>>>>>>>>> I think this is already reflected in the document
>>>> since
>>>>>>>>>>>>>>>>> "maturity"
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the
>>>> operator
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> core
>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
>>>>>>>>>>> incubating
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> confuse
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> user even more. For people who aren't as known
>> with
>>>>>>>>>>>>> Airflow it
>>>>>>>>>>>>>>>>> isn't
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>> transparent which operator lives where, especially
>>> if
>>>> you
>>>>>>>>>>>>>>>>> don't use
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> IDE
>>>>>>>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is
>>> worth
>>>>>>>>>>>>> changing
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>>>>>>>>> API just for a different name.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with
>>> this.
>>>> It
>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>>>>> easier
>>>>>>>>>>>>>>>>>>>>>> as we will not have to define other rules as those
>>>> that we
>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>> have.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's
>>> say a
>>>>>>>>>>> +0)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search
>>> for
>>>>>>>>>>>>>>>>> BashOperator.
>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from
>>> your
>>>>>>>>>>> mail
>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>> state: -
>>>>>>>>>>>>>>>>>>>>>>> *Case 2: B:
>>> Airflow.operators.foo_operator.FooOperator
>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>> become
>>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B
>>>> solution is
>>>>>>>>>>>>>>>>> keeping it
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
>>>>>>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would
>>> stay
>>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a
>>>> little
>>>>>>>>>>>>>>>>> confusion
>>>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>> This case is not for changing BashOperator into
>> Bash
>>>> but
>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> changing
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> *module* name not the *class* name. So
>>>>>>>>>>>>>>>>> bash_operator.BashOperator ->
>>>>>>>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for
>>>>>>>>>>>>> BashOperator
>>>>>>>>>>>>>>>>> in this
>>>>>>>>>>>>>>>>>>>> case.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when
>> it
>>>> gets
>>>>>>>>>>>>> its
>>>>>>>>>>>>>>>>> own
>>>>>>>>>>>>>>>>>>>> package
>>>>>>>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are
>>>> companies
>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own
>>>> package
>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving
>> this
>>>> is a
>>>>>>>>>>>>> bit
>>>>>>>>>>>>>>>>>>> painful
>>>>>>>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move
>>>>>>>>>>>>> everything
>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>>>> introduces noise into version control etc.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's
>> not
>>> as
>>>>>>>>>>>>>>>>> difficult as
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> seems to group operators in packages and it has
>>>> numerous
>>>>>>>>>>>>>>>>> advantages.
>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in
>> the
>>>>>>>>>>>>>>>>> __init__.py of
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git
>>> will
>>>>>>>>>>>>> actually
>>>>>>>>>>>>>>>>>>> realise
>>>>>>>>>>>>>>>>>>>>>> that the file has been moved and you can follow
>> that.
>>>>>>>>>>> Maybe
>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>> super
>>>>>>>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's
>> not a
>>>>>>>>>>> rocket
>>>>>>>>>>>>>>>>> science
>>>>>>>>>>>>>>>>>>>> either.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Apart from all, great work!
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix
>> Uellendall
>>>>>>>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
>>>>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change
>>> of
>>>>>>>>>>> these
>>>>>>>>>>>>>>>>> cases.
>>>>>>>>>>>>>>>>>>> So I
>>>>>>>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>>>>>>> with +1 (non-binding).
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and
>>> your
>>>>>>>>>>>>> intense
>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>>>>>>> towards
>>>>>>>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
>>>>>>>>>>> creating
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>> document. I
>>>>>>>>>>>>>>>>>>>>>>>> like how the code is getting more and more
>>> consistent
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> clean :)
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>> Felix
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> -------- Original Message --------
>>>>>>>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in
>>>> import
>>>>>>>>>>>>> paths.
>>>>>>>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>> discussed in
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>
>>>
>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July
>> 30th
>>>> 6pm
>>>>>>>>>>>>> CEST),
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>> least
>>>>>>>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document
>> from
>>>>>>>>>>> Kamil
>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>
>>>
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The proposed solution is:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that
>> are
>>>>>>>>>>> "well"
>>>>>>>>>>>>>>>>> tested
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
>>>>>>>>>>>>>>>>>>>>>>>>> - *Case 2: B:
>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator
>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
>>>>>>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>>>> become
>>>> airflow.gcp.operators.bigtable.BigTableOperator*
>>>>>>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
>>>>>>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
>>>> suffixes
>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> class
>>>>>>>>>>>>>>>>>>> names*
>>>>>>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
>>>> case-by-case
>>>>>>>>>>>>> (and
>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>> team
>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
>>>>>>>>>>> Software
>>>>>>>>>>>>>>>>> Engineer
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
>>>>>>>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
>>>>>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
>>>> Software
>>>>>>>>>>>>>>>>> Engineer
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
>>> Software
>>>>>>>>>>>>>> Engineer
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> *Kaxil Naik*
>>>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>>>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
>>> Apache
>>>>>>>>>>> Spark
>>>>>>>>>>>>> &
>>>>>>>>>>>>>>>>> Neo4j
>>>>>>>>>>>>>>>>>> Developer
>>>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> *Kaxil Naik*
>>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
>>> Apache
>>>>>>>> Spark
>>>>>>>>>>> &
>>>>>>>>>>>>>>>>> Neo4j
>>>>>>>>>>>>>>>>> Developer
>>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
>>> Engineer
>>>>>>>>>>>
>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Kaxil Naik*
>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
>> Spark &
>>>> Neo4j
>>>>>>>> Developer
>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Jarek Potiuk
>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>>
>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jarek Potiuk
>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>
>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>
>>>>
>>>
>>
Re: [VOTE] Changes in import paths
Posted by Tomasz Urbaszek <to...@polidea.com>.
Maybe we can put all those AtoB operators under one name like “transfer”,
then it would be easier to look for such operator?
Best,
Tomek
On Tue, 30 Jul 2019 at 21:39, Chen Tong <ci...@gmail.com> wrote:
> Daniel mentioned a good point. Such composed operator may also involves
> both cloud and non-cloud provider saying FTPtoS3Operator. Should it in AWS
> folder?
>
> Also, saying in the future, another cloud provider is growing large enough.
> Will we rename all plugins related to this provider? What's the criteria we
> say it's big enough? And option D gives an impression like these tools may
> be maintained and supported by the Cloud provider. I am not sure if it will
> be a truth or not.
>
> Best,
>
>
> On Tue, Jul 30, 2019 at 11:18 AM Daniel Standish <dp...@gmail.com>
> wrote:
>
> > One wrinkle with case 3+4 option D is inter-provider operators. Mainly
> > it's storage I think e.g. XToS3Operator or XToGCSOperator where the X is
> a
> > service some different provider.
> >
> > Maybe the rule should be to locate the operator according to the first
> > provider referenced. So e.g. s3_to_gcs_transfer_operator.py would go to
> > aws.
> >
> >
> > On Tue, Jul 30, 2019 at 6:21 AM Kamil Breguła <kamil.bregula@polidea.com
> >
> > wrote:
> >
> > > Yes. All changes will be backwards compatible. In the case of using
> > > the old path, a message containing a proposal for change will be
> > > reported to the user.
> > >
> > > I prepared an example of how to change the name of a class in a case
> > > with the use of a native solution.
> > > Source code:
> > >
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename
> > > Preview from IDE: https://imgur.com/a/45NxS5W
> > >
> > > On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > > >
> > > > Just cos I'm not sure it's _explicitly_ stated, but all of the moves
> > > will have a deprecation of the old name right?
> > > >
> > > > 3+4 case D gets my vote too.
> > > >
> > > > -a
> > > >
> > > > > On 30 Jul 2019, at 09:58, Jarek Potiuk <Ja...@polidea.com>
> > > wrote:
> > > > >
> > > > > I went ahead and updated the page (just to speed it up) as I think
> it
> > > > > really makes sense to join those two cases (and I do not see any
> > > drawbacks
> > > > > - I think the options we have cover all possible approaches) and we
> > can
> > > > > always go back if we need to.
> > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > > > >
> > > > > My vote is *D*.
> > > > >
> > > > > - airflow/contrib/operators/gcp_bigtable_operator.py →
> > airflow/gcp
> > > > > /operators/bigtable_operator.py
> > > > > - airflow/contrib/operators/ssh_operator.py ->
> > > > > airflow/operators/ssh_operator.py
> > > > >
> > > > > I updated the page with my vote.
> > > > > I propose that we vote till Friday on that one (all the rest is
> > already
> > > > > agreed).
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > > On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <
> > Jarek.Potiuk@polidea.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> I think almost everyone voted and we have almost perfect
> consensus.
> > > We all
> > > > >> agree amongst other on moving all operators out of contrib
> (Great!).
> > > > >>
> > > > >> The only doubts are for *Case 3* (Cloud provider prefix) and *Case
> > 4*
> > > > >> (Using Namespaces).
> > > > >> I think there was actually an overlap between those two. Also
> Ash's
> > > > >> comment/veto on that is quite clear.
> > > > >> So I have final (I hope!) proposal to join those two cases and
> have
> > > one
> > > > >> voting (till Friday) only on that.
> > > > >>
> > > > >> I will update the doc if you all agree with me that makes more
> sense
> > > as
> > > > >> single case to vote on:
> > > > >>
> > > > >> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
> > > > >>
> > > > >> *Option A*. Keep operators/sensors/hooks in
> > airflow/operators(sensors,
> > > > >> hooks) and keep/add prefixes in file names.
> > > > >>
> > > > >> -
> > > > >> *airflow/contrib/operators/sns_publish_operator.py becomes
> > > > >> airflow/operators/**aws_sns_publish_operator.py*
> > > > >>
> > > > >>
> > > > >> - *airflow/contrib/operators/dataproc_operator.py*
> > > > >> becomes *airflow/operators/gcp_dataproc_operator.py*
> > > > >>
> > > > >>
> > > > >> -
> > > > >> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/*
> > > > >> *hooks/gcp_bigtable_hook.py*
> > > > >> -
> > > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > > > >> *operators/ssh_operator.py*
> > > > >>
> > > > >>
> > > > >>
> > > > >> *Option B*. Group operators/sensors/hooks in
> > > airflow/operators(sensors,
> > > > >> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
> > > > >> airflow/operators(sensors/hooks), Drop the prefix.
> > > > >>
> > > > >> -
> > > > >> *airflow/contrib/operators/sns_publish_operator.py
> > > > >> becomes airflow/operators/**aws/sns_publish_operator.py*
> > > > >>
> > > > >>
> > > > >> - *airflow/contrib/operators/dataproc_operator.py*
> > > > >> becomes *airflow/operators/gcp/dataproc_operator.py*
> > > > >>
> > > > >>
> > > > >> -
> > > > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> > > *airflow/*
> > > > >> *operators/gcp/bigtable_operator.py*
> > > > >> -
> > > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > > > >> *operators/ssh_operator.py*
> > > > >>
> > > > >>
> > > > >> *Option C*. Group operators/sensors/hooks in
> > > airflow/operators(sensors,
> > > > >> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
> > > > >> airflow/operators(sensors/hooks), Keep the prefix.
> > > > >>
> > > > >> -
> > > > >> *airflow/contrib/operators/sns_publish_operator.py
> > > > >> becomes airflow/operators/**aws/aws_sns_publish_operator.py*
> > > > >>
> > > > >>
> > > > >> - *airflow/contrib/operators/dataproc_operator.py*
> > > > >> becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
> > > > >>
> > > > >>
> > > > >> -
> > > > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> > > *airflow/*
> > > > >> *operators/gcp/gcp_bigtable_operator.py*
> > > > >> -
> > > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > > > >> *operators/ssh_operator.py*
> > > > >>
> > > > >>
> > > > >> *Option D.* Group operators/sensors/hooks in
> > > *airflow/<PROVIDER>*/operators(sensors,
> > > > >> hooks). Non-cloud provider ones are moved to
> > > > >> airflow/operators(sensors/hooks). Drop the prefix.
> > > > >>
> > > > >> -
> > > > >> *airflow/contrib/operators/sns_publish_operator.py
> > > > >> becomes airflow/aws/operators/**sns_publish_operator.py*
> > > > >>
> > > > >>
> > > > >> - *airflow/contrib/operators/dataproc_operator.py*
> > > > >> becomes *airflow/gcp/operators/dataproc_operator.py*
> > > > >>
> > > > >>
> > > > >> -
> > > > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> > > *airflow/*
> > > > >> *gcp/operators/bigtable_operator.py*
> > > > >> -
> > > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > > > >> *operators/ssh_operator.py*
> > > > >>
> > > > >>
> > > > >> *Option E.* Group operators/sensors/hooks in
> > > *airflow/<PROVIDER>*/operators(sensors,
> > > > >> hooks). Non-cloud provider ones are moved to
> > > > >> airflow/operators(sensors/hooks).* Keep the prefix*.
> > > > >>
> > > > >> -
> > > > >> *airflow/contrib/operators/sns_publish_operator.py
> > > > >> becomes airflow/aws/operators/aws_**sns_publish_operator.py*
> > > > >>
> > > > >>
> > > > >> - *airflow/contrib/operators/dataproc_operator.py*
> > > > >> becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
> > > > >>
> > > > >>
> > > > >> -
> > > > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> > > *airflow/*
> > > > >> *gcp/operators/gcp_bigtable_operator.py*
> > > > >> -
> > > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > > > >> *operators/ssh_operator.py*
> > > > >>
> > > > >>
> > > > >> Shall I proceed with updating the page ?
> > > > >>
> > > > >>
> > > > >> J.
> > > > >>
> > > > >>
> > > > >> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <ka...@gmail.com>
> > > wrote:
> > > > >>
> > > > >>> Thanks Jarek, adding my vote.
> > > > >>>
> > > > >>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <
> ash@apache.org>
> > > wrote:
> > > > >>>
> > > > >>>> Thanks Jarek, much clearer. I have voted there too.
> > > > >>>>
> > > > >>>> -ash
> > > > >>>>
> > > > >>>>
> > > > >>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko
> <fokko@driesprong.frl
> > >
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>> Thanks Jarek, the Wiki works much better for me. I've added my
> > vote
> > > > >>> there
> > > > >>>>> as well.
> > > > >>>>>
> > > > >>>>> You've convinced me on the second case. I've changed my vote on
> > > Case 2
> > > > >>>> from
> > > > >>>>> A to B :-)
> > > > >>>>>
> > > > >>>>> Maybe we should leave the vote open a bit longer since it
> > involves
> > > > >>> quite
> > > > >>>>> some reading, and I would like to get some proper feedback and
> > > > >>> opinions
> > > > >>>> on
> > > > >>>>> this. Plus I only see two votes right now. If we don't think
> this
> > > > >>>> through,
> > > > >>>>> it might be that we're having this vote again in the near
> future
> > > :-)
> > > > >>>>>
> > > > >>>>> Cheers, Fokko
> > > > >>>>>
> > > > >>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
> > > > >>>> Jarek.Potiuk@polidea.com>:
> > > > >>>>>
> > > > >>>>>> I updated the AIP-21 proposal in the way that should answer
> all
> > > the
> > > > >>>>>> concerns in this thread.
> > > > >>>>>>
> > > > >>>>>> NOTE! There is an action for all of those who are interested:
> > > Please
> > > > >>>>>> fill-in your voting in the voting table till Tuesday 30th of
> > July
> > > 6pm
> > > > >>>> CEST
> > > > >>>>>> (5pm BST, 9 am PST).
> > > > >>>>>>
> > > > >>>>>> I created a summary of the proposal
> > > > >>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > > > >>>>>>>
> > > > >>>>>> in table form - which contains also examples for each case.
> > > > >>>>>>
> > > > >>>>>> I added a voting table
> > > > >>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> > > > >>>>>>>
> > > > >>>>>> where you should add your votes:
> > > > >>>>>>
> > > > >>>>>> I propose this voting mechanism:
> > > > >>>>>>
> > > > >>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5
> > pm
> > > > >>> BST,
> > > > >>>> 9am
> > > > >>>>>> PST) .
> > > > >>>>>>
> > > > >>>>>> After the choice, the final consistent set of choices will be
> > > > >>> announced
> > > > >>>>>> (taking into account majority of binding vote, also including
> > > > >>> potential
> > > > >>>>>> vetos and consistency between the choices. Non-binding votes
> > will
> > > be
> > > > >>>> taken
> > > > >>>>>> into account in case there is a draw. The final set of choices
> > > will
> > > > >>> be
> > > > >>>>>> announced at devlist thread
> > > > >>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > >
> >
> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> > > > >>>>>>>
> > > > >>>>>> after
> > > > >>>>>> the voting completes.
> > > > >>>>>>
> > > > >>>>>> Unless there is a veto raised to the final proposal, the final
> > > > >>> proposal
> > > > >>>> is
> > > > >>>>>> accepted by Lazy Consensus
> > > > >>>>>> <https://community.apache.org/committers/lazyConsensus.html>
> > on
> > > > >>>> *Friday*
> > > > >>>>>> 02
> > > > >>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
> > > > >>>>>>
> > > > >>>>>> Please let me know if what I proposed can be further improved
> to
> > > > >>> avoid
> > > > >>>>>> chaos.
> > > > >>>>>>
> > > > >>>>>> J.
> > > > >>>>>>
> > > > >>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
> > > > >>> Jarek.Potiuk@polidea.com
> > > > >>>>>
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Ok. I see that indeed voting could have been organised a bit
> > > better
> > > > >>> -
> > > > >>>> dev
> > > > >>>>>>> list does not seem to cope well with such a complex case :).
> > > > >>>>>>>
> > > > >>>>>>> As Kamil noticed - we already have a formal AIP-21
> > > > >>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > > > >>>>>>>
> > > > >>>>>>> in the Wiki - which I should have mentioned before. I have
> not
> > > much
> > > > >>>> time
> > > > >>>>>>> today - but tomorrow I will try to summarise the options -
> > based
> > > on
> > > > >>>> some
> > > > >>>>>>> real code examples (to make it easier to digest). I think the
> > > best
> > > > >>>>>> approach
> > > > >>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where
> > > people
> > > > >>>> will
> > > > >>>>>>> be able to state their preferences for each case (by adding a
> > > row to
> > > > >>>> the
> > > > >>>>>>> table). I think that might provide much better clarity and a
> > > single
> > > > >>>> place
> > > > >>>>>>> which will show the current aggregated state of voting.
> > > > >>>>>>>
> > > > >>>>>>> However - as Ash also stated - some of the options are
> > > > >>> inter-dependent
> > > > >>>> so
> > > > >>>>>>> at the end we will have to adjust the results in case they
> are
> > > not
> > > > >>>>>>> consistent - and make final voting on aggregated set of
> cases
> > -
> > > > >>> but I
> > > > >>>>>>> think in this case following "lasy consensus" - i,e. if noone
> > > > >>> objects
> > > > >>>> to
> > > > >>>>>>> final set of cases, we will proceed with it..
> > > > >>>>>>>
> > > > >>>>>>> Do you think that will work better ?
> > > > >>>>>>>
> > > > >>>>>>> J.
> > > > >>>>>>>
> > > > >>>>>>> Principal Software Engineer
> > > > >>>>>>> Phone: +48660796129
> > > > >>>>>>>
> > > > >>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> > > > >>>>>>> kamil.bregula@polidea.com> napisał:
> > > > >>>>>>>
> > > > >>>>>>>> Document is also available as AIP on wiki:
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > > > >>>>>>>>
> > > > >>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
> > > > >>> <fokko@driesprong.frl
> > > > >>>>>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> I agree with all, this is a bit chaotic. For the sake of
> > > clarity,
> > > > >>> I
> > > > >>>>>>>> would
> > > > >>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP.
> Create a
> > > > >>>>>> structural
> > > > >>>>>>>>> code improvement AIP and then and add a matrix of
> users/cases
> > > > >>> where
> > > > >>>>>>>>> everybody can add his vote per case. This gives also the
> > > > >>> opportunity
> > > > >>>>>> for
> > > > >>>>>>>>> changing your vote on a specific case. WDYT?
> > > > >>>>>>>>>
> > > > >>>>>>>>> Cheers, Fokko
> > > > >>>>>>>>>
> > > > >>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <
> > > cixuuz@gmail.com>:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> I agree with Ash. It is clear to vote on each case rather
> > than
> > > > >>> all
> > > > >>>>>>>>>> together.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
> > > > >>> ash@apache.org>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> A number of proposals overlap or add on to each other,
> so I
> > > > >>> think
> > > > >>>>>>>> (?) a
> > > > >>>>>>>>>>> single vote makes sense.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what
> we
> > > are
> > > > >>>>>>>> voting
> > > > >>>>>>>>> on.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <
> > > kaxilnaik@gmail.com>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases
> > > together,
> > > > >>>>>> how
> > > > >>>>>>>>>>>> about
> > > > >>>>>>>>>>>> separate vote emails for all the cases? Each email can
> > have
> > > the
> > > > >>>>>>>>>>>> description
> > > > >>>>>>>>>>>> copied over from the doc.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> It would probably be a bit easier to track a decision in
> > the
> > > > >>>>>> future
> > > > >>>>>>>> as
> > > > >>>>>>>>>>>> well.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
> > > > >>> Jarek.Potiuk@polidea.com>
> > > > >>>>>>>>>>>> @Driesprong,
> > > > >>>>>>>>>>>> Fokko <fo...@driesprong.frl>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
> > > > >>> kaxilnaik@gmail.com>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A
> *
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> *Case #4* Separate namespace for resources :
> > > > >>>>>>>>>>>>> *Solution A*
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> *Case #7* : *Solution A*
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> > > > >>>>>>>>>>>> <dp...@gmail.com>
> > > > >>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> I have tried to add some clarification to Jarek's
> > summary,
> > > > >>> but
> > > > >>>>>> I
> > > > >>>>>>>> am
> > > > >>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully
> > > Jarek
> > > > >>>>>> can
> > > > >>>>>>>>>>>> further
> > > > >>>>>>>>>>>>>> clarify.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6
> are
> > > all
> > > > >>>>>>>> either
> > > > >>>>>>>>>>>>>> proposal
> > > > >>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention
> > > can be
> > > > >>>>>>>>>>>> focused on
> > > > >>>>>>>>>>>>>> cases 1,2,3 and 7.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> *Motion is to adopt the following:*
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
> > > > >>>>>>>>>>>>>> All objects will be moved out of contrib folder and
> into
> > > > >>>>>>>> respective
> > > > >>>>>>>>>>>>>> non-contrib folder.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from
> > > modules
> > > > >>>>>>>>>>>>>> Example:
> > > > >>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
> > > > >>>>>>>>>>>>>> airflow.operators.foo.FooOperator
> > > > >>>>>>>>>>>>>> Example:
> > > > >>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
> > > > >>>>>>>> airflow.hooks.foo.FooHook
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
> > > > >>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
> > > > >>>>>>>>>>>>>> Example case is
> > > > >>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
> > > > >>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume
> it
> > is
> > > > >>> now
> > > > >>>>>>>> named
> > > > >>>>>>>>>>>> like
> > > > >>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
> > > > >>>>>>>>>>>>>> So what is proposed for this case?
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Case 4 - Solution B - *no change*
> > > > >>>>>>>>>>>>>> This proposal was about namespaces but the motion is
> to
> > > > >>> reject
> > > > >>>>>>>> the
> > > > >>>>>>>>>>>>>> proposal.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Case 5 - Solution B - *no change*
> > > > >>>>>>>>>>>>>> The proposal was to remove class type suffix e.g.
> remove
> > > the
> > > > >>>>>>>>>>>> Operator in
> > > > >>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal
> > --
> > > > >>> i.e.
> > > > >>>>>>>>>>>> motion is
> > > > >>>>>>>>>>>>>> no
> > > > >>>>>>>>>>>>>> change on this case, i.e. we resolve to keep
> "Operator"
> > > (and
> > > > >>>>>>>>>>>> "Sensor")
> > > > >>>>>>>>>>>>>> suffixes on class names
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Case 6 - *No change*
> > > > >>>>>>>>>>>>>> This case concerns object naming inconsistencies, but
> > > motion
> > > > >>>>>> does
> > > > >>>>>>>>>>>> not
> > > > >>>>>>>>>>>>>> enact
> > > > >>>>>>>>>>>>>> any specific proposal.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
> > > > >>>>>> deprecation
> > > > >>>>>>>>>>>> warnings
> > > > >>>>>>>>>>>>>> (instead of zope)
> > > > >>>>>>>>>>>>>> Example:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> # in old_package/old_mod.py
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> import warnings
> > > > >>>>>>>>>>>>>> from new_package.new_mod import *
> > > > >>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
> > > > >>>>>>>> new_package.new_mod.
> > > > >>>>>>>>>>>>>> Import
> > > > >>>>>>>>>>>>>> of "
> > > > >>>>>>>>>>>>>> "old_package.old_mod will become unsupported in
> version
> > > 2",
> > > > >>>>>>>>>>>>>> DeprecationWarning, 2)
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Reference implementation here:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > >
> >
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> I am very happy that this motion now gets rid of
> > contrib.
> > > > >>>>>> Thank
> > > > >>>>>>>> you
> > > > >>>>>>>>>>>>>> Sergio
> > > > >>>>>>>>>>>>>> for turning the tide with your effective argumentation
> > ;)
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
> > > > >>>>>>>> ash@apache.org>
> > > > >>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Could someone summarise what the proposed name
> changes
> > > are,
> > > > >>>>>>>> here,
> > > > >>>>>>>>>>>> in
> > > > >>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving
> > partial
> > > > >>>>>>>> examples
> > > > >>>>>>>>>>>> is 1)
> > > > >>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting
> > on,
> > > and
> > > > >>>>>> 2)
> > > > >>>>>>>>>>>> using a
> > > > >>>>>>>>>>>>>>> google doc isn't great for a longer term record.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>> Ash
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
> > > > >>>>>>>>>>>> <Ja...@polidea.com>
> > > > >>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0
> :)
> > > . I
> > > > >>>>>>>> would
> > > > >>>>>>>>>>>> like
> > > > >>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three
> > +1
> > > > >>>>>>>>>>>> (binding)
> > > > >>>>>>>>>>>>>> votes
> > > > >>>>>>>>>>>>>>> are
> > > > >>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to
> > the
> > > > >>>>>>>>>>>> previous
> > > > >>>>>>>>>>>>>>> proposal.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Here is the proposal (details here
> > > > >>>>>>>>>>>>>>>> <
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > >
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> )
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
> > > > >>>>>>>>>>>>>>>> - *Case 2: A:
> > Airflow.operators.foo_operator.FooOperator
> > > > >>>>>>>> could
> > > > >>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > > > >>>>>>>>>>>>>>>> - *Case 3: C:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > > > >>>>>>>>>>>>>> could
> > > > >>>>>>>>>>>>>>>> become
> > airflow.gcp.operators.bigtable.BigTableOperator*
> > > > >>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > > > >>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
> suffixes
> > on
> > > > >>>>>>>> class
> > > > >>>>>>>>>>>>>> names*
> > > > >>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> > case-by-case
> > > > >>>>>>>> (and
> > > > >>>>>>>>>>>> my team
> > > > >>>>>>>>>>>>>>> can
> > > > >>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > > > >>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
> > > > >>>>>>>>>>>> integration)*
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> This is my binding (+1) vote.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> > > > >>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
> > > > >>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Hey Felix,
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one
> as
> > > well
> > > > >>>>>> (I
> > > > >>>>>>>>>>>> use IDE
> > > > >>>>>>>>>>>>>>> quite
> > > > >>>>>>>>>>>>>>>>> often ;) )
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> J.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> > > > >>>>>>>>>>>>>> <fokko@driesprong.frl
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I
> > > wasn't
> > > > >>>>>>>> able
> > > > >>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>> respond
> > > > >>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops
> > > throughout
> > > > >>>>>>>> Europe
> > > > >>>>>>>>>>>> :-)
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from
> > contrib
> > > > >>>>>> into
> > > > >>>>>>>> a
> > > > >>>>>>>>>>>> single
> > > > >>>>>>>>>>>>>>>>>> package.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference
> > > would be
> > > > >>>>>>>> to
> > > > >>>>>>>>>>>> get
> > > > >>>>>>>>>>>>>> rid of
> > > > >>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or
> not
> > > is a
> > > > >>>>>>>>>>>> tricky
> > > > >>>>>>>>>>>>>>> question,
> > > > >>>>>>>>>>>>>>>>>> I think this is already reflected in the document
> > > since
> > > > >>>>>>>>>>>> "maturity"
> > > > >>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>> in
> > > > >>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the
> > > operator
> > > > >>>>>>>> to
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>> core
> > > > >>>>>>>>>>>>>>> or
> > > > >>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
> > > > >>>>>> incubating
> > > > >>>>>>>>>>>> will
> > > > >>>>>>>>>>>>>> confuse
> > > > >>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> user even more. For people who aren't as known
> with
> > > > >>>>>>>> Airflow it
> > > > >>>>>>>>>>>> isn't
> > > > >>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>> transparent which operator lives where, especially
> > if
> > > you
> > > > >>>>>>>>>>>> don't use
> > > > >>>>>>>>>>>>>> an
> > > > >>>>>>>>>>>>>>> IDE
> > > > >>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is
> > worth
> > > > >>>>>>>> changing
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>> public
> > > > >>>>>>>>>>>>>>>>>> API just for a different name.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with
> > this.
> > > It
> > > > >>>>>>>> will
> > > > >>>>>>>>>>>> make
> > > > >>>>>>>>>>>>>>> easier
> > > > >>>>>>>>>>>>>>>>> as we will not have to define other rules as those
> > > that we
> > > > >>>>>>>>>>>> already
> > > > >>>>>>>>>>>>>> have.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's
> > say a
> > > > >>>>>> +0)
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search
> > for
> > > > >>>>>>>>>>>> BashOperator.
> > > > >>>>>>>>>>>>>>> After
> > > > >>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from
> > your
> > > > >>>>>> mail
> > > > >>>>>>>> you
> > > > >>>>>>>>>>>>>> state: -
> > > > >>>>>>>>>>>>>>>>>> *Case 2: B:
> > Airflow.operators.foo_operator.FooOperator
> > > > >>>>>>>> could
> > > > >>>>>>>>>>>> become
> > > > >>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B
> > > solution is
> > > > >>>>>>>>>>>> keeping it
> > > > >>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
> > > > >>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would
> > stay
> > > > >>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a
> > > little
> > > > >>>>>>>>>>>> confusion
> > > > >>>>>>>>>>>>>>> here.
> > > > >>>>>>>>>>>>>>>>> This case is not for changing BashOperator into
> Bash
> > > but
> > > > >>>>>> for
> > > > >>>>>>>>>>>> changing
> > > > >>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>> *module* name not the *class* name. So
> > > > >>>>>>>>>>>> bash_operator.BashOperator ->
> > > > >>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for
> > > > >>>>>>>> BashOperator
> > > > >>>>>>>>>>>> in this
> > > > >>>>>>>>>>>>>>> case.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when
> it
> > > gets
> > > > >>>>>>>> its
> > > > >>>>>>>>>>>> own
> > > > >>>>>>>>>>>>>>> package
> > > > >>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are
> > > companies
> > > > >>>>>>>> like
> > > > >>>>>>>>>>>>>>> Databricks
> > > > >>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own
> > > package
> > > > >>>>>>>>>>>> because
> > > > >>>>>>>>>>>>>> their
> > > > >>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving
> this
> > > is a
> > > > >>>>>>>> bit
> > > > >>>>>>>>>>>>>> painful
> > > > >>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move
> > > > >>>>>>>> everything
> > > > >>>>>>>>>>>> which
> > > > >>>>>>>>>>>>>>>>>> introduces noise into version control etc.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's
> not
> > as
> > > > >>>>>>>>>>>> difficult as
> > > > >>>>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>>>>> seems to group operators in packages and it has
> > > numerous
> > > > >>>>>>>>>>>> advantages.
> > > > >>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in
> the
> > > > >>>>>>>>>>>> __init__.py of
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git
> > will
> > > > >>>>>>>> actually
> > > > >>>>>>>>>>>>>> realise
> > > > >>>>>>>>>>>>>>>>> that the file has been moved and you can follow
> that.
> > > > >>>>>> Maybe
> > > > >>>>>>>> not
> > > > >>>>>>>>>>>> as
> > > > >>>>>>>>>>>>>> super
> > > > >>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's
> not a
> > > > >>>>>> rocket
> > > > >>>>>>>>>>>> science
> > > > >>>>>>>>>>>>>>> either.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Yes.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> ---
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Apart from all, great work!
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Cheers, Fokko
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix
> Uellendall
> > > > >>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
> > > > >>>>>>>>>>>>>>>>>>> :
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change
> > of
> > > > >>>>>> these
> > > > >>>>>>>>>>>> cases.
> > > > >>>>>>>>>>>>>> So I
> > > > >>>>>>>>>>>>>>>>>> go
> > > > >>>>>>>>>>>>>>>>>>> with +1 (non-binding).
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and
> > your
> > > > >>>>>>>> intense
> > > > >>>>>>>>>>>> work
> > > > >>>>>>>>>>>>>>>>>> towards
> > > > >>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
> > > > >>>>>> creating
> > > > >>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>>> document. I
> > > > >>>>>>>>>>>>>>>>>>> like how the code is getting more and more
> > consistent
> > > > >>>>>> and
> > > > >>>>>>>>>>>> clean :)
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Kind regards,
> > > > >>>>>>>>>>>>>>>>>>> Felix
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> -------- Original Message --------
> > > > >>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in
> > > import
> > > > >>>>>>>> paths.
> > > > >>>>>>>>>>>> It's
> > > > >>>>>>>>>>>>>>> been
> > > > >>>>>>>>>>>>>>>>>>>> discussed in
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > >
> >
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July
> 30th
> > > 6pm
> > > > >>>>>>>> CEST),
> > > > >>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>> at
> > > > >>>>>>>>>>>>>>>>>> least
> > > > >>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document
> from
> > > > >>>>>> Kamil
> > > > >>>>>>>>>>>>>>>>>>>> <
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > >
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> The proposed solution is:
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that
> are
> > > > >>>>>> "well"
> > > > >>>>>>>>>>>> tested
> > > > >>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
> > > > >>>>>>>>>>>>>>>>>>>> - *Case 2: B:
> > > > >>>>>> Airflow.operators.foo_operator.FooOperator
> > > > >>>>>>>>>>>> could
> > > > >>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > > > >>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > > > >>>>>>>>>>>>>>> could
> > > > >>>>>>>>>>>>>>>>>>>> become
> > > airflow.gcp.operators.bigtable.BigTableOperator*
> > > > >>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > > > >>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
> > > suffixes
> > > > >>>>>> on
> > > > >>>>>>>>>>>> class
> > > > >>>>>>>>>>>>>> names*
> > > > >>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> > > case-by-case
> > > > >>>>>>>> (and
> > > > >>>>>>>>>>>> my
> > > > >>>>>>>>>>>>>> team
> > > > >>>>>>>>>>>>>>>>>> can
> > > > >>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Best regards,
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> J.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> --
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> > > > >>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> > > > >>>>>> Software
> > > > >>>>>>>>>>>> Engineer
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
> > > > >>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
> > > > >>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> --
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Jarek Potiuk
> > > > >>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> > > Software
> > > > >>>>>>>>>>>> Engineer
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > > > >>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> --
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Jarek Potiuk
> > > > >>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> > Software
> > > > >>>>>>>>> Engineer
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > > > >>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> --
> > > > >>>>>>>>>>>>> *Kaxil Naik*
> > > > >>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > > > >>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
> > Apache
> > > > >>>>>> Spark
> > > > >>>>>>>> &
> > > > >>>>>>>>>>>> Neo4j
> > > > >>>>>>>>>>>>> Developer
> > > > >>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> --
> > > > >>>>>>>>>>>> *Kaxil Naik*
> > > > >>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > > > >>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
> > Apache
> > > > >>> Spark
> > > > >>>>>> &
> > > > >>>>>>>>>>>> Neo4j
> > > > >>>>>>>>>>>> Developer
> > > > >>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>> --
> > > > >>>>>>
> > > > >>>>>> Jarek Potiuk
> > > > >>>>>> Polidea <https://www.polidea.com/> | Principal Software
> > Engineer
> > > > >>>>>>
> > > > >>>>>> M: +48 660 796 129 <+48660796129>
> > > > >>>>>> [image: Polidea] <https://www.polidea.com/>
> > > > >>>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>> --
> > > > >>> *Kaxil Naik*
> > > > >>> *Big Data Consultant | DevOps Data Engineer*
> > > > >>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> Spark &
> > > Neo4j
> > > > >>> Developer
> > > > >>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > > >>>
> > > > >>
> > > > >>
> > > > >> --
> > > > >>
> > > > >> Jarek Potiuk
> > > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > >>
> > > > >> M: +48 660 796 129 <+48660796129>
> > > > >> [image: Polidea] <https://www.polidea.com/>
> > > > >>
> > > > >>
> > > > >
> > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129 <+48660796129>
> > > > > [image: Polidea] <https://www.polidea.com/>
> > > >
> > >
> >
>
Re: [VOTE] Changes in import paths
Posted by Chen Tong <ci...@gmail.com>.
Daniel mentioned a good point. Such composed operator may also involves
both cloud and non-cloud provider saying FTPtoS3Operator. Should it in AWS
folder?
Also, saying in the future, another cloud provider is growing large enough.
Will we rename all plugins related to this provider? What's the criteria we
say it's big enough? And option D gives an impression like these tools may
be maintained and supported by the Cloud provider. I am not sure if it will
be a truth or not.
Best,
On Tue, Jul 30, 2019 at 11:18 AM Daniel Standish <dp...@gmail.com>
wrote:
> One wrinkle with case 3+4 option D is inter-provider operators. Mainly
> it's storage I think e.g. XToS3Operator or XToGCSOperator where the X is a
> service some different provider.
>
> Maybe the rule should be to locate the operator according to the first
> provider referenced. So e.g. s3_to_gcs_transfer_operator.py would go to
> aws.
>
>
> On Tue, Jul 30, 2019 at 6:21 AM Kamil Breguła <ka...@polidea.com>
> wrote:
>
> > Yes. All changes will be backwards compatible. In the case of using
> > the old path, a message containing a proposal for change will be
> > reported to the user.
> >
> > I prepared an example of how to change the name of a class in a case
> > with the use of a native solution.
> > Source code:
> > https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename
> > Preview from IDE: https://imgur.com/a/45NxS5W
> >
> > On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> > >
> > > Just cos I'm not sure it's _explicitly_ stated, but all of the moves
> > will have a deprecation of the old name right?
> > >
> > > 3+4 case D gets my vote too.
> > >
> > > -a
> > >
> > > > On 30 Jul 2019, at 09:58, Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> > > >
> > > > I went ahead and updated the page (just to speed it up) as I think it
> > > > really makes sense to join those two cases (and I do not see any
> > drawbacks
> > > > - I think the options we have cover all possible approaches) and we
> can
> > > > always go back if we need to.
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > > >
> > > > My vote is *D*.
> > > >
> > > > - airflow/contrib/operators/gcp_bigtable_operator.py →
> airflow/gcp
> > > > /operators/bigtable_operator.py
> > > > - airflow/contrib/operators/ssh_operator.py ->
> > > > airflow/operators/ssh_operator.py
> > > >
> > > > I updated the page with my vote.
> > > > I propose that we vote till Friday on that one (all the rest is
> already
> > > > agreed).
> > > >
> > > > J.
> > > >
> > > >
> > > > On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >
> > > > wrote:
> > > >
> > > >> I think almost everyone voted and we have almost perfect consensus.
> > We all
> > > >> agree amongst other on moving all operators out of contrib (Great!).
> > > >>
> > > >> The only doubts are for *Case 3* (Cloud provider prefix) and *Case
> 4*
> > > >> (Using Namespaces).
> > > >> I think there was actually an overlap between those two. Also Ash's
> > > >> comment/veto on that is quite clear.
> > > >> So I have final (I hope!) proposal to join those two cases and have
> > one
> > > >> voting (till Friday) only on that.
> > > >>
> > > >> I will update the doc if you all agree with me that makes more sense
> > as
> > > >> single case to vote on:
> > > >>
> > > >> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
> > > >>
> > > >> *Option A*. Keep operators/sensors/hooks in
> airflow/operators(sensors,
> > > >> hooks) and keep/add prefixes in file names.
> > > >>
> > > >> -
> > > >> *airflow/contrib/operators/sns_publish_operator.py becomes
> > > >> airflow/operators/**aws_sns_publish_operator.py*
> > > >>
> > > >>
> > > >> - *airflow/contrib/operators/dataproc_operator.py*
> > > >> becomes *airflow/operators/gcp_dataproc_operator.py*
> > > >>
> > > >>
> > > >> -
> > > >> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/*
> > > >> *hooks/gcp_bigtable_hook.py*
> > > >> -
> > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > > >> *operators/ssh_operator.py*
> > > >>
> > > >>
> > > >>
> > > >> *Option B*. Group operators/sensors/hooks in
> > airflow/operators(sensors,
> > > >> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
> > > >> airflow/operators(sensors/hooks), Drop the prefix.
> > > >>
> > > >> -
> > > >> *airflow/contrib/operators/sns_publish_operator.py
> > > >> becomes airflow/operators/**aws/sns_publish_operator.py*
> > > >>
> > > >>
> > > >> - *airflow/contrib/operators/dataproc_operator.py*
> > > >> becomes *airflow/operators/gcp/dataproc_operator.py*
> > > >>
> > > >>
> > > >> -
> > > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> > *airflow/*
> > > >> *operators/gcp/bigtable_operator.py*
> > > >> -
> > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > > >> *operators/ssh_operator.py*
> > > >>
> > > >>
> > > >> *Option C*. Group operators/sensors/hooks in
> > airflow/operators(sensors,
> > > >> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
> > > >> airflow/operators(sensors/hooks), Keep the prefix.
> > > >>
> > > >> -
> > > >> *airflow/contrib/operators/sns_publish_operator.py
> > > >> becomes airflow/operators/**aws/aws_sns_publish_operator.py*
> > > >>
> > > >>
> > > >> - *airflow/contrib/operators/dataproc_operator.py*
> > > >> becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
> > > >>
> > > >>
> > > >> -
> > > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> > *airflow/*
> > > >> *operators/gcp/gcp_bigtable_operator.py*
> > > >> -
> > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > > >> *operators/ssh_operator.py*
> > > >>
> > > >>
> > > >> *Option D.* Group operators/sensors/hooks in
> > *airflow/<PROVIDER>*/operators(sensors,
> > > >> hooks). Non-cloud provider ones are moved to
> > > >> airflow/operators(sensors/hooks). Drop the prefix.
> > > >>
> > > >> -
> > > >> *airflow/contrib/operators/sns_publish_operator.py
> > > >> becomes airflow/aws/operators/**sns_publish_operator.py*
> > > >>
> > > >>
> > > >> - *airflow/contrib/operators/dataproc_operator.py*
> > > >> becomes *airflow/gcp/operators/dataproc_operator.py*
> > > >>
> > > >>
> > > >> -
> > > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> > *airflow/*
> > > >> *gcp/operators/bigtable_operator.py*
> > > >> -
> > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > > >> *operators/ssh_operator.py*
> > > >>
> > > >>
> > > >> *Option E.* Group operators/sensors/hooks in
> > *airflow/<PROVIDER>*/operators(sensors,
> > > >> hooks). Non-cloud provider ones are moved to
> > > >> airflow/operators(sensors/hooks).* Keep the prefix*.
> > > >>
> > > >> -
> > > >> *airflow/contrib/operators/sns_publish_operator.py
> > > >> becomes airflow/aws/operators/aws_**sns_publish_operator.py*
> > > >>
> > > >>
> > > >> - *airflow/contrib/operators/dataproc_operator.py*
> > > >> becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
> > > >>
> > > >>
> > > >> -
> > > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> > *airflow/*
> > > >> *gcp/operators/gcp_bigtable_operator.py*
> > > >> -
> > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > > >> *operators/ssh_operator.py*
> > > >>
> > > >>
> > > >> Shall I proceed with updating the page ?
> > > >>
> > > >>
> > > >> J.
> > > >>
> > > >>
> > > >> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <ka...@gmail.com>
> > wrote:
> > > >>
> > > >>> Thanks Jarek, adding my vote.
> > > >>>
> > > >>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > > >>>
> > > >>>> Thanks Jarek, much clearer. I have voted there too.
> > > >>>>
> > > >>>> -ash
> > > >>>>
> > > >>>>
> > > >>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko <fokko@driesprong.frl
> >
> > > >>>> wrote:
> > > >>>>>
> > > >>>>> Thanks Jarek, the Wiki works much better for me. I've added my
> vote
> > > >>> there
> > > >>>>> as well.
> > > >>>>>
> > > >>>>> You've convinced me on the second case. I've changed my vote on
> > Case 2
> > > >>>> from
> > > >>>>> A to B :-)
> > > >>>>>
> > > >>>>> Maybe we should leave the vote open a bit longer since it
> involves
> > > >>> quite
> > > >>>>> some reading, and I would like to get some proper feedback and
> > > >>> opinions
> > > >>>> on
> > > >>>>> this. Plus I only see two votes right now. If we don't think this
> > > >>>> through,
> > > >>>>> it might be that we're having this vote again in the near future
> > :-)
> > > >>>>>
> > > >>>>> Cheers, Fokko
> > > >>>>>
> > > >>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
> > > >>>> Jarek.Potiuk@polidea.com>:
> > > >>>>>
> > > >>>>>> I updated the AIP-21 proposal in the way that should answer all
> > the
> > > >>>>>> concerns in this thread.
> > > >>>>>>
> > > >>>>>> NOTE! There is an action for all of those who are interested:
> > Please
> > > >>>>>> fill-in your voting in the voting table till Tuesday 30th of
> July
> > 6pm
> > > >>>> CEST
> > > >>>>>> (5pm BST, 9 am PST).
> > > >>>>>>
> > > >>>>>> I created a summary of the proposal
> > > >>>>>> <
> > > >>>>>>
> > > >>>>
> > > >>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > > >>>>>>>
> > > >>>>>> in table form - which contains also examples for each case.
> > > >>>>>>
> > > >>>>>> I added a voting table
> > > >>>>>> <
> > > >>>>>>
> > > >>>>
> > > >>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> > > >>>>>>>
> > > >>>>>> where you should add your votes:
> > > >>>>>>
> > > >>>>>> I propose this voting mechanism:
> > > >>>>>>
> > > >>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5
> pm
> > > >>> BST,
> > > >>>> 9am
> > > >>>>>> PST) .
> > > >>>>>>
> > > >>>>>> After the choice, the final consistent set of choices will be
> > > >>> announced
> > > >>>>>> (taking into account majority of binding vote, also including
> > > >>> potential
> > > >>>>>> vetos and consistency between the choices. Non-binding votes
> will
> > be
> > > >>>> taken
> > > >>>>>> into account in case there is a draw. The final set of choices
> > will
> > > >>> be
> > > >>>>>> announced at devlist thread
> > > >>>>>> <
> > > >>>>>>
> > > >>>>
> > > >>>
> >
> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> > > >>>>>>>
> > > >>>>>> after
> > > >>>>>> the voting completes.
> > > >>>>>>
> > > >>>>>> Unless there is a veto raised to the final proposal, the final
> > > >>> proposal
> > > >>>> is
> > > >>>>>> accepted by Lazy Consensus
> > > >>>>>> <https://community.apache.org/committers/lazyConsensus.html>
> on
> > > >>>> *Friday*
> > > >>>>>> 02
> > > >>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
> > > >>>>>>
> > > >>>>>> Please let me know if what I proposed can be further improved to
> > > >>> avoid
> > > >>>>>> chaos.
> > > >>>>>>
> > > >>>>>> J.
> > > >>>>>>
> > > >>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
> > > >>> Jarek.Potiuk@polidea.com
> > > >>>>>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Ok. I see that indeed voting could have been organised a bit
> > better
> > > >>> -
> > > >>>> dev
> > > >>>>>>> list does not seem to cope well with such a complex case :).
> > > >>>>>>>
> > > >>>>>>> As Kamil noticed - we already have a formal AIP-21
> > > >>>>>>> <
> > > >>>>>>
> > > >>>>
> > > >>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > > >>>>>>>
> > > >>>>>>> in the Wiki - which I should have mentioned before. I have not
> > much
> > > >>>> time
> > > >>>>>>> today - but tomorrow I will try to summarise the options -
> based
> > on
> > > >>>> some
> > > >>>>>>> real code examples (to make it easier to digest). I think the
> > best
> > > >>>>>> approach
> > > >>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where
> > people
> > > >>>> will
> > > >>>>>>> be able to state their preferences for each case (by adding a
> > row to
> > > >>>> the
> > > >>>>>>> table). I think that might provide much better clarity and a
> > single
> > > >>>> place
> > > >>>>>>> which will show the current aggregated state of voting.
> > > >>>>>>>
> > > >>>>>>> However - as Ash also stated - some of the options are
> > > >>> inter-dependent
> > > >>>> so
> > > >>>>>>> at the end we will have to adjust the results in case they are
> > not
> > > >>>>>>> consistent - and make final voting on aggregated set of cases
> -
> > > >>> but I
> > > >>>>>>> think in this case following "lasy consensus" - i,e. if noone
> > > >>> objects
> > > >>>> to
> > > >>>>>>> final set of cases, we will proceed with it..
> > > >>>>>>>
> > > >>>>>>> Do you think that will work better ?
> > > >>>>>>>
> > > >>>>>>> J.
> > > >>>>>>>
> > > >>>>>>> Principal Software Engineer
> > > >>>>>>> Phone: +48660796129
> > > >>>>>>>
> > > >>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> > > >>>>>>> kamil.bregula@polidea.com> napisał:
> > > >>>>>>>
> > > >>>>>>>> Document is also available as AIP on wiki:
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > > >>>>>>>>
> > > >>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
> > > >>> <fokko@driesprong.frl
> > > >>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> I agree with all, this is a bit chaotic. For the sake of
> > clarity,
> > > >>> I
> > > >>>>>>>> would
> > > >>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a
> > > >>>>>> structural
> > > >>>>>>>>> code improvement AIP and then and add a matrix of users/cases
> > > >>> where
> > > >>>>>>>>> everybody can add his vote per case. This gives also the
> > > >>> opportunity
> > > >>>>>> for
> > > >>>>>>>>> changing your vote on a specific case. WDYT?
> > > >>>>>>>>>
> > > >>>>>>>>> Cheers, Fokko
> > > >>>>>>>>>
> > > >>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <
> > cixuuz@gmail.com>:
> > > >>>>>>>>>
> > > >>>>>>>>>> I agree with Ash. It is clear to vote on each case rather
> than
> > > >>> all
> > > >>>>>>>>>> together.
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
> > > >>> ash@apache.org>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> A number of proposals overlap or add on to each other, so I
> > > >>> think
> > > >>>>>>>> (?) a
> > > >>>>>>>>>>> single vote makes sense.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what we
> > are
> > > >>>>>>>> voting
> > > >>>>>>>>> on.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <
> > kaxilnaik@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases
> > together,
> > > >>>>>> how
> > > >>>>>>>>>>>> about
> > > >>>>>>>>>>>> separate vote emails for all the cases? Each email can
> have
> > the
> > > >>>>>>>>>>>> description
> > > >>>>>>>>>>>> copied over from the doc.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> It would probably be a bit easier to track a decision in
> the
> > > >>>>>> future
> > > >>>>>>>> as
> > > >>>>>>>>>>>> well.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
> > > >>> Jarek.Potiuk@polidea.com>
> > > >>>>>>>>>>>> @Driesprong,
> > > >>>>>>>>>>>> Fokko <fo...@driesprong.frl>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
> > > >>> kaxilnaik@gmail.com>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> *Case #4* Separate namespace for resources :
> > > >>>>>>>>>>>>> *Solution A*
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> *Case #7* : *Solution A*
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> > > >>>>>>>>>>>> <dp...@gmail.com>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I have tried to add some clarification to Jarek's
> summary,
> > > >>> but
> > > >>>>>> I
> > > >>>>>>>> am
> > > >>>>>>>>>>>> a
> > > >>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully
> > Jarek
> > > >>>>>> can
> > > >>>>>>>>>>>> further
> > > >>>>>>>>>>>>>> clarify.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6 are
> > all
> > > >>>>>>>> either
> > > >>>>>>>>>>>>>> proposal
> > > >>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention
> > can be
> > > >>>>>>>>>>>> focused on
> > > >>>>>>>>>>>>>> cases 1,2,3 and 7.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> *Motion is to adopt the following:*
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
> > > >>>>>>>>>>>>>> All objects will be moved out of contrib folder and into
> > > >>>>>>>> respective
> > > >>>>>>>>>>>>>> non-contrib folder.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from
> > modules
> > > >>>>>>>>>>>>>> Example:
> > > >>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
> > > >>>>>>>>>>>>>> airflow.operators.foo.FooOperator
> > > >>>>>>>>>>>>>> Example:
> > > >>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
> > > >>>>>>>> airflow.hooks.foo.FooHook
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
> > > >>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
> > > >>>>>>>>>>>>>> Example case is
> > > >>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
> > > >>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume it
> is
> > > >>> now
> > > >>>>>>>> named
> > > >>>>>>>>>>>> like
> > > >>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
> > > >>>>>>>>>>>>>> So what is proposed for this case?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Case 4 - Solution B - *no change*
> > > >>>>>>>>>>>>>> This proposal was about namespaces but the motion is to
> > > >>> reject
> > > >>>>>>>> the
> > > >>>>>>>>>>>>>> proposal.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Case 5 - Solution B - *no change*
> > > >>>>>>>>>>>>>> The proposal was to remove class type suffix e.g. remove
> > the
> > > >>>>>>>>>>>> Operator in
> > > >>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal
> --
> > > >>> i.e.
> > > >>>>>>>>>>>> motion is
> > > >>>>>>>>>>>>>> no
> > > >>>>>>>>>>>>>> change on this case, i.e. we resolve to keep "Operator"
> > (and
> > > >>>>>>>>>>>> "Sensor")
> > > >>>>>>>>>>>>>> suffixes on class names
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Case 6 - *No change*
> > > >>>>>>>>>>>>>> This case concerns object naming inconsistencies, but
> > motion
> > > >>>>>> does
> > > >>>>>>>>>>>> not
> > > >>>>>>>>>>>>>> enact
> > > >>>>>>>>>>>>>> any specific proposal.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
> > > >>>>>> deprecation
> > > >>>>>>>>>>>> warnings
> > > >>>>>>>>>>>>>> (instead of zope)
> > > >>>>>>>>>>>>>> Example:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> # in old_package/old_mod.py
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> import warnings
> > > >>>>>>>>>>>>>> from new_package.new_mod import *
> > > >>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
> > > >>>>>>>> new_package.new_mod.
> > > >>>>>>>>>>>>>> Import
> > > >>>>>>>>>>>>>> of "
> > > >>>>>>>>>>>>>> "old_package.old_mod will become unsupported in version
> > 2",
> > > >>>>>>>>>>>>>> DeprecationWarning, 2)
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Reference implementation here:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> >
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I am very happy that this motion now gets rid of
> contrib.
> > > >>>>>> Thank
> > > >>>>>>>> you
> > > >>>>>>>>>>>>>> Sergio
> > > >>>>>>>>>>>>>> for turning the tide with your effective argumentation
> ;)
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
> > > >>>>>>>> ash@apache.org>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Could someone summarise what the proposed name changes
> > are,
> > > >>>>>>>> here,
> > > >>>>>>>>>>>> in
> > > >>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving
> partial
> > > >>>>>>>> examples
> > > >>>>>>>>>>>> is 1)
> > > >>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting
> on,
> > and
> > > >>>>>> 2)
> > > >>>>>>>>>>>> using a
> > > >>>>>>>>>>>>>>> google doc isn't great for a longer term record.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>> Ash
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
> > > >>>>>>>>>>>> <Ja...@polidea.com>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :)
> > . I
> > > >>>>>>>> would
> > > >>>>>>>>>>>> like
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three
> +1
> > > >>>>>>>>>>>> (binding)
> > > >>>>>>>>>>>>>> votes
> > > >>>>>>>>>>>>>>> are
> > > >>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to
> the
> > > >>>>>>>>>>>> previous
> > > >>>>>>>>>>>>>>> proposal.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Here is the proposal (details here
> > > >>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> )
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
> > > >>>>>>>>>>>>>>>> - *Case 2: A:
> Airflow.operators.foo_operator.FooOperator
> > > >>>>>>>> could
> > > >>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > > >>>>>>>>>>>>>>>> - *Case 3: C:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > > >>>>>>>>>>>>>> could
> > > >>>>>>>>>>>>>>>> become
> airflow.gcp.operators.bigtable.BigTableOperator*
> > > >>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > > >>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes
> on
> > > >>>>>>>> class
> > > >>>>>>>>>>>>>> names*
> > > >>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> case-by-case
> > > >>>>>>>> (and
> > > >>>>>>>>>>>> my team
> > > >>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > > >>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
> > > >>>>>>>>>>>> integration)*
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> This is my binding (+1) vote.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> > > >>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
> > > >>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hey Felix,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as
> > well
> > > >>>>>> (I
> > > >>>>>>>>>>>> use IDE
> > > >>>>>>>>>>>>>>> quite
> > > >>>>>>>>>>>>>>>>> often ;) )
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> J.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> > > >>>>>>>>>>>>>> <fokko@driesprong.frl
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I
> > wasn't
> > > >>>>>>>> able
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> respond
> > > >>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops
> > throughout
> > > >>>>>>>> Europe
> > > >>>>>>>>>>>> :-)
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from
> contrib
> > > >>>>>> into
> > > >>>>>>>> a
> > > >>>>>>>>>>>> single
> > > >>>>>>>>>>>>>>>>>> package.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference
> > would be
> > > >>>>>>>> to
> > > >>>>>>>>>>>> get
> > > >>>>>>>>>>>>>> rid of
> > > >>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or not
> > is a
> > > >>>>>>>>>>>> tricky
> > > >>>>>>>>>>>>>>> question,
> > > >>>>>>>>>>>>>>>>>> I think this is already reflected in the document
> > since
> > > >>>>>>>>>>>> "maturity"
> > > >>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the
> > operator
> > > >>>>>>>> to
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> core
> > > >>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
> > > >>>>>> incubating
> > > >>>>>>>>>>>> will
> > > >>>>>>>>>>>>>> confuse
> > > >>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> user even more. For people who aren't as known with
> > > >>>>>>>> Airflow it
> > > >>>>>>>>>>>> isn't
> > > >>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>> transparent which operator lives where, especially
> if
> > you
> > > >>>>>>>>>>>> don't use
> > > >>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>> IDE
> > > >>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is
> worth
> > > >>>>>>>> changing
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> public
> > > >>>>>>>>>>>>>>>>>> API just for a different name.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with
> this.
> > It
> > > >>>>>>>> will
> > > >>>>>>>>>>>> make
> > > >>>>>>>>>>>>>>> easier
> > > >>>>>>>>>>>>>>>>> as we will not have to define other rules as those
> > that we
> > > >>>>>>>>>>>> already
> > > >>>>>>>>>>>>>> have.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's
> say a
> > > >>>>>> +0)
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search
> for
> > > >>>>>>>>>>>> BashOperator.
> > > >>>>>>>>>>>>>>> After
> > > >>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from
> your
> > > >>>>>> mail
> > > >>>>>>>> you
> > > >>>>>>>>>>>>>> state: -
> > > >>>>>>>>>>>>>>>>>> *Case 2: B:
> Airflow.operators.foo_operator.FooOperator
> > > >>>>>>>> could
> > > >>>>>>>>>>>> become
> > > >>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B
> > solution is
> > > >>>>>>>>>>>> keeping it
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
> > > >>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would
> stay
> > > >>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a
> > little
> > > >>>>>>>>>>>> confusion
> > > >>>>>>>>>>>>>>> here.
> > > >>>>>>>>>>>>>>>>> This case is not for changing BashOperator into Bash
> > but
> > > >>>>>> for
> > > >>>>>>>>>>>> changing
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> *module* name not the *class* name. So
> > > >>>>>>>>>>>> bash_operator.BashOperator ->
> > > >>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for
> > > >>>>>>>> BashOperator
> > > >>>>>>>>>>>> in this
> > > >>>>>>>>>>>>>>> case.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it
> > gets
> > > >>>>>>>> its
> > > >>>>>>>>>>>> own
> > > >>>>>>>>>>>>>>> package
> > > >>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are
> > companies
> > > >>>>>>>> like
> > > >>>>>>>>>>>>>>> Databricks
> > > >>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own
> > package
> > > >>>>>>>>>>>> because
> > > >>>>>>>>>>>>>> their
> > > >>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving this
> > is a
> > > >>>>>>>> bit
> > > >>>>>>>>>>>>>> painful
> > > >>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move
> > > >>>>>>>> everything
> > > >>>>>>>>>>>> which
> > > >>>>>>>>>>>>>>>>>> introduces noise into version control etc.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not
> as
> > > >>>>>>>>>>>> difficult as
> > > >>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>> seems to group operators in packages and it has
> > numerous
> > > >>>>>>>>>>>> advantages.
> > > >>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in the
> > > >>>>>>>>>>>> __init__.py of
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git
> will
> > > >>>>>>>> actually
> > > >>>>>>>>>>>>>> realise
> > > >>>>>>>>>>>>>>>>> that the file has been moved and you can follow that.
> > > >>>>>> Maybe
> > > >>>>>>>> not
> > > >>>>>>>>>>>> as
> > > >>>>>>>>>>>>>> super
> > > >>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's not a
> > > >>>>>> rocket
> > > >>>>>>>>>>>> science
> > > >>>>>>>>>>>>>>> either.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Yes.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> ---
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Apart from all, great work!
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Cheers, Fokko
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> > > >>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
> > > >>>>>>>>>>>>>>>>>>> :
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change
> of
> > > >>>>>> these
> > > >>>>>>>>>>>> cases.
> > > >>>>>>>>>>>>>> So I
> > > >>>>>>>>>>>>>>>>>> go
> > > >>>>>>>>>>>>>>>>>>> with +1 (non-binding).
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and
> your
> > > >>>>>>>> intense
> > > >>>>>>>>>>>> work
> > > >>>>>>>>>>>>>>>>>> towards
> > > >>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
> > > >>>>>> creating
> > > >>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>> document. I
> > > >>>>>>>>>>>>>>>>>>> like how the code is getting more and more
> consistent
> > > >>>>>> and
> > > >>>>>>>>>>>> clean :)
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Kind regards,
> > > >>>>>>>>>>>>>>>>>>> Felix
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> -------- Original Message --------
> > > >>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in
> > import
> > > >>>>>>>> paths.
> > > >>>>>>>>>>>> It's
> > > >>>>>>>>>>>>>>> been
> > > >>>>>>>>>>>>>>>>>>>> discussed in
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> >
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th
> > 6pm
> > > >>>>>>>> CEST),
> > > >>>>>>>>>>>> and
> > > >>>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>>>>>> least
> > > >>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document from
> > > >>>>>> Kamil
> > > >>>>>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> The proposed solution is:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are
> > > >>>>>> "well"
> > > >>>>>>>>>>>> tested
> > > >>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
> > > >>>>>>>>>>>>>>>>>>>> - *Case 2: B:
> > > >>>>>> Airflow.operators.foo_operator.FooOperator
> > > >>>>>>>>>>>> could
> > > >>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > > >>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > > >>>>>>>>>>>>>>> could
> > > >>>>>>>>>>>>>>>>>>>> become
> > airflow.gcp.operators.bigtable.BigTableOperator*
> > > >>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > > >>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
> > suffixes
> > > >>>>>> on
> > > >>>>>>>>>>>> class
> > > >>>>>>>>>>>>>> names*
> > > >>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> > case-by-case
> > > >>>>>>>> (and
> > > >>>>>>>>>>>> my
> > > >>>>>>>>>>>>>> team
> > > >>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> J.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> > > >>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> > > >>>>>> Software
> > > >>>>>>>>>>>> Engineer
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
> > > >>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
> > > >>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Jarek Potiuk
> > > >>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> > Software
> > > >>>>>>>>>>>> Engineer
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > > >>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Jarek Potiuk
> > > >>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> Software
> > > >>>>>>>>> Engineer
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > > >>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> --
> > > >>>>>>>>>>>>> *Kaxil Naik*
> > > >>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > > >>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
> Apache
> > > >>>>>> Spark
> > > >>>>>>>> &
> > > >>>>>>>>>>>> Neo4j
> > > >>>>>>>>>>>>> Developer
> > > >>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> --
> > > >>>>>>>>>>>> *Kaxil Naik*
> > > >>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > > >>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified*
> Apache
> > > >>> Spark
> > > >>>>>> &
> > > >>>>>>>>>>>> Neo4j
> > > >>>>>>>>>>>> Developer
> > > >>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>>
> > > >>>>>> Jarek Potiuk
> > > >>>>>> Polidea <https://www.polidea.com/> | Principal Software
> Engineer
> > > >>>>>>
> > > >>>>>> M: +48 660 796 129 <+48660796129>
> > > >>>>>> [image: Polidea] <https://www.polidea.com/>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>> --
> > > >>> *Kaxil Naik*
> > > >>> *Big Data Consultant | DevOps Data Engineer*
> > > >>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> > Neo4j
> > > >>> Developer
> > > >>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Jarek Potiuk
> > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >>
> > > >> M: +48 660 796 129 <+48660796129>
> > > >> [image: Polidea] <https://www.polidea.com/>
> > > >>
> > > >>
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > >
> >
>
Re: [VOTE] Changes in import paths
Posted by Daniel Standish <dp...@gmail.com>.
One wrinkle with case 3+4 option D is inter-provider operators. Mainly
it's storage I think e.g. XToS3Operator or XToGCSOperator where the X is a
service some different provider.
Maybe the rule should be to locate the operator according to the first
provider referenced. So e.g. s3_to_gcs_transfer_operator.py would go to
aws.
On Tue, Jul 30, 2019 at 6:21 AM Kamil Breguła <ka...@polidea.com>
wrote:
> Yes. All changes will be backwards compatible. In the case of using
> the old path, a message containing a proposal for change will be
> reported to the user.
>
> I prepared an example of how to change the name of a class in a case
> with the use of a native solution.
> Source code:
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename
> Preview from IDE: https://imgur.com/a/45NxS5W
>
> On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > Just cos I'm not sure it's _explicitly_ stated, but all of the moves
> will have a deprecation of the old name right?
> >
> > 3+4 case D gets my vote too.
> >
> > -a
> >
> > > On 30 Jul 2019, at 09:58, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> > >
> > > I went ahead and updated the page (just to speed it up) as I think it
> > > really makes sense to join those two cases (and I do not see any
> drawbacks
> > > - I think the options we have cover all possible approaches) and we can
> > > always go back if we need to.
> > >
> > >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > >
> > > My vote is *D*.
> > >
> > > - airflow/contrib/operators/gcp_bigtable_operator.py → airflow/gcp
> > > /operators/bigtable_operator.py
> > > - airflow/contrib/operators/ssh_operator.py ->
> > > airflow/operators/ssh_operator.py
> > >
> > > I updated the page with my vote.
> > > I propose that we vote till Friday on that one (all the rest is already
> > > agreed).
> > >
> > > J.
> > >
> > >
> > > On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > > wrote:
> > >
> > >> I think almost everyone voted and we have almost perfect consensus.
> We all
> > >> agree amongst other on moving all operators out of contrib (Great!).
> > >>
> > >> The only doubts are for *Case 3* (Cloud provider prefix) and *Case 4*
> > >> (Using Namespaces).
> > >> I think there was actually an overlap between those two. Also Ash's
> > >> comment/veto on that is quite clear.
> > >> So I have final (I hope!) proposal to join those two cases and have
> one
> > >> voting (till Friday) only on that.
> > >>
> > >> I will update the doc if you all agree with me that makes more sense
> as
> > >> single case to vote on:
> > >>
> > >> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
> > >>
> > >> *Option A*. Keep operators/sensors/hooks in airflow/operators(sensors,
> > >> hooks) and keep/add prefixes in file names.
> > >>
> > >> -
> > >> *airflow/contrib/operators/sns_publish_operator.py becomes
> > >> airflow/operators/**aws_sns_publish_operator.py*
> > >>
> > >>
> > >> - *airflow/contrib/operators/dataproc_operator.py*
> > >> becomes *airflow/operators/gcp_dataproc_operator.py*
> > >>
> > >>
> > >> -
> > >> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/*
> > >> *hooks/gcp_bigtable_hook.py*
> > >> -
> > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > >> *operators/ssh_operator.py*
> > >>
> > >>
> > >>
> > >> *Option B*. Group operators/sensors/hooks in
> airflow/operators(sensors,
> > >> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
> > >> airflow/operators(sensors/hooks), Drop the prefix.
> > >>
> > >> -
> > >> *airflow/contrib/operators/sns_publish_operator.py
> > >> becomes airflow/operators/**aws/sns_publish_operator.py*
> > >>
> > >>
> > >> - *airflow/contrib/operators/dataproc_operator.py*
> > >> becomes *airflow/operators/gcp/dataproc_operator.py*
> > >>
> > >>
> > >> -
> > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> *airflow/*
> > >> *operators/gcp/bigtable_operator.py*
> > >> -
> > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > >> *operators/ssh_operator.py*
> > >>
> > >>
> > >> *Option C*. Group operators/sensors/hooks in
> airflow/operators(sensors,
> > >> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
> > >> airflow/operators(sensors/hooks), Keep the prefix.
> > >>
> > >> -
> > >> *airflow/contrib/operators/sns_publish_operator.py
> > >> becomes airflow/operators/**aws/aws_sns_publish_operator.py*
> > >>
> > >>
> > >> - *airflow/contrib/operators/dataproc_operator.py*
> > >> becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
> > >>
> > >>
> > >> -
> > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> *airflow/*
> > >> *operators/gcp/gcp_bigtable_operator.py*
> > >> -
> > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > >> *operators/ssh_operator.py*
> > >>
> > >>
> > >> *Option D.* Group operators/sensors/hooks in
> *airflow/<PROVIDER>*/operators(sensors,
> > >> hooks). Non-cloud provider ones are moved to
> > >> airflow/operators(sensors/hooks). Drop the prefix.
> > >>
> > >> -
> > >> *airflow/contrib/operators/sns_publish_operator.py
> > >> becomes airflow/aws/operators/**sns_publish_operator.py*
> > >>
> > >>
> > >> - *airflow/contrib/operators/dataproc_operator.py*
> > >> becomes *airflow/gcp/operators/dataproc_operator.py*
> > >>
> > >>
> > >> -
> > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> *airflow/*
> > >> *gcp/operators/bigtable_operator.py*
> > >> -
> > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > >> *operators/ssh_operator.py*
> > >>
> > >>
> > >> *Option E.* Group operators/sensors/hooks in
> *airflow/<PROVIDER>*/operators(sensors,
> > >> hooks). Non-cloud provider ones are moved to
> > >> airflow/operators(sensors/hooks).* Keep the prefix*.
> > >>
> > >> -
> > >> *airflow/contrib/operators/sns_publish_operator.py
> > >> becomes airflow/aws/operators/aws_**sns_publish_operator.py*
> > >>
> > >>
> > >> - *airflow/contrib/operators/dataproc_operator.py*
> > >> becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
> > >>
> > >>
> > >> -
> > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes
> *airflow/*
> > >> *gcp/operators/gcp_bigtable_operator.py*
> > >> -
> > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> > >> *operators/ssh_operator.py*
> > >>
> > >>
> > >> Shall I proceed with updating the page ?
> > >>
> > >>
> > >> J.
> > >>
> > >>
> > >> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <ka...@gmail.com>
> wrote:
> > >>
> > >>> Thanks Jarek, adding my vote.
> > >>>
> > >>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> > >>>
> > >>>> Thanks Jarek, much clearer. I have voted there too.
> > >>>>
> > >>>> -ash
> > >>>>
> > >>>>
> > >>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko <fo...@driesprong.frl>
> > >>>> wrote:
> > >>>>>
> > >>>>> Thanks Jarek, the Wiki works much better for me. I've added my vote
> > >>> there
> > >>>>> as well.
> > >>>>>
> > >>>>> You've convinced me on the second case. I've changed my vote on
> Case 2
> > >>>> from
> > >>>>> A to B :-)
> > >>>>>
> > >>>>> Maybe we should leave the vote open a bit longer since it involves
> > >>> quite
> > >>>>> some reading, and I would like to get some proper feedback and
> > >>> opinions
> > >>>> on
> > >>>>> this. Plus I only see two votes right now. If we don't think this
> > >>>> through,
> > >>>>> it might be that we're having this vote again in the near future
> :-)
> > >>>>>
> > >>>>> Cheers, Fokko
> > >>>>>
> > >>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
> > >>>> Jarek.Potiuk@polidea.com>:
> > >>>>>
> > >>>>>> I updated the AIP-21 proposal in the way that should answer all
> the
> > >>>>>> concerns in this thread.
> > >>>>>>
> > >>>>>> NOTE! There is an action for all of those who are interested:
> Please
> > >>>>>> fill-in your voting in the voting table till Tuesday 30th of July
> 6pm
> > >>>> CEST
> > >>>>>> (5pm BST, 9 am PST).
> > >>>>>>
> > >>>>>> I created a summary of the proposal
> > >>>>>> <
> > >>>>>>
> > >>>>
> > >>>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > >>>>>>>
> > >>>>>> in table form - which contains also examples for each case.
> > >>>>>>
> > >>>>>> I added a voting table
> > >>>>>> <
> > >>>>>>
> > >>>>
> > >>>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> > >>>>>>>
> > >>>>>> where you should add your votes:
> > >>>>>>
> > >>>>>> I propose this voting mechanism:
> > >>>>>>
> > >>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 pm
> > >>> BST,
> > >>>> 9am
> > >>>>>> PST) .
> > >>>>>>
> > >>>>>> After the choice, the final consistent set of choices will be
> > >>> announced
> > >>>>>> (taking into account majority of binding vote, also including
> > >>> potential
> > >>>>>> vetos and consistency between the choices. Non-binding votes will
> be
> > >>>> taken
> > >>>>>> into account in case there is a draw. The final set of choices
> will
> > >>> be
> > >>>>>> announced at devlist thread
> > >>>>>> <
> > >>>>>>
> > >>>>
> > >>>
> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> > >>>>>>>
> > >>>>>> after
> > >>>>>> the voting completes.
> > >>>>>>
> > >>>>>> Unless there is a veto raised to the final proposal, the final
> > >>> proposal
> > >>>> is
> > >>>>>> accepted by Lazy Consensus
> > >>>>>> <https://community.apache.org/committers/lazyConsensus.html> on
> > >>>> *Friday*
> > >>>>>> 02
> > >>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
> > >>>>>>
> > >>>>>> Please let me know if what I proposed can be further improved to
> > >>> avoid
> > >>>>>> chaos.
> > >>>>>>
> > >>>>>> J.
> > >>>>>>
> > >>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
> > >>> Jarek.Potiuk@polidea.com
> > >>>>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Ok. I see that indeed voting could have been organised a bit
> better
> > >>> -
> > >>>> dev
> > >>>>>>> list does not seem to cope well with such a complex case :).
> > >>>>>>>
> > >>>>>>> As Kamil noticed - we already have a formal AIP-21
> > >>>>>>> <
> > >>>>>>
> > >>>>
> > >>>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > >>>>>>>
> > >>>>>>> in the Wiki - which I should have mentioned before. I have not
> much
> > >>>> time
> > >>>>>>> today - but tomorrow I will try to summarise the options - based
> on
> > >>>> some
> > >>>>>>> real code examples (to make it easier to digest). I think the
> best
> > >>>>>> approach
> > >>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where
> people
> > >>>> will
> > >>>>>>> be able to state their preferences for each case (by adding a
> row to
> > >>>> the
> > >>>>>>> table). I think that might provide much better clarity and a
> single
> > >>>> place
> > >>>>>>> which will show the current aggregated state of voting.
> > >>>>>>>
> > >>>>>>> However - as Ash also stated - some of the options are
> > >>> inter-dependent
> > >>>> so
> > >>>>>>> at the end we will have to adjust the results in case they are
> not
> > >>>>>>> consistent - and make final voting on aggregated set of cases -
> > >>> but I
> > >>>>>>> think in this case following "lasy consensus" - i,e. if noone
> > >>> objects
> > >>>> to
> > >>>>>>> final set of cases, we will proceed with it..
> > >>>>>>>
> > >>>>>>> Do you think that will work better ?
> > >>>>>>>
> > >>>>>>> J.
> > >>>>>>>
> > >>>>>>> Principal Software Engineer
> > >>>>>>> Phone: +48660796129
> > >>>>>>>
> > >>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> > >>>>>>> kamil.bregula@polidea.com> napisał:
> > >>>>>>>
> > >>>>>>>> Document is also available as AIP on wiki:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > >>>>>>>>
> > >>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
> > >>> <fokko@driesprong.frl
> > >>>>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> I agree with all, this is a bit chaotic. For the sake of
> clarity,
> > >>> I
> > >>>>>>>> would
> > >>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a
> > >>>>>> structural
> > >>>>>>>>> code improvement AIP and then and add a matrix of users/cases
> > >>> where
> > >>>>>>>>> everybody can add his vote per case. This gives also the
> > >>> opportunity
> > >>>>>> for
> > >>>>>>>>> changing your vote on a specific case. WDYT?
> > >>>>>>>>>
> > >>>>>>>>> Cheers, Fokko
> > >>>>>>>>>
> > >>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <
> cixuuz@gmail.com>:
> > >>>>>>>>>
> > >>>>>>>>>> I agree with Ash. It is clear to vote on each case rather than
> > >>> all
> > >>>>>>>>>> together.
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
> > >>> ash@apache.org>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> A number of proposals overlap or add on to each other, so I
> > >>> think
> > >>>>>>>> (?) a
> > >>>>>>>>>>> single vote makes sense.
> > >>>>>>>>>>>
> > >>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what we
> are
> > >>>>>>>> voting
> > >>>>>>>>> on.
> > >>>>>>>>>>>
> > >>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <
> kaxilnaik@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases
> together,
> > >>>>>> how
> > >>>>>>>>>>>> about
> > >>>>>>>>>>>> separate vote emails for all the cases? Each email can have
> the
> > >>>>>>>>>>>> description
> > >>>>>>>>>>>> copied over from the doc.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> It would probably be a bit easier to track a decision in the
> > >>>>>> future
> > >>>>>>>> as
> > >>>>>>>>>>>> well.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
> > >>> Jarek.Potiuk@polidea.com>
> > >>>>>>>>>>>> @Driesprong,
> > >>>>>>>>>>>> Fokko <fo...@driesprong.frl>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
> > >>> kaxilnaik@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> *Case #4* Separate namespace for resources :
> > >>>>>>>>>>>>> *Solution A*
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> *Case #7* : *Solution A*
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> > >>>>>>>>>>>> <dp...@gmail.com>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I have tried to add some clarification to Jarek's summary,
> > >>> but
> > >>>>>> I
> > >>>>>>>> am
> > >>>>>>>>>>>> a
> > >>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully
> Jarek
> > >>>>>> can
> > >>>>>>>>>>>> further
> > >>>>>>>>>>>>>> clarify.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6 are
> all
> > >>>>>>>> either
> > >>>>>>>>>>>>>> proposal
> > >>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention
> can be
> > >>>>>>>>>>>> focused on
> > >>>>>>>>>>>>>> cases 1,2,3 and 7.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> *Motion is to adopt the following:*
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
> > >>>>>>>>>>>>>> All objects will be moved out of contrib folder and into
> > >>>>>>>> respective
> > >>>>>>>>>>>>>> non-contrib folder.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from
> modules
> > >>>>>>>>>>>>>> Example:
> > >>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
> > >>>>>>>>>>>>>> airflow.operators.foo.FooOperator
> > >>>>>>>>>>>>>> Example:
> > >>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
> > >>>>>>>> airflow.hooks.foo.FooHook
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
> > >>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
> > >>>>>>>>>>>>>> Example case is
> > >>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
> > >>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume it is
> > >>> now
> > >>>>>>>> named
> > >>>>>>>>>>>> like
> > >>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
> > >>>>>>>>>>>>>> So what is proposed for this case?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Case 4 - Solution B - *no change*
> > >>>>>>>>>>>>>> This proposal was about namespaces but the motion is to
> > >>> reject
> > >>>>>>>> the
> > >>>>>>>>>>>>>> proposal.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Case 5 - Solution B - *no change*
> > >>>>>>>>>>>>>> The proposal was to remove class type suffix e.g. remove
> the
> > >>>>>>>>>>>> Operator in
> > >>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal --
> > >>> i.e.
> > >>>>>>>>>>>> motion is
> > >>>>>>>>>>>>>> no
> > >>>>>>>>>>>>>> change on this case, i.e. we resolve to keep "Operator"
> (and
> > >>>>>>>>>>>> "Sensor")
> > >>>>>>>>>>>>>> suffixes on class names
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Case 6 - *No change*
> > >>>>>>>>>>>>>> This case concerns object naming inconsistencies, but
> motion
> > >>>>>> does
> > >>>>>>>>>>>> not
> > >>>>>>>>>>>>>> enact
> > >>>>>>>>>>>>>> any specific proposal.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
> > >>>>>> deprecation
> > >>>>>>>>>>>> warnings
> > >>>>>>>>>>>>>> (instead of zope)
> > >>>>>>>>>>>>>> Example:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> # in old_package/old_mod.py
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> import warnings
> > >>>>>>>>>>>>>> from new_package.new_mod import *
> > >>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
> > >>>>>>>> new_package.new_mod.
> > >>>>>>>>>>>>>> Import
> > >>>>>>>>>>>>>> of "
> > >>>>>>>>>>>>>> "old_package.old_mod will become unsupported in version
> 2",
> > >>>>>>>>>>>>>> DeprecationWarning, 2)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Reference implementation here:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I am very happy that this motion now gets rid of contrib.
> > >>>>>> Thank
> > >>>>>>>> you
> > >>>>>>>>>>>>>> Sergio
> > >>>>>>>>>>>>>> for turning the tide with your effective argumentation ;)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
> > >>>>>>>> ash@apache.org>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Could someone summarise what the proposed name changes
> are,
> > >>>>>>>> here,
> > >>>>>>>>>>>> in
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving partial
> > >>>>>>>> examples
> > >>>>>>>>>>>> is 1)
> > >>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting on,
> and
> > >>>>>> 2)
> > >>>>>>>>>>>> using a
> > >>>>>>>>>>>>>>> google doc isn't great for a longer term record.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>> Ash
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
> > >>>>>>>>>>>> <Ja...@polidea.com>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :)
> . I
> > >>>>>>>> would
> > >>>>>>>>>>>> like
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three +1
> > >>>>>>>>>>>> (binding)
> > >>>>>>>>>>>>>> votes
> > >>>>>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to the
> > >>>>>>>>>>>> previous
> > >>>>>>>>>>>>>>> proposal.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Here is the proposal (details here
> > >>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> )
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
> > >>>>>>>>>>>>>>>> - *Case 2: A: Airflow.operators.foo_operator.FooOperator
> > >>>>>>>> could
> > >>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > >>>>>>>>>>>>>>>> - *Case 3: C:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > >>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> > >>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > >>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
> > >>>>>>>> class
> > >>>>>>>>>>>>>> names*
> > >>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
> > >>>>>>>> (and
> > >>>>>>>>>>>> my team
> > >>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > >>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
> > >>>>>>>>>>>> integration)*
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> This is my binding (+1) vote.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> > >>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hey Felix,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as
> well
> > >>>>>> (I
> > >>>>>>>>>>>> use IDE
> > >>>>>>>>>>>>>>> quite
> > >>>>>>>>>>>>>>>>> often ;) )
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> J.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> > >>>>>>>>>>>>>> <fokko@driesprong.frl
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I
> wasn't
> > >>>>>>>> able
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> respond
> > >>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops
> throughout
> > >>>>>>>> Europe
> > >>>>>>>>>>>> :-)
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from contrib
> > >>>>>> into
> > >>>>>>>> a
> > >>>>>>>>>>>> single
> > >>>>>>>>>>>>>>>>>> package.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference
> would be
> > >>>>>>>> to
> > >>>>>>>>>>>> get
> > >>>>>>>>>>>>>> rid of
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or not
> is a
> > >>>>>>>>>>>> tricky
> > >>>>>>>>>>>>>>> question,
> > >>>>>>>>>>>>>>>>>> I think this is already reflected in the document
> since
> > >>>>>>>>>>>> "maturity"
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the
> operator
> > >>>>>>>> to
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>> core
> > >>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
> > >>>>>> incubating
> > >>>>>>>>>>>> will
> > >>>>>>>>>>>>>> confuse
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> user even more. For people who aren't as known with
> > >>>>>>>> Airflow it
> > >>>>>>>>>>>> isn't
> > >>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>> transparent which operator lives where, especially if
> you
> > >>>>>>>>>>>> don't use
> > >>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>> IDE
> > >>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is worth
> > >>>>>>>> changing
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> public
> > >>>>>>>>>>>>>>>>>> API just for a different name.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with this.
> It
> > >>>>>>>> will
> > >>>>>>>>>>>> make
> > >>>>>>>>>>>>>>> easier
> > >>>>>>>>>>>>>>>>> as we will not have to define other rules as those
> that we
> > >>>>>>>>>>>> already
> > >>>>>>>>>>>>>> have.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's say a
> > >>>>>> +0)
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search for
> > >>>>>>>>>>>> BashOperator.
> > >>>>>>>>>>>>>>> After
> > >>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from your
> > >>>>>> mail
> > >>>>>>>> you
> > >>>>>>>>>>>>>> state: -
> > >>>>>>>>>>>>>>>>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator
> > >>>>>>>> could
> > >>>>>>>>>>>> become
> > >>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B
> solution is
> > >>>>>>>>>>>> keeping it
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
> > >>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would stay
> > >>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a
> little
> > >>>>>>>>>>>> confusion
> > >>>>>>>>>>>>>>> here.
> > >>>>>>>>>>>>>>>>> This case is not for changing BashOperator into Bash
> but
> > >>>>>> for
> > >>>>>>>>>>>> changing
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> *module* name not the *class* name. So
> > >>>>>>>>>>>> bash_operator.BashOperator ->
> > >>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for
> > >>>>>>>> BashOperator
> > >>>>>>>>>>>> in this
> > >>>>>>>>>>>>>>> case.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it
> gets
> > >>>>>>>> its
> > >>>>>>>>>>>> own
> > >>>>>>>>>>>>>>> package
> > >>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are
> companies
> > >>>>>>>> like
> > >>>>>>>>>>>>>>> Databricks
> > >>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own
> package
> > >>>>>>>>>>>> because
> > >>>>>>>>>>>>>> their
> > >>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving this
> is a
> > >>>>>>>> bit
> > >>>>>>>>>>>>>> painful
> > >>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move
> > >>>>>>>> everything
> > >>>>>>>>>>>> which
> > >>>>>>>>>>>>>>>>>> introduces noise into version control etc.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not as
> > >>>>>>>>>>>> difficult as
> > >>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>> seems to group operators in packages and it has
> numerous
> > >>>>>>>>>>>> advantages.
> > >>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in the
> > >>>>>>>>>>>> __init__.py of
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git will
> > >>>>>>>> actually
> > >>>>>>>>>>>>>> realise
> > >>>>>>>>>>>>>>>>> that the file has been moved and you can follow that.
> > >>>>>> Maybe
> > >>>>>>>> not
> > >>>>>>>>>>>> as
> > >>>>>>>>>>>>>> super
> > >>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's not a
> > >>>>>> rocket
> > >>>>>>>>>>>> science
> > >>>>>>>>>>>>>>> either.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Yes.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> ---
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Apart from all, great work!
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Cheers, Fokko
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> > >>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
> > >>>>>>>>>>>>>>>>>>> :
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change of
> > >>>>>> these
> > >>>>>>>>>>>> cases.
> > >>>>>>>>>>>>>> So I
> > >>>>>>>>>>>>>>>>>> go
> > >>>>>>>>>>>>>>>>>>> with +1 (non-binding).
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and your
> > >>>>>>>> intense
> > >>>>>>>>>>>> work
> > >>>>>>>>>>>>>>>>>> towards
> > >>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
> > >>>>>> creating
> > >>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>> document. I
> > >>>>>>>>>>>>>>>>>>> like how the code is getting more and more consistent
> > >>>>>> and
> > >>>>>>>>>>>> clean :)
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Kind regards,
> > >>>>>>>>>>>>>>>>>>> Felix
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> -------- Original Message --------
> > >>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in
> import
> > >>>>>>>> paths.
> > >>>>>>>>>>>> It's
> > >>>>>>>>>>>>>>> been
> > >>>>>>>>>>>>>>>>>>>> discussed in
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th
> 6pm
> > >>>>>>>> CEST),
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>>> at
> > >>>>>>>>>>>>>>>>>> least
> > >>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document from
> > >>>>>> Kamil
> > >>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> The proposed solution is:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are
> > >>>>>> "well"
> > >>>>>>>>>>>> tested
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
> > >>>>>>>>>>>>>>>>>>>> - *Case 2: B:
> > >>>>>> Airflow.operators.foo_operator.FooOperator
> > >>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > >>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > >>>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>>>>> become
> airflow.gcp.operators.bigtable.BigTableOperator*
> > >>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > >>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor")
> suffixes
> > >>>>>> on
> > >>>>>>>>>>>> class
> > >>>>>>>>>>>>>> names*
> > >>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on
> case-by-case
> > >>>>>>>> (and
> > >>>>>>>>>>>> my
> > >>>>>>>>>>>>>> team
> > >>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> J.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> > >>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> > >>>>>> Software
> > >>>>>>>>>>>> Engineer
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
> > >>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
> > >>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Jarek Potiuk
> > >>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> Software
> > >>>>>>>>>>>> Engineer
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > >>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jarek Potiuk
> > >>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> > >>>>>>>>> Engineer
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > >>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> --
> > >>>>>>>>>>>>> *Kaxil Naik*
> > >>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > >>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> > >>>>>> Spark
> > >>>>>>>> &
> > >>>>>>>>>>>> Neo4j
> > >>>>>>>>>>>>> Developer
> > >>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>> *Kaxil Naik*
> > >>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > >>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> > >>> Spark
> > >>>>>> &
> > >>>>>>>>>>>> Neo4j
> > >>>>>>>>>>>> Developer
> > >>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>>
> > >>>>>> Jarek Potiuk
> > >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>>>>>
> > >>>>>> M: +48 660 796 129 <+48660796129>
> > >>>>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>
> > >>> --
> > >>> *Kaxil Naik*
> > >>> *Big Data Consultant | DevOps Data Engineer*
> > >>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> Neo4j
> > >>> Developer
> > >>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >>>
> > >>
> > >>
> > >> --
> > >>
> > >> Jarek Potiuk
> > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>
> > >> M: +48 660 796 129 <+48660796129>
> > >> [image: Polidea] <https://www.polidea.com/>
> > >>
> > >>
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> >
>
Re: [VOTE] Changes in import paths
Posted by Kamil Breguła <ka...@polidea.com>.
Yes. All changes will be backwards compatible. In the case of using
the old path, a message containing a proposal for change will be
reported to the user.
I prepared an example of how to change the name of a class in a case
with the use of a native solution.
Source code: https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename
Preview from IDE: https://imgur.com/a/45NxS5W
On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> Just cos I'm not sure it's _explicitly_ stated, but all of the moves will have a deprecation of the old name right?
>
> 3+4 case D gets my vote too.
>
> -a
>
> > On 30 Jul 2019, at 09:58, Jarek Potiuk <Ja...@polidea.com> wrote:
> >
> > I went ahead and updated the page (just to speed it up) as I think it
> > really makes sense to join those two cases (and I do not see any drawbacks
> > - I think the options we have cover all possible approaches) and we can
> > always go back if we need to.
> >
> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> >
> > My vote is *D*.
> >
> > - airflow/contrib/operators/gcp_bigtable_operator.py → airflow/gcp
> > /operators/bigtable_operator.py
> > - airflow/contrib/operators/ssh_operator.py ->
> > airflow/operators/ssh_operator.py
> >
> > I updated the page with my vote.
> > I propose that we vote till Friday on that one (all the rest is already
> > agreed).
> >
> > J.
> >
> >
> > On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> >> I think almost everyone voted and we have almost perfect consensus. We all
> >> agree amongst other on moving all operators out of contrib (Great!).
> >>
> >> The only doubts are for *Case 3* (Cloud provider prefix) and *Case 4*
> >> (Using Namespaces).
> >> I think there was actually an overlap between those two. Also Ash's
> >> comment/veto on that is quite clear.
> >> So I have final (I hope!) proposal to join those two cases and have one
> >> voting (till Friday) only on that.
> >>
> >> I will update the doc if you all agree with me that makes more sense as
> >> single case to vote on:
> >>
> >> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
> >>
> >> *Option A*. Keep operators/sensors/hooks in airflow/operators(sensors,
> >> hooks) and keep/add prefixes in file names.
> >>
> >> -
> >> *airflow/contrib/operators/sns_publish_operator.py becomes
> >> airflow/operators/**aws_sns_publish_operator.py*
> >>
> >>
> >> - *airflow/contrib/operators/dataproc_operator.py*
> >> becomes *airflow/operators/gcp_dataproc_operator.py*
> >>
> >>
> >> -
> >> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/*
> >> *hooks/gcp_bigtable_hook.py*
> >> -
> >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >> *operators/ssh_operator.py*
> >>
> >>
> >>
> >> *Option B*. Group operators/sensors/hooks in airflow/operators(sensors,
> >> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
> >> airflow/operators(sensors/hooks), Drop the prefix.
> >>
> >> -
> >> *airflow/contrib/operators/sns_publish_operator.py
> >> becomes airflow/operators/**aws/sns_publish_operator.py*
> >>
> >>
> >> - *airflow/contrib/operators/dataproc_operator.py*
> >> becomes *airflow/operators/gcp/dataproc_operator.py*
> >>
> >>
> >> -
> >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
> >> *operators/gcp/bigtable_operator.py*
> >> -
> >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >> *operators/ssh_operator.py*
> >>
> >>
> >> *Option C*. Group operators/sensors/hooks in airflow/operators(sensors,
> >> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
> >> airflow/operators(sensors/hooks), Keep the prefix.
> >>
> >> -
> >> *airflow/contrib/operators/sns_publish_operator.py
> >> becomes airflow/operators/**aws/aws_sns_publish_operator.py*
> >>
> >>
> >> - *airflow/contrib/operators/dataproc_operator.py*
> >> becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
> >>
> >>
> >> -
> >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
> >> *operators/gcp/gcp_bigtable_operator.py*
> >> -
> >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >> *operators/ssh_operator.py*
> >>
> >>
> >> *Option D.* Group operators/sensors/hooks in *airflow/<PROVIDER>*/operators(sensors,
> >> hooks). Non-cloud provider ones are moved to
> >> airflow/operators(sensors/hooks). Drop the prefix.
> >>
> >> -
> >> *airflow/contrib/operators/sns_publish_operator.py
> >> becomes airflow/aws/operators/**sns_publish_operator.py*
> >>
> >>
> >> - *airflow/contrib/operators/dataproc_operator.py*
> >> becomes *airflow/gcp/operators/dataproc_operator.py*
> >>
> >>
> >> -
> >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
> >> *gcp/operators/bigtable_operator.py*
> >> -
> >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >> *operators/ssh_operator.py*
> >>
> >>
> >> *Option E.* Group operators/sensors/hooks in *airflow/<PROVIDER>*/operators(sensors,
> >> hooks). Non-cloud provider ones are moved to
> >> airflow/operators(sensors/hooks).* Keep the prefix*.
> >>
> >> -
> >> *airflow/contrib/operators/sns_publish_operator.py
> >> becomes airflow/aws/operators/aws_**sns_publish_operator.py*
> >>
> >>
> >> - *airflow/contrib/operators/dataproc_operator.py*
> >> becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
> >>
> >>
> >> -
> >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
> >> *gcp/operators/gcp_bigtable_operator.py*
> >> -
> >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> >> *operators/ssh_operator.py*
> >>
> >>
> >> Shall I proceed with updating the page ?
> >>
> >>
> >> J.
> >>
> >>
> >> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <ka...@gmail.com> wrote:
> >>
> >>> Thanks Jarek, adding my vote.
> >>>
> >>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> >>>
> >>>> Thanks Jarek, much clearer. I have voted there too.
> >>>>
> >>>> -ash
> >>>>
> >>>>
> >>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko <fo...@driesprong.frl>
> >>>> wrote:
> >>>>>
> >>>>> Thanks Jarek, the Wiki works much better for me. I've added my vote
> >>> there
> >>>>> as well.
> >>>>>
> >>>>> You've convinced me on the second case. I've changed my vote on Case 2
> >>>> from
> >>>>> A to B :-)
> >>>>>
> >>>>> Maybe we should leave the vote open a bit longer since it involves
> >>> quite
> >>>>> some reading, and I would like to get some proper feedback and
> >>> opinions
> >>>> on
> >>>>> this. Plus I only see two votes right now. If we don't think this
> >>>> through,
> >>>>> it might be that we're having this vote again in the near future :-)
> >>>>>
> >>>>> Cheers, Fokko
> >>>>>
> >>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
> >>>> Jarek.Potiuk@polidea.com>:
> >>>>>
> >>>>>> I updated the AIP-21 proposal in the way that should answer all the
> >>>>>> concerns in this thread.
> >>>>>>
> >>>>>> NOTE! There is an action for all of those who are interested: Please
> >>>>>> fill-in your voting in the voting table till Tuesday 30th of July 6pm
> >>>> CEST
> >>>>>> (5pm BST, 9 am PST).
> >>>>>>
> >>>>>> I created a summary of the proposal
> >>>>>> <
> >>>>>>
> >>>>
> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> >>>>>>>
> >>>>>> in table form - which contains also examples for each case.
> >>>>>>
> >>>>>> I added a voting table
> >>>>>> <
> >>>>>>
> >>>>
> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> >>>>>>>
> >>>>>> where you should add your votes:
> >>>>>>
> >>>>>> I propose this voting mechanism:
> >>>>>>
> >>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 pm
> >>> BST,
> >>>> 9am
> >>>>>> PST) .
> >>>>>>
> >>>>>> After the choice, the final consistent set of choices will be
> >>> announced
> >>>>>> (taking into account majority of binding vote, also including
> >>> potential
> >>>>>> vetos and consistency between the choices. Non-binding votes will be
> >>>> taken
> >>>>>> into account in case there is a draw. The final set of choices will
> >>> be
> >>>>>> announced at devlist thread
> >>>>>> <
> >>>>>>
> >>>>
> >>> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> >>>>>>>
> >>>>>> after
> >>>>>> the voting completes.
> >>>>>>
> >>>>>> Unless there is a veto raised to the final proposal, the final
> >>> proposal
> >>>> is
> >>>>>> accepted by Lazy Consensus
> >>>>>> <https://community.apache.org/committers/lazyConsensus.html> on
> >>>> *Friday*
> >>>>>> 02
> >>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
> >>>>>>
> >>>>>> Please let me know if what I proposed can be further improved to
> >>> avoid
> >>>>>> chaos.
> >>>>>>
> >>>>>> J.
> >>>>>>
> >>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
> >>> Jarek.Potiuk@polidea.com
> >>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Ok. I see that indeed voting could have been organised a bit better
> >>> -
> >>>> dev
> >>>>>>> list does not seem to cope well with such a complex case :).
> >>>>>>>
> >>>>>>> As Kamil noticed - we already have a formal AIP-21
> >>>>>>> <
> >>>>>>
> >>>>
> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >>>>>>>
> >>>>>>> in the Wiki - which I should have mentioned before. I have not much
> >>>> time
> >>>>>>> today - but tomorrow I will try to summarise the options - based on
> >>>> some
> >>>>>>> real code examples (to make it easier to digest). I think the best
> >>>>>> approach
> >>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where people
> >>>> will
> >>>>>>> be able to state their preferences for each case (by adding a row to
> >>>> the
> >>>>>>> table). I think that might provide much better clarity and a single
> >>>> place
> >>>>>>> which will show the current aggregated state of voting.
> >>>>>>>
> >>>>>>> However - as Ash also stated - some of the options are
> >>> inter-dependent
> >>>> so
> >>>>>>> at the end we will have to adjust the results in case they are not
> >>>>>>> consistent - and make final voting on aggregated set of cases -
> >>> but I
> >>>>>>> think in this case following "lasy consensus" - i,e. if noone
> >>> objects
> >>>> to
> >>>>>>> final set of cases, we will proceed with it..
> >>>>>>>
> >>>>>>> Do you think that will work better ?
> >>>>>>>
> >>>>>>> J.
> >>>>>>>
> >>>>>>> Principal Software Engineer
> >>>>>>> Phone: +48660796129
> >>>>>>>
> >>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> >>>>>>> kamil.bregula@polidea.com> napisał:
> >>>>>>>
> >>>>>>>> Document is also available as AIP on wiki:
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >>>>>>>>
> >>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
> >>> <fokko@driesprong.frl
> >>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I agree with all, this is a bit chaotic. For the sake of clarity,
> >>> I
> >>>>>>>> would
> >>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a
> >>>>>> structural
> >>>>>>>>> code improvement AIP and then and add a matrix of users/cases
> >>> where
> >>>>>>>>> everybody can add his vote per case. This gives also the
> >>> opportunity
> >>>>>> for
> >>>>>>>>> changing your vote on a specific case. WDYT?
> >>>>>>>>>
> >>>>>>>>> Cheers, Fokko
> >>>>>>>>>
> >>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <ci...@gmail.com>:
> >>>>>>>>>
> >>>>>>>>>> I agree with Ash. It is clear to vote on each case rather than
> >>> all
> >>>>>>>>>> together.
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
> >>> ash@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> A number of proposals overlap or add on to each other, so I
> >>> think
> >>>>>>>> (?) a
> >>>>>>>>>>> single vote makes sense.
> >>>>>>>>>>>
> >>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what we are
> >>>>>>>> voting
> >>>>>>>>> on.
> >>>>>>>>>>>
> >>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <ka...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases together,
> >>>>>> how
> >>>>>>>>>>>> about
> >>>>>>>>>>>> separate vote emails for all the cases? Each email can have the
> >>>>>>>>>>>> description
> >>>>>>>>>>>> copied over from the doc.
> >>>>>>>>>>>>
> >>>>>>>>>>>> It would probably be a bit easier to track a decision in the
> >>>>>> future
> >>>>>>>> as
> >>>>>>>>>>>> well.
> >>>>>>>>>>>>
> >>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
> >>> Jarek.Potiuk@polidea.com>
> >>>>>>>>>>>> @Driesprong,
> >>>>>>>>>>>> Fokko <fo...@driesprong.frl>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
> >>> kaxilnaik@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Case #4* Separate namespace for resources :
> >>>>>>>>>>>>> *Solution A*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Case #7* : *Solution A*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> >>>>>>>>>>>> <dp...@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I have tried to add some clarification to Jarek's summary,
> >>> but
> >>>>>> I
> >>>>>>>> am
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully Jarek
> >>>>>> can
> >>>>>>>>>>>> further
> >>>>>>>>>>>>>> clarify.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6 are all
> >>>>>>>> either
> >>>>>>>>>>>>>> proposal
> >>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention can be
> >>>>>>>>>>>> focused on
> >>>>>>>>>>>>>> cases 1,2,3 and 7.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Motion is to adopt the following:*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
> >>>>>>>>>>>>>> All objects will be moved out of contrib folder and into
> >>>>>>>> respective
> >>>>>>>>>>>>>> non-contrib folder.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from modules
> >>>>>>>>>>>>>> Example:
> >>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
> >>>>>>>>>>>>>> airflow.operators.foo.FooOperator
> >>>>>>>>>>>>>> Example:
> >>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
> >>>>>>>> airflow.hooks.foo.FooHook
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
> >>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
> >>>>>>>>>>>>>> Example case is
> >>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
> >>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume it is
> >>> now
> >>>>>>>> named
> >>>>>>>>>>>> like
> >>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
> >>>>>>>>>>>>>> So what is proposed for this case?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 4 - Solution B - *no change*
> >>>>>>>>>>>>>> This proposal was about namespaces but the motion is to
> >>> reject
> >>>>>>>> the
> >>>>>>>>>>>>>> proposal.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 5 - Solution B - *no change*
> >>>>>>>>>>>>>> The proposal was to remove class type suffix e.g. remove the
> >>>>>>>>>>>> Operator in
> >>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal --
> >>> i.e.
> >>>>>>>>>>>> motion is
> >>>>>>>>>>>>>> no
> >>>>>>>>>>>>>> change on this case, i.e. we resolve to keep "Operator" (and
> >>>>>>>>>>>> "Sensor")
> >>>>>>>>>>>>>> suffixes on class names
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 6 - *No change*
> >>>>>>>>>>>>>> This case concerns object naming inconsistencies, but motion
> >>>>>> does
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>> enact
> >>>>>>>>>>>>>> any specific proposal.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
> >>>>>> deprecation
> >>>>>>>>>>>> warnings
> >>>>>>>>>>>>>> (instead of zope)
> >>>>>>>>>>>>>> Example:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> # in old_package/old_mod.py
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> import warnings
> >>>>>>>>>>>>>> from new_package.new_mod import *
> >>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
> >>>>>>>> new_package.new_mod.
> >>>>>>>>>>>>>> Import
> >>>>>>>>>>>>>> of "
> >>>>>>>>>>>>>> "old_package.old_mod will become unsupported in version 2",
> >>>>>>>>>>>>>> DeprecationWarning, 2)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Reference implementation here:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I am very happy that this motion now gets rid of contrib.
> >>>>>> Thank
> >>>>>>>> you
> >>>>>>>>>>>>>> Sergio
> >>>>>>>>>>>>>> for turning the tide with your effective argumentation ;)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
> >>>>>>>> ash@apache.org>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Could someone summarise what the proposed name changes are,
> >>>>>>>> here,
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving partial
> >>>>>>>> examples
> >>>>>>>>>>>> is 1)
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting on, and
> >>>>>> 2)
> >>>>>>>>>>>> using a
> >>>>>>>>>>>>>>> google doc isn't great for a longer term record.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Ash
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
> >>>>>>>>>>>> <Ja...@polidea.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :) . I
> >>>>>>>> would
> >>>>>>>>>>>> like
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three +1
> >>>>>>>>>>>> (binding)
> >>>>>>>>>>>>>> votes
> >>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to the
> >>>>>>>>>>>> previous
> >>>>>>>>>>>>>>> proposal.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Here is the proposal (details here
> >>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
> >>>>>>>>>>>>>>>> - *Case 2: A: Airflow.operators.foo_operator.FooOperator
> >>>>>>>> could
> >>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> >>>>>>>>>>>>>>>> - *Case 3: C:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> >>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> >>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
> >>>>>>>> class
> >>>>>>>>>>>>>> names*
> >>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
> >>>>>>>> (and
> >>>>>>>>>>>> my team
> >>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> >>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
> >>>>>>>>>>>> integration)*
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This is my binding (+1) vote.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> >>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hey Felix,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as well
> >>>>>> (I
> >>>>>>>>>>>> use IDE
> >>>>>>>>>>>>>>> quite
> >>>>>>>>>>>>>>>>> often ;) )
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> J.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> >>>>>>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I wasn't
> >>>>>>>> able
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> respond
> >>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops throughout
> >>>>>>>> Europe
> >>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from contrib
> >>>>>> into
> >>>>>>>> a
> >>>>>>>>>>>> single
> >>>>>>>>>>>>>>>>>> package.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference would be
> >>>>>>>> to
> >>>>>>>>>>>> get
> >>>>>>>>>>>>>> rid of
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or not is a
> >>>>>>>>>>>> tricky
> >>>>>>>>>>>>>>> question,
> >>>>>>>>>>>>>>>>>> I think this is already reflected in the document since
> >>>>>>>>>>>> "maturity"
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the operator
> >>>>>>>> to
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> core
> >>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
> >>>>>> incubating
> >>>>>>>>>>>> will
> >>>>>>>>>>>>>> confuse
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> user even more. For people who aren't as known with
> >>>>>>>> Airflow it
> >>>>>>>>>>>> isn't
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> transparent which operator lives where, especially if you
> >>>>>>>>>>>> don't use
> >>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>> IDE
> >>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is worth
> >>>>>>>> changing
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> public
> >>>>>>>>>>>>>>>>>> API just for a different name.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with this. It
> >>>>>>>> will
> >>>>>>>>>>>> make
> >>>>>>>>>>>>>>> easier
> >>>>>>>>>>>>>>>>> as we will not have to define other rules as those that we
> >>>>>>>>>>>> already
> >>>>>>>>>>>>>> have.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's say a
> >>>>>> +0)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search for
> >>>>>>>>>>>> BashOperator.
> >>>>>>>>>>>>>>> After
> >>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from your
> >>>>>> mail
> >>>>>>>> you
> >>>>>>>>>>>>>> state: -
> >>>>>>>>>>>>>>>>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator
> >>>>>>>> could
> >>>>>>>>>>>> become
> >>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B solution is
> >>>>>>>>>>>> keeping it
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
> >>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would stay
> >>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a little
> >>>>>>>>>>>> confusion
> >>>>>>>>>>>>>>> here.
> >>>>>>>>>>>>>>>>> This case is not for changing BashOperator into Bash but
> >>>>>> for
> >>>>>>>>>>>> changing
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> *module* name not the *class* name. So
> >>>>>>>>>>>> bash_operator.BashOperator ->
> >>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for
> >>>>>>>> BashOperator
> >>>>>>>>>>>> in this
> >>>>>>>>>>>>>>> case.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it gets
> >>>>>>>> its
> >>>>>>>>>>>> own
> >>>>>>>>>>>>>>> package
> >>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are companies
> >>>>>>>> like
> >>>>>>>>>>>>>>> Databricks
> >>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own package
> >>>>>>>>>>>> because
> >>>>>>>>>>>>>> their
> >>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving this is a
> >>>>>>>> bit
> >>>>>>>>>>>>>> painful
> >>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move
> >>>>>>>> everything
> >>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>> introduces noise into version control etc.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not as
> >>>>>>>>>>>> difficult as
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>> seems to group operators in packages and it has numerous
> >>>>>>>>>>>> advantages.
> >>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in the
> >>>>>>>>>>>> __init__.py of
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git will
> >>>>>>>> actually
> >>>>>>>>>>>>>> realise
> >>>>>>>>>>>>>>>>> that the file has been moved and you can follow that.
> >>>>>> Maybe
> >>>>>>>> not
> >>>>>>>>>>>> as
> >>>>>>>>>>>>>> super
> >>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's not a
> >>>>>> rocket
> >>>>>>>>>>>> science
> >>>>>>>>>>>>>>> either.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Yes.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Apart from all, great work!
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> >>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
> >>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change of
> >>>>>> these
> >>>>>>>>>>>> cases.
> >>>>>>>>>>>>>> So I
> >>>>>>>>>>>>>>>>>> go
> >>>>>>>>>>>>>>>>>>> with +1 (non-binding).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and your
> >>>>>>>> intense
> >>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>>> towards
> >>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
> >>>>>> creating
> >>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> document. I
> >>>>>>>>>>>>>>>>>>> like how the code is getting more and more consistent
> >>>>>> and
> >>>>>>>>>>>> clean :)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Kind regards,
> >>>>>>>>>>>>>>>>>>> Felix
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -------- Original Message --------
> >>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in import
> >>>>>>>> paths.
> >>>>>>>>>>>> It's
> >>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>> discussed in
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th 6pm
> >>>>>>>> CEST),
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>> least
> >>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document from
> >>>>>> Kamil
> >>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The proposed solution is:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are
> >>>>>> "well"
> >>>>>>>>>>>> tested
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
> >>>>>>>>>>>>>>>>>>>> - *Case 2: B:
> >>>>>> Airflow.operators.foo_operator.FooOperator
> >>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> >>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> >>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> >>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes
> >>>>>> on
> >>>>>>>>>>>> class
> >>>>>>>>>>>>>> names*
> >>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
> >>>>>>>> (and
> >>>>>>>>>>>> my
> >>>>>>>>>>>>>> team
> >>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> J.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> >>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> >>>>>> Software
> >>>>>>>>>>>> Engineer
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
> >>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
> >>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Jarek Potiuk
> >>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> >>>>>>>>>>>> Engineer
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jarek Potiuk
> >>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> >>>>>>>>> Engineer
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> *Kaxil Naik*
> >>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> >>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> >>>>>> Spark
> >>>>>>>> &
> >>>>>>>>>>>> Neo4j
> >>>>>>>>>>>>> Developer
> >>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> *Kaxil Naik*
> >>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> >>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> >>> Spark
> >>>>>> &
> >>>>>>>>>>>> Neo4j
> >>>>>>>>>>>> Developer
> >>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Jarek Potiuk
> >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>>>>
> >>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>> --
> >>> *Kaxil Naik*
> >>> *Big Data Consultant | DevOps Data Engineer*
> >>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> >>> Developer
> >>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>>
> >>
> >>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>
> >> M: +48 660 796 129 <+48660796129>
> >> [image: Polidea] <https://www.polidea.com/>
> >>
> >>
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
>
Re: [VOTE] Changes in import paths
Posted by Ash Berlin-Taylor <as...@apache.org>.
Just cos I'm not sure it's _explicitly_ stated, but all of the moves will have a deprecation of the old name right?
3+4 case D gets my vote too.
-a
> On 30 Jul 2019, at 09:58, Jarek Potiuk <Ja...@polidea.com> wrote:
>
> I went ahead and updated the page (just to speed it up) as I think it
> really makes sense to join those two cases (and I do not see any drawbacks
> - I think the options we have cover all possible approaches) and we can
> always go back if we need to.
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
>
> My vote is *D*.
>
> - airflow/contrib/operators/gcp_bigtable_operator.py → airflow/gcp
> /operators/bigtable_operator.py
> - airflow/contrib/operators/ssh_operator.py ->
> airflow/operators/ssh_operator.py
>
> I updated the page with my vote.
> I propose that we vote till Friday on that one (all the rest is already
> agreed).
>
> J.
>
>
> On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> I think almost everyone voted and we have almost perfect consensus. We all
>> agree amongst other on moving all operators out of contrib (Great!).
>>
>> The only doubts are for *Case 3* (Cloud provider prefix) and *Case 4*
>> (Using Namespaces).
>> I think there was actually an overlap between those two. Also Ash's
>> comment/veto on that is quite clear.
>> So I have final (I hope!) proposal to join those two cases and have one
>> voting (till Friday) only on that.
>>
>> I will update the doc if you all agree with me that makes more sense as
>> single case to vote on:
>>
>> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
>>
>> *Option A*. Keep operators/sensors/hooks in airflow/operators(sensors,
>> hooks) and keep/add prefixes in file names.
>>
>> -
>> *airflow/contrib/operators/sns_publish_operator.py becomes
>> airflow/operators/**aws_sns_publish_operator.py*
>>
>>
>> - *airflow/contrib/operators/dataproc_operator.py*
>> becomes *airflow/operators/gcp_dataproc_operator.py*
>>
>>
>> -
>> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/*
>> *hooks/gcp_bigtable_hook.py*
>> -
>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>> *operators/ssh_operator.py*
>>
>>
>>
>> *Option B*. Group operators/sensors/hooks in airflow/operators(sensors,
>> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
>> airflow/operators(sensors/hooks), Drop the prefix.
>>
>> -
>> *airflow/contrib/operators/sns_publish_operator.py
>> becomes airflow/operators/**aws/sns_publish_operator.py*
>>
>>
>> - *airflow/contrib/operators/dataproc_operator.py*
>> becomes *airflow/operators/gcp/dataproc_operator.py*
>>
>>
>> -
>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
>> *operators/gcp/bigtable_operator.py*
>> -
>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>> *operators/ssh_operator.py*
>>
>>
>> *Option C*. Group operators/sensors/hooks in airflow/operators(sensors,
>> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
>> airflow/operators(sensors/hooks), Keep the prefix.
>>
>> -
>> *airflow/contrib/operators/sns_publish_operator.py
>> becomes airflow/operators/**aws/aws_sns_publish_operator.py*
>>
>>
>> - *airflow/contrib/operators/dataproc_operator.py*
>> becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
>>
>>
>> -
>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
>> *operators/gcp/gcp_bigtable_operator.py*
>> -
>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>> *operators/ssh_operator.py*
>>
>>
>> *Option D.* Group operators/sensors/hooks in *airflow/<PROVIDER>*/operators(sensors,
>> hooks). Non-cloud provider ones are moved to
>> airflow/operators(sensors/hooks). Drop the prefix.
>>
>> -
>> *airflow/contrib/operators/sns_publish_operator.py
>> becomes airflow/aws/operators/**sns_publish_operator.py*
>>
>>
>> - *airflow/contrib/operators/dataproc_operator.py*
>> becomes *airflow/gcp/operators/dataproc_operator.py*
>>
>>
>> -
>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
>> *gcp/operators/bigtable_operator.py*
>> -
>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>> *operators/ssh_operator.py*
>>
>>
>> *Option E.* Group operators/sensors/hooks in *airflow/<PROVIDER>*/operators(sensors,
>> hooks). Non-cloud provider ones are moved to
>> airflow/operators(sensors/hooks).* Keep the prefix*.
>>
>> -
>> *airflow/contrib/operators/sns_publish_operator.py
>> becomes airflow/aws/operators/aws_**sns_publish_operator.py*
>>
>>
>> - *airflow/contrib/operators/dataproc_operator.py*
>> becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
>>
>>
>> -
>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
>> *gcp/operators/gcp_bigtable_operator.py*
>> -
>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
>> *operators/ssh_operator.py*
>>
>>
>> Shall I proceed with updating the page ?
>>
>>
>> J.
>>
>>
>> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <ka...@gmail.com> wrote:
>>
>>> Thanks Jarek, adding my vote.
>>>
>>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>>>
>>>> Thanks Jarek, much clearer. I have voted there too.
>>>>
>>>> -ash
>>>>
>>>>
>>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko <fo...@driesprong.frl>
>>>> wrote:
>>>>>
>>>>> Thanks Jarek, the Wiki works much better for me. I've added my vote
>>> there
>>>>> as well.
>>>>>
>>>>> You've convinced me on the second case. I've changed my vote on Case 2
>>>> from
>>>>> A to B :-)
>>>>>
>>>>> Maybe we should leave the vote open a bit longer since it involves
>>> quite
>>>>> some reading, and I would like to get some proper feedback and
>>> opinions
>>>> on
>>>>> this. Plus I only see two votes right now. If we don't think this
>>>> through,
>>>>> it might be that we're having this vote again in the near future :-)
>>>>>
>>>>> Cheers, Fokko
>>>>>
>>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
>>>> Jarek.Potiuk@polidea.com>:
>>>>>
>>>>>> I updated the AIP-21 proposal in the way that should answer all the
>>>>>> concerns in this thread.
>>>>>>
>>>>>> NOTE! There is an action for all of those who are interested: Please
>>>>>> fill-in your voting in the voting table till Tuesday 30th of July 6pm
>>>> CEST
>>>>>> (5pm BST, 9 am PST).
>>>>>>
>>>>>> I created a summary of the proposal
>>>>>> <
>>>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
>>>>>>>
>>>>>> in table form - which contains also examples for each case.
>>>>>>
>>>>>> I added a voting table
>>>>>> <
>>>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
>>>>>>>
>>>>>> where you should add your votes:
>>>>>>
>>>>>> I propose this voting mechanism:
>>>>>>
>>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 pm
>>> BST,
>>>> 9am
>>>>>> PST) .
>>>>>>
>>>>>> After the choice, the final consistent set of choices will be
>>> announced
>>>>>> (taking into account majority of binding vote, also including
>>> potential
>>>>>> vetos and consistency between the choices. Non-binding votes will be
>>>> taken
>>>>>> into account in case there is a draw. The final set of choices will
>>> be
>>>>>> announced at devlist thread
>>>>>> <
>>>>>>
>>>>
>>> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
>>>>>>>
>>>>>> after
>>>>>> the voting completes.
>>>>>>
>>>>>> Unless there is a veto raised to the final proposal, the final
>>> proposal
>>>> is
>>>>>> accepted by Lazy Consensus
>>>>>> <https://community.apache.org/committers/lazyConsensus.html> on
>>>> *Friday*
>>>>>> 02
>>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
>>>>>>
>>>>>> Please let me know if what I proposed can be further improved to
>>> avoid
>>>>>> chaos.
>>>>>>
>>>>>> J.
>>>>>>
>>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
>>> Jarek.Potiuk@polidea.com
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>> Ok. I see that indeed voting could have been organised a bit better
>>> -
>>>> dev
>>>>>>> list does not seem to cope well with such a complex case :).
>>>>>>>
>>>>>>> As Kamil noticed - we already have a formal AIP-21
>>>>>>> <
>>>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>>>>>>>
>>>>>>> in the Wiki - which I should have mentioned before. I have not much
>>>> time
>>>>>>> today - but tomorrow I will try to summarise the options - based on
>>>> some
>>>>>>> real code examples (to make it easier to digest). I think the best
>>>>>> approach
>>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where people
>>>> will
>>>>>>> be able to state their preferences for each case (by adding a row to
>>>> the
>>>>>>> table). I think that might provide much better clarity and a single
>>>> place
>>>>>>> which will show the current aggregated state of voting.
>>>>>>>
>>>>>>> However - as Ash also stated - some of the options are
>>> inter-dependent
>>>> so
>>>>>>> at the end we will have to adjust the results in case they are not
>>>>>>> consistent - and make final voting on aggregated set of cases -
>>> but I
>>>>>>> think in this case following "lasy consensus" - i,e. if noone
>>> objects
>>>> to
>>>>>>> final set of cases, we will proceed with it..
>>>>>>>
>>>>>>> Do you think that will work better ?
>>>>>>>
>>>>>>> J.
>>>>>>>
>>>>>>> Principal Software Engineer
>>>>>>> Phone: +48660796129
>>>>>>>
>>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
>>>>>>> kamil.bregula@polidea.com> napisał:
>>>>>>>
>>>>>>>> Document is also available as AIP on wiki:
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>>>>>>>>
>>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
>>> <fokko@driesprong.frl
>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I agree with all, this is a bit chaotic. For the sake of clarity,
>>> I
>>>>>>>> would
>>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a
>>>>>> structural
>>>>>>>>> code improvement AIP and then and add a matrix of users/cases
>>> where
>>>>>>>>> everybody can add his vote per case. This gives also the
>>> opportunity
>>>>>> for
>>>>>>>>> changing your vote on a specific case. WDYT?
>>>>>>>>>
>>>>>>>>> Cheers, Fokko
>>>>>>>>>
>>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <ci...@gmail.com>:
>>>>>>>>>
>>>>>>>>>> I agree with Ash. It is clear to vote on each case rather than
>>> all
>>>>>>>>>> together.
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
>>> ash@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> A number of proposals overlap or add on to each other, so I
>>> think
>>>>>>>> (?) a
>>>>>>>>>>> single vote makes sense.
>>>>>>>>>>>
>>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what we are
>>>>>>>> voting
>>>>>>>>> on.
>>>>>>>>>>>
>>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <ka...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases together,
>>>>>> how
>>>>>>>>>>>> about
>>>>>>>>>>>> separate vote emails for all the cases? Each email can have the
>>>>>>>>>>>> description
>>>>>>>>>>>> copied over from the doc.
>>>>>>>>>>>>
>>>>>>>>>>>> It would probably be a bit easier to track a decision in the
>>>>>> future
>>>>>>>> as
>>>>>>>>>>>> well.
>>>>>>>>>>>>
>>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
>>> Jarek.Potiuk@polidea.com>
>>>>>>>>>>>> @Driesprong,
>>>>>>>>>>>> Fokko <fo...@driesprong.frl>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
>>> kaxilnaik@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Case #4* Separate namespace for resources :
>>>>>>>>>>>>> *Solution A*
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Case #7* : *Solution A*
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
>>>>>>>>>>>> <dp...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have tried to add some clarification to Jarek's summary,
>>> but
>>>>>> I
>>>>>>>> am
>>>>>>>>>>>> a
>>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully Jarek
>>>>>> can
>>>>>>>>>>>> further
>>>>>>>>>>>>>> clarify.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6 are all
>>>>>>>> either
>>>>>>>>>>>>>> proposal
>>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention can be
>>>>>>>>>>>> focused on
>>>>>>>>>>>>>> cases 1,2,3 and 7.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Motion is to adopt the following:*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
>>>>>>>>>>>>>> All objects will be moved out of contrib folder and into
>>>>>>>> respective
>>>>>>>>>>>>>> non-contrib folder.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from modules
>>>>>>>>>>>>>> Example:
>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
>>>>>>>>>>>>>> airflow.operators.foo.FooOperator
>>>>>>>>>>>>>> Example:
>>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
>>>>>>>> airflow.hooks.foo.FooHook
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
>>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
>>>>>>>>>>>>>> Example case is
>>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
>>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume it is
>>> now
>>>>>>>> named
>>>>>>>>>>>> like
>>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
>>>>>>>>>>>>>> So what is proposed for this case?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Case 4 - Solution B - *no change*
>>>>>>>>>>>>>> This proposal was about namespaces but the motion is to
>>> reject
>>>>>>>> the
>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Case 5 - Solution B - *no change*
>>>>>>>>>>>>>> The proposal was to remove class type suffix e.g. remove the
>>>>>>>>>>>> Operator in
>>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal --
>>> i.e.
>>>>>>>>>>>> motion is
>>>>>>>>>>>>>> no
>>>>>>>>>>>>>> change on this case, i.e. we resolve to keep "Operator" (and
>>>>>>>>>>>> "Sensor")
>>>>>>>>>>>>>> suffixes on class names
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Case 6 - *No change*
>>>>>>>>>>>>>> This case concerns object naming inconsistencies, but motion
>>>>>> does
>>>>>>>>>>>> not
>>>>>>>>>>>>>> enact
>>>>>>>>>>>>>> any specific proposal.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
>>>>>> deprecation
>>>>>>>>>>>> warnings
>>>>>>>>>>>>>> (instead of zope)
>>>>>>>>>>>>>> Example:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # in old_package/old_mod.py
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> import warnings
>>>>>>>>>>>>>> from new_package.new_mod import *
>>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
>>>>>>>> new_package.new_mod.
>>>>>>>>>>>>>> Import
>>>>>>>>>>>>>> of "
>>>>>>>>>>>>>> "old_package.old_mod will become unsupported in version 2",
>>>>>>>>>>>>>> DeprecationWarning, 2)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Reference implementation here:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am very happy that this motion now gets rid of contrib.
>>>>>> Thank
>>>>>>>> you
>>>>>>>>>>>>>> Sergio
>>>>>>>>>>>>>> for turning the tide with your effective argumentation ;)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
>>>>>>>> ash@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Could someone summarise what the proposed name changes are,
>>>>>>>> here,
>>>>>>>>>>>> in
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving partial
>>>>>>>> examples
>>>>>>>>>>>> is 1)
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting on, and
>>>>>> 2)
>>>>>>>>>>>> using a
>>>>>>>>>>>>>>> google doc isn't great for a longer term record.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Ash
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
>>>>>>>>>>>> <Ja...@polidea.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :) . I
>>>>>>>> would
>>>>>>>>>>>> like
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three +1
>>>>>>>>>>>> (binding)
>>>>>>>>>>>>>> votes
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to the
>>>>>>>>>>>> previous
>>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here is the proposal (details here
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
>>>>>>>>>>>>>>>> - *Case 2: A: Airflow.operators.foo_operator.FooOperator
>>>>>>>> could
>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
>>>>>>>>>>>>>>>> - *Case 3: C:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
>>>>>>>> class
>>>>>>>>>>>>>> names*
>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
>>>>>>>> (and
>>>>>>>>>>>> my team
>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
>>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
>>>>>>>>>>>> integration)*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is my binding (+1) vote.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
>>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hey Felix,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as well
>>>>>> (I
>>>>>>>>>>>> use IDE
>>>>>>>>>>>>>>> quite
>>>>>>>>>>>>>>>>> often ;) )
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
>>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I wasn't
>>>>>>>> able
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> respond
>>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops throughout
>>>>>>>> Europe
>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from contrib
>>>>>> into
>>>>>>>> a
>>>>>>>>>>>> single
>>>>>>>>>>>>>>>>>> package.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference would be
>>>>>>>> to
>>>>>>>>>>>> get
>>>>>>>>>>>>>> rid of
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or not is a
>>>>>>>>>>>> tricky
>>>>>>>>>>>>>>> question,
>>>>>>>>>>>>>>>>>> I think this is already reflected in the document since
>>>>>>>>>>>> "maturity"
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the operator
>>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>>>> core
>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
>>>>>> incubating
>>>>>>>>>>>> will
>>>>>>>>>>>>>> confuse
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> user even more. For people who aren't as known with
>>>>>>>> Airflow it
>>>>>>>>>>>> isn't
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> transparent which operator lives where, especially if you
>>>>>>>>>>>> don't use
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>> IDE
>>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is worth
>>>>>>>> changing
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>>>> API just for a different name.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with this. It
>>>>>>>> will
>>>>>>>>>>>> make
>>>>>>>>>>>>>>> easier
>>>>>>>>>>>>>>>>> as we will not have to define other rules as those that we
>>>>>>>>>>>> already
>>>>>>>>>>>>>> have.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's say a
>>>>>> +0)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search for
>>>>>>>>>>>> BashOperator.
>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from your
>>>>>> mail
>>>>>>>> you
>>>>>>>>>>>>>> state: -
>>>>>>>>>>>>>>>>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator
>>>>>>>> could
>>>>>>>>>>>> become
>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B solution is
>>>>>>>>>>>> keeping it
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would stay
>>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a little
>>>>>>>>>>>> confusion
>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>> This case is not for changing BashOperator into Bash but
>>>>>> for
>>>>>>>>>>>> changing
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> *module* name not the *class* name. So
>>>>>>>>>>>> bash_operator.BashOperator ->
>>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for
>>>>>>>> BashOperator
>>>>>>>>>>>> in this
>>>>>>>>>>>>>>> case.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it gets
>>>>>>>> its
>>>>>>>>>>>> own
>>>>>>>>>>>>>>> package
>>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are companies
>>>>>>>> like
>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own package
>>>>>>>>>>>> because
>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving this is a
>>>>>>>> bit
>>>>>>>>>>>>>> painful
>>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move
>>>>>>>> everything
>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>> introduces noise into version control etc.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not as
>>>>>>>>>>>> difficult as
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> seems to group operators in packages and it has numerous
>>>>>>>>>>>> advantages.
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in the
>>>>>>>>>>>> __init__.py of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git will
>>>>>>>> actually
>>>>>>>>>>>>>> realise
>>>>>>>>>>>>>>>>> that the file has been moved and you can follow that.
>>>>>> Maybe
>>>>>>>> not
>>>>>>>>>>>> as
>>>>>>>>>>>>>> super
>>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's not a
>>>>>> rocket
>>>>>>>>>>>> science
>>>>>>>>>>>>>>> either.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Apart from all, great work!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
>>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change of
>>>>>> these
>>>>>>>>>>>> cases.
>>>>>>>>>>>>>> So I
>>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>> with +1 (non-binding).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and your
>>>>>>>> intense
>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>> towards
>>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
>>>>>> creating
>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> document. I
>>>>>>>>>>>>>>>>>>> like how the code is getting more and more consistent
>>>>>> and
>>>>>>>>>>>> clean :)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>> Felix
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -------- Original Message --------
>>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in import
>>>>>>>> paths.
>>>>>>>>>>>> It's
>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>> discussed in
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th 6pm
>>>>>>>> CEST),
>>>>>>>>>>>> and
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> least
>>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document from
>>>>>> Kamil
>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The proposed solution is:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are
>>>>>> "well"
>>>>>>>>>>>> tested
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
>>>>>>>>>>>>>>>>>>>> - *Case 2: B:
>>>>>> Airflow.operators.foo_operator.FooOperator
>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes
>>>>>> on
>>>>>>>>>>>> class
>>>>>>>>>>>>>> names*
>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
>>>>>>>> (and
>>>>>>>>>>>> my
>>>>>>>>>>>>>> team
>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
>>>>>> Software
>>>>>>>>>>>> Engineer
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
>>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
>>>>>>>>>>>> Engineer
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
>>>>>>>>> Engineer
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> *Kaxil Naik*
>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
>>>>>> Spark
>>>>>>>> &
>>>>>>>>>>>> Neo4j
>>>>>>>>>>>>> Developer
>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> *Kaxil Naik*
>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
>>> Spark
>>>>>> &
>>>>>>>>>>>> Neo4j
>>>>>>>>>>>> Developer
>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jarek Potiuk
>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>
>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>
>>>>
>>>>
>>>
>>> --
>>> *Kaxil Naik*
>>> *Big Data Consultant | DevOps Data Engineer*
>>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
>>> Developer
>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Changes in import paths
Posted by Jarek Potiuk <Ja...@polidea.com>.
I went ahead and updated the page (just to speed it up) as I think it
really makes sense to join those two cases (and I do not see any drawbacks
- I think the options we have cover all possible approaches) and we can
always go back if we need to.
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
My vote is *D*.
- airflow/contrib/operators/gcp_bigtable_operator.py → airflow/gcp
/operators/bigtable_operator.py
- airflow/contrib/operators/ssh_operator.py ->
airflow/operators/ssh_operator.py
I updated the page with my vote.
I propose that we vote till Friday on that one (all the rest is already
agreed).
J.
On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I think almost everyone voted and we have almost perfect consensus. We all
> agree amongst other on moving all operators out of contrib (Great!).
>
> The only doubts are for *Case 3* (Cloud provider prefix) and *Case 4*
> (Using Namespaces).
> I think there was actually an overlap between those two. Also Ash's
> comment/veto on that is quite clear.
> So I have final (I hope!) proposal to join those two cases and have one
> voting (till Friday) only on that.
>
> I will update the doc if you all agree with me that makes more sense as
> single case to vote on:
>
> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
>
> *Option A*. Keep operators/sensors/hooks in airflow/operators(sensors,
> hooks) and keep/add prefixes in file names.
>
> -
> *airflow/contrib/operators/sns_publish_operator.py becomes
> airflow/operators/**aws_sns_publish_operator.py*
>
>
> - *airflow/contrib/operators/dataproc_operator.py*
> becomes *airflow/operators/gcp_dataproc_operator.py*
>
>
> -
> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/*
> *hooks/gcp_bigtable_hook.py*
> -
> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> *operators/ssh_operator.py*
>
>
>
> *Option B*. Group operators/sensors/hooks in airflow/operators(sensors,
> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
> airflow/operators(sensors/hooks), Drop the prefix.
>
> -
> *airflow/contrib/operators/sns_publish_operator.py
> becomes airflow/operators/**aws/sns_publish_operator.py*
>
>
> - *airflow/contrib/operators/dataproc_operator.py*
> becomes *airflow/operators/gcp/dataproc_operator.py*
>
>
> -
> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
> *operators/gcp/bigtable_operator.py*
> -
> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> *operators/ssh_operator.py*
>
>
> *Option C*. Group operators/sensors/hooks in airflow/operators(sensors,
> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
> airflow/operators(sensors/hooks), Keep the prefix.
>
> -
> *airflow/contrib/operators/sns_publish_operator.py
> becomes airflow/operators/**aws/aws_sns_publish_operator.py*
>
>
> - *airflow/contrib/operators/dataproc_operator.py*
> becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
>
>
> -
> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
> *operators/gcp/gcp_bigtable_operator.py*
> -
> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> *operators/ssh_operator.py*
>
>
> *Option D.* Group operators/sensors/hooks in *airflow/<PROVIDER>*/operators(sensors,
> hooks). Non-cloud provider ones are moved to
> airflow/operators(sensors/hooks). Drop the prefix.
>
> -
> *airflow/contrib/operators/sns_publish_operator.py
> becomes airflow/aws/operators/**sns_publish_operator.py*
>
>
> - *airflow/contrib/operators/dataproc_operator.py*
> becomes *airflow/gcp/operators/dataproc_operator.py*
>
>
> -
> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
> *gcp/operators/bigtable_operator.py*
> -
> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> *operators/ssh_operator.py*
>
>
> *Option E.* Group operators/sensors/hooks in *airflow/<PROVIDER>*/operators(sensors,
> hooks). Non-cloud provider ones are moved to
> airflow/operators(sensors/hooks).* Keep the prefix*.
>
> -
> *airflow/contrib/operators/sns_publish_operator.py
> becomes airflow/aws/operators/aws_**sns_publish_operator.py*
>
>
> - *airflow/contrib/operators/dataproc_operator.py*
> becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
>
>
> -
> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
> *gcp/operators/gcp_bigtable_operator.py*
> -
> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
> *operators/ssh_operator.py*
>
>
> Shall I proceed with updating the page ?
>
>
> J.
>
>
> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <ka...@gmail.com> wrote:
>
>> Thanks Jarek, adding my vote.
>>
>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>> > Thanks Jarek, much clearer. I have voted there too.
>> >
>> > -ash
>> >
>> >
>> > > On 29 Jul 2019, at 08:13, Driesprong, Fokko <fo...@driesprong.frl>
>> > wrote:
>> > >
>> > > Thanks Jarek, the Wiki works much better for me. I've added my vote
>> there
>> > > as well.
>> > >
>> > > You've convinced me on the second case. I've changed my vote on Case 2
>> > from
>> > > A to B :-)
>> > >
>> > > Maybe we should leave the vote open a bit longer since it involves
>> quite
>> > > some reading, and I would like to get some proper feedback and
>> opinions
>> > on
>> > > this. Plus I only see two votes right now. If we don't think this
>> > through,
>> > > it might be that we're having this vote again in the near future :-)
>> > >
>> > > Cheers, Fokko
>> > >
>> > > Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
>> > Jarek.Potiuk@polidea.com>:
>> > >
>> > >> I updated the AIP-21 proposal in the way that should answer all the
>> > >> concerns in this thread.
>> > >>
>> > >> NOTE! There is an action for all of those who are interested: Please
>> > >> fill-in your voting in the voting table till Tuesday 30th of July 6pm
>> > CEST
>> > >> (5pm BST, 9 am PST).
>> > >>
>> > >> I created a summary of the proposal
>> > >> <
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
>> > >>>
>> > >> in table form - which contains also examples for each case.
>> > >>
>> > >> I added a voting table
>> > >> <
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
>> > >>>
>> > >> where you should add your votes:
>> > >>
>> > >> I propose this voting mechanism:
>> > >>
>> > >> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 pm
>> BST,
>> > 9am
>> > >> PST) .
>> > >>
>> > >> After the choice, the final consistent set of choices will be
>> announced
>> > >> (taking into account majority of binding vote, also including
>> potential
>> > >> vetos and consistency between the choices. Non-binding votes will be
>> > taken
>> > >> into account in case there is a draw. The final set of choices will
>> be
>> > >> announced at devlist thread
>> > >> <
>> > >>
>> >
>> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
>> > >>>
>> > >> after
>> > >> the voting completes.
>> > >>
>> > >> Unless there is a veto raised to the final proposal, the final
>> proposal
>> > is
>> > >> accepted by Lazy Consensus
>> > >> <https://community.apache.org/committers/lazyConsensus.html> on
>> > *Friday*
>> > >> 02
>> > >> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
>> > >>
>> > >> Please let me know if what I proposed can be further improved to
>> avoid
>> > >> chaos.
>> > >>
>> > >> J.
>> > >>
>> > >> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
>> Jarek.Potiuk@polidea.com
>> > >
>> > >> wrote:
>> > >>
>> > >>> Ok. I see that indeed voting could have been organised a bit better
>> -
>> > dev
>> > >>> list does not seem to cope well with such a complex case :).
>> > >>>
>> > >>> As Kamil noticed - we already have a formal AIP-21
>> > >>> <
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>> > >>>
>> > >>> in the Wiki - which I should have mentioned before. I have not much
>> > time
>> > >>> today - but tomorrow I will try to summarise the options - based on
>> > some
>> > >>> real code examples (to make it easier to digest). I think the best
>> > >> approach
>> > >>> will be to make a voting matrix in the AIP-21 Wiki page where people
>> > will
>> > >>> be able to state their preferences for each case (by adding a row to
>> > the
>> > >>> table). I think that might provide much better clarity and a single
>> > place
>> > >>> which will show the current aggregated state of voting.
>> > >>>
>> > >>> However - as Ash also stated - some of the options are
>> inter-dependent
>> > so
>> > >>> at the end we will have to adjust the results in case they are not
>> > >>> consistent - and make final voting on aggregated set of cases -
>> but I
>> > >>> think in this case following "lasy consensus" - i,e. if noone
>> objects
>> > to
>> > >>> final set of cases, we will proceed with it..
>> > >>>
>> > >>> Do you think that will work better ?
>> > >>>
>> > >>> J.
>> > >>>
>> > >>> Principal Software Engineer
>> > >>> Phone: +48660796129
>> > >>>
>> > >>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
>> > >>> kamil.bregula@polidea.com> napisał:
>> > >>>
>> > >>>> Document is also available as AIP on wiki:
>> > >>>>
>> > >>>>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>> > >>>>
>> > >>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
>> <fokko@driesprong.frl
>> > >
>> > >>>> wrote:
>> > >>>>
>> > >>>>> I agree with all, this is a bit chaotic. For the sake of clarity,
>> I
>> > >>>> would
>> > >>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a
>> > >> structural
>> > >>>>> code improvement AIP and then and add a matrix of users/cases
>> where
>> > >>>>> everybody can add his vote per case. This gives also the
>> opportunity
>> > >> for
>> > >>>>> changing your vote on a specific case. WDYT?
>> > >>>>>
>> > >>>>> Cheers, Fokko
>> > >>>>>
>> > >>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <ci...@gmail.com>:
>> > >>>>>
>> > >>>>>> I agree with Ash. It is clear to vote on each case rather than
>> all
>> > >>>>>> together.
>> > >>>>>>
>> > >>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
>> ash@apache.org>
>> > >>>>> wrote:
>> > >>>>>>
>> > >>>>>>> A number of proposals overlap or add on to each other, so I
>> think
>> > >>>> (?) a
>> > >>>>>>> single vote makes sense.
>> > >>>>>>>
>> > >>>>>>> To be clear, my vote is -1 until it's clear exactly what we are
>> > >>>> voting
>> > >>>>> on.
>> > >>>>>>>
>> > >>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <ka...@gmail.com>
>> > >>>> wrote:
>> > >>>>>>>> I agree with Ash and instead of voting on all 7 Cases together,
>> > >> how
>> > >>>>>>>> about
>> > >>>>>>>> separate vote emails for all the cases? Each email can have the
>> > >>>>>>>> description
>> > >>>>>>>> copied over from the doc.
>> > >>>>>>>>
>> > >>>>>>>> It would probably be a bit easier to track a decision in the
>> > >> future
>> > >>>> as
>> > >>>>>>>> well.
>> > >>>>>>>>
>> > >>>>>>>> What do you guys think? @Jarek Potiuk <
>> Jarek.Potiuk@polidea.com>
>> > >>>>>>>> @Driesprong,
>> > >>>>>>>> Fokko <fo...@driesprong.frl>
>> > >>>>>>>>
>> > >>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
>> kaxilnaik@gmail.com>
>> > >>>>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
>> > >>>>>>>>>
>> > >>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
>> > >>>>>>>>>
>> > >>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
>> > >>>>>>>>>
>> > >>>>>>>>> *Case #4* Separate namespace for resources :
>> > >>>>>>>>> *Solution A*
>> > >>>>>>>>>
>> > >>>>>>>>> *Case #5* *Operator : *Solution A*
>> > >>>>>>>>>
>> > >>>>>>>>> *Case #7* : *Solution A*
>> > >>>>>>>>>
>> > >>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
>> > >>>>>>>> <dp...@gmail.com>
>> > >>>>>>>>> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>> I have tried to add some clarification to Jarek's summary,
>> but
>> > >> I
>> > >>>> am
>> > >>>>>>>> a
>> > >>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully Jarek
>> > >> can
>> > >>>>>>>> further
>> > >>>>>>>>>> clarify.
>> > >>>>>>>>>>
>> > >>>>>>>>>> According to my reading, in this motion cases 4,5,6 are all
>> > >>>> either
>> > >>>>>>>>>> proposal
>> > >>>>>>>>>> *rejections* or otherwise have no effect, so attention can be
>> > >>>>>>>> focused on
>> > >>>>>>>>>> cases 1,2,3 and 7.
>> > >>>>>>>>>>
>> > >>>>>>>>>> *Motion is to adopt the following:*
>> > >>>>>>>>>>
>> > >>>>>>>>>> Case 1 - Solution A - Get rid of contrib
>> > >>>>>>>>>> All objects will be moved out of contrib folder and into
>> > >>>> respective
>> > >>>>>>>>>> non-contrib folder.
>> > >>>>>>>>>>
>> > >>>>>>>>>> Case 2 - Solution A - Remove object type suffix from modules
>> > >>>>>>>>>> Example:
>> > >>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
>> > >>>>>>>>>> airflow.operators.foo.FooOperator
>> > >>>>>>>>>> Example:
>> > >>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
>> > >>>> airflow.hooks.foo.FooHook
>> > >>>>>>>>>>
>> > >>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
>> > >>>>>>>>>> *I am not sure exactly what the proposal is.*
>> > >>>>>>>>>> Example case is
>> > >>>> airflow/contrib/operators/gcp_bigtable_operator.py
>> > >>>>>>>>>> Since motion adopts case 1 solution A, we can assume it is
>> now
>> > >>>> named
>> > >>>>>>>> like
>> > >>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
>> > >>>>>>>>>> So what is proposed for this case?
>> > >>>>>>>>>>
>> > >>>>>>>>>> Case 4 - Solution B - *no change*
>> > >>>>>>>>>> This proposal was about namespaces but the motion is to
>> reject
>> > >>>> the
>> > >>>>>>>>>> proposal.
>> > >>>>>>>>>>
>> > >>>>>>>>>> Case 5 - Solution B - *no change*
>> > >>>>>>>>>> The proposal was to remove class type suffix e.g. remove the
>> > >>>>>>>> Operator in
>> > >>>>>>>>>> FooOperator, but the motion is to reject this proposal --
>> i.e.
>> > >>>>>>>> motion is
>> > >>>>>>>>>> no
>> > >>>>>>>>>> change on this case, i.e. we resolve to keep "Operator" (and
>> > >>>>>>>> "Sensor")
>> > >>>>>>>>>> suffixes on class names
>> > >>>>>>>>>>
>> > >>>>>>>>>> Case 6 - *No change*
>> > >>>>>>>>>> This case concerns object naming inconsistencies, but motion
>> > >> does
>> > >>>>>>>> not
>> > >>>>>>>>>> enact
>> > >>>>>>>>>> any specific proposal.
>> > >>>>>>>>>>
>> > >>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
>> > >> deprecation
>> > >>>>>>>> warnings
>> > >>>>>>>>>> (instead of zope)
>> > >>>>>>>>>> Example:
>> > >>>>>>>>>>
>> > >>>>>>>>>> # in old_package/old_mod.py
>> > >>>>>>>>>>
>> > >>>>>>>>>> import warnings
>> > >>>>>>>>>> from new_package.new_mod import *
>> > >>>>>>>>>> warnings.warn("old_package.old_mod has moved to
>> > >>>> new_package.new_mod.
>> > >>>>>>>>>> Import
>> > >>>>>>>>>> of "
>> > >>>>>>>>>> "old_package.old_mod will become unsupported in version 2",
>> > >>>>>>>>>> DeprecationWarning, 2)
>> > >>>>>>>>>>
>> > >>>>>>>>>> Reference implementation here:
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>
>> > >>>>>
>> > >>>>
>> > >>
>> >
>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
>> > >>>>>>>>>>
>> > >>>>>>>>>> I am very happy that this motion now gets rid of contrib.
>> > >> Thank
>> > >>>> you
>> > >>>>>>>>>> Sergio
>> > >>>>>>>>>> for turning the tide with your effective argumentation ;)
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
>> > >>>> ash@apache.org>
>> > >>>>>>>> wrote:
>> > >>>>>>>>>>
>> > >>>>>>>>>>> Could someone summarise what the proposed name changes are,
>> > >>>> here,
>> > >>>>>>>> in
>> > >>>>>>>>>> this
>> > >>>>>>>>>>> thread? Pointing at a google doc and only giving partial
>> > >>>> examples
>> > >>>>>>>> is 1)
>> > >>>>>>>>>> a
>> > >>>>>>>>>>> bit difficult to quickly work out what we are voting on, and
>> > >> 2)
>> > >>>>>>>> using a
>> > >>>>>>>>>>> google doc isn't great for a longer term record.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> Thanks,
>> > >>>>>>>>>>> Ash
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
>> > >>>>>>>> <Ja...@polidea.com>
>> > >>>>>>>>>> wrote:
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :) . I
>> > >>>> would
>> > >>>>>>>> like
>> > >>>>>>>>>> to
>> > >>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three +1
>> > >>>>>>>> (binding)
>> > >>>>>>>>>> votes
>> > >>>>>>>>>>> are
>> > >>>>>>>>>>>> needed to pass. I marked in red the modifications to the
>> > >>>>>>>> previous
>> > >>>>>>>>>>> proposal.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> Here is the proposal (details here
>> > >>>>>>>>>>>> <
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>
>> > >>>>>
>> > >>>>
>> > >>
>> >
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> )
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
>> > >>>>>>>>>>>> - *Case 2: A: Airflow.operators.foo_operator.FooOperator
>> > >>>> could
>> > >>>>>>>>>>>> become airflow.operators.foo.FooOperator*
>> > >>>>>>>>>>>> - *Case 3: C:
>> > >>>>>>>>>>>>
>> > >>>>>>>>
>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>> > >>>>>>>>>> could
>> > >>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
>> > >>>>>>>>>>>> - *Case 4: B: no namespace introduction*
>> > >>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
>> > >>>> class
>> > >>>>>>>>>> names*
>> > >>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
>> > >>>> (and
>> > >>>>>>>> my team
>> > >>>>>>>>>>> can
>> > >>>>>>>>>>>> do the job of GCP-related operators)*
>> > >>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
>> > >>>>>>>> integration)*
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> This is my binding (+1) vote.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
>> > >>>>>>>>>> Jarek.Potiuk@polidea.com>
>> > >>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>> Hey Felix,
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as well
>> > >> (I
>> > >>>>>>>> use IDE
>> > >>>>>>>>>>> quite
>> > >>>>>>>>>>>>> often ;) )
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> J.
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
>> > >>>>>>>>>> <fokko@driesprong.frl
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Thanks Kamil for putting the document together. I wasn't
>> > >>>> able
>> > >>>>>>>> to
>> > >>>>>>>>>>> respond
>> > >>>>>>>>>>>>>> earlier since I was giving Airflow workshops throughout
>> > >>>> Europe
>> > >>>>>>>> :-)
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from contrib
>> > >> into
>> > >>>> a
>> > >>>>>>>> single
>> > >>>>>>>>>>>>>> package.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> To make it easier to the end-user, my preference would be
>> > >>>> to
>> > >>>>>>>> get
>> > >>>>>>>>>> rid of
>> > >>>>>>>>>>>>>> the
>> > >>>>>>>>>>>>>> contrib package altogether. What's in contrib or not is a
>> > >>>>>>>> tricky
>> > >>>>>>>>>>> question,
>> > >>>>>>>>>>>>>> I think this is already reflected in the document since
>> > >>>>>>>> "maturity"
>> > >>>>>>>>>> is
>> > >>>>>>>>>>> in
>> > >>>>>>>>>>>>>> quotes. What would be the moment to transfer the operator
>> > >>>> to
>> > >>>>>>>> the
>> > >>>>>>>>>> core
>> > >>>>>>>>>>> or
>> > >>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
>> > >> incubating
>> > >>>>>>>> will
>> > >>>>>>>>>> confuse
>> > >>>>>>>>>>>>>> the
>> > >>>>>>>>>>>>>> user even more. For people who aren't as known with
>> > >>>> Airflow it
>> > >>>>>>>> isn't
>> > >>>>>>>>>>> that
>> > >>>>>>>>>>>>>> transparent which operator lives where, especially if you
>> > >>>>>>>> don't use
>> > >>>>>>>>>> an
>> > >>>>>>>>>>> IDE
>> > >>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is worth
>> > >>>> changing
>> > >>>>>>>> the
>> > >>>>>>>>>>> public
>> > >>>>>>>>>>>>>> API just for a different name.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with this. It
>> > >>>> will
>> > >>>>>>>> make
>> > >>>>>>>>>>> easier
>> > >>>>>>>>>>>>> as we will not have to define other rules as those that we
>> > >>>>>>>> already
>> > >>>>>>>>>> have.
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's say a
>> > >> +0)
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> I like to search on Github with t, and then search for
>> > >>>>>>>> BashOperator.
>> > >>>>>>>>>>> After
>> > >>>>>>>>>>>>>> the change, you should search for Operator/Bash.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from your
>> > >> mail
>> > >>>> you
>> > >>>>>>>>>> state: -
>> > >>>>>>>>>>>>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator
>> > >>>> could
>> > >>>>>>>> become
>> > >>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B solution is
>> > >>>>>>>> keeping it
>> > >>>>>>>>>>> the
>> > >>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
>> > >>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would stay
>> > >>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> My mistake. It was of course A. I think there is a little
>> > >>>>>>>> confusion
>> > >>>>>>>>>>> here.
>> > >>>>>>>>>>>>> This case is not for changing BashOperator into Bash but
>> > >> for
>> > >>>>>>>> changing
>> > >>>>>>>>>>> the
>> > >>>>>>>>>>>>> *module* name not the *class* name. So
>> > >>>>>>>> bash_operator.BashOperator ->
>> > >>>>>>>>>>>>> bash.BashOperator :) . you will still search for
>> > >>>> BashOperator
>> > >>>>>>>> in this
>> > >>>>>>>>>>> case.
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it gets
>> > >>>> its
>> > >>>>>>>> own
>> > >>>>>>>>>>> package
>> > >>>>>>>>>>>>>> or not. Besides the cloud providers, there are companies
>> > >>>> like
>> > >>>>>>>>>>> Databricks
>> > >>>>>>>>>>>>>> and Qubole which might also end up with their own package
>> > >>>>>>>> because
>> > >>>>>>>>>> their
>> > >>>>>>>>>>>>>> integration is getting richer over time. Moving this is a
>> > >>>> bit
>> > >>>>>>>>>> painful
>> > >>>>>>>>>>>>>> because we need to deprecate the old path and move
>> > >>>> everything
>> > >>>>>>>> which
>> > >>>>>>>>>>>>>> introduces noise into version control etc.
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not as
>> > >>>>>>>> difficult as
>> > >>>>>>>>>> it
>> > >>>>>>>>>>>>> seems to group operators in packages and it has numerous
>> > >>>>>>>> advantages.
>> > >>>>>>>>>> The
>> > >>>>>>>>>>>>> nice thing about deprecations - we can do them in the
>> > >>>>>>>> __init__.py of
>> > >>>>>>>>>> the
>> > >>>>>>>>>>>>> packages which causes the noise fairly small (Git will
>> > >>>> actually
>> > >>>>>>>>>> realise
>> > >>>>>>>>>>>>> that the file has been moved and you can follow that.
>> > >> Maybe
>> > >>>> not
>> > >>>>>>>> as
>> > >>>>>>>>>> super
>> > >>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's not a
>> > >> rocket
>> > >>>>>>>> science
>> > >>>>>>>>>>> either.
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> *Case 7:* Python native solution
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> Yes.
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> ---
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Apart from all, great work!
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Cheers, Fokko
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
>> > >>>>>>>>>>>>>> <feluelle@pm.me.invalid
>> > >>>>>>>>>>>>>>> :
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> I have no strong "No" against any proposed change of
>> > >> these
>> > >>>>>>>> cases.
>> > >>>>>>>>>> So I
>> > >>>>>>>>>>>>>> go
>> > >>>>>>>>>>>>>>> with +1 (non-binding).
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and your
>> > >>>> intense
>> > >>>>>>>> work
>> > >>>>>>>>>>>>>> towards
>> > >>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
>> > >> creating
>> > >>>>>>>> this
>> > >>>>>>>>>>>>>> document. I
>> > >>>>>>>>>>>>>>> like how the code is getting more and more consistent
>> > >> and
>> > >>>>>>>> clean :)
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> Kind regards,
>> > >>>>>>>>>>>>>>> Felix
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> Sent from ProtonMail mobile
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> -------- Original Message --------
>> > >>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> Hello everyone,
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> This email is calling a vote on the changes in import
>> > >>>> paths.
>> > >>>>>>>> It's
>> > >>>>>>>>>>> been
>> > >>>>>>>>>>>>>>>> discussed in
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>
>> > >>>>>
>> > >>>>
>> > >>
>> >
>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th 6pm
>> > >>>> CEST),
>> > >>>>>>>> and
>> > >>>>>>>>>> at
>> > >>>>>>>>>>>>>> least
>> > >>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> The proposal to vote is based on the document from
>> > >> Kamil
>> > >>>>>>>>>>>>>>>> <
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>
>> > >>>>>
>> > >>>>
>> > >>
>> >
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> The proposed solution is:
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are
>> > >> "well"
>> > >>>>>>>> tested
>> > >>>>>>>>>> and
>> > >>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
>> > >>>>>>>>>>>>>>>> - *Case 2: B:
>> > >> Airflow.operators.foo_operator.FooOperator
>> > >>>>>>>> could
>> > >>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
>> > >>>>>>>>>>>>>>>> - *Case 3: C:
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>
>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>> > >>>>>>>>>>> could
>> > >>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
>> > >>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
>> > >>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes
>> > >> on
>> > >>>>>>>> class
>> > >>>>>>>>>> names*
>> > >>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
>> > >>>> (and
>> > >>>>>>>> my
>> > >>>>>>>>>> team
>> > >>>>>>>>>>>>>> can
>> > >>>>>>>>>>>>>>>> do the job of GCP-related operators)*
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> This is my (binding) +1 vote.
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> Best regards,
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> J.
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> --
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> Jarek Potiuk
>> > >>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
>> > >> Software
>> > >>>>>>>> Engineer
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
>> > >>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
>> > >>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> --
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> Jarek Potiuk
>> > >>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
>> > >>>>>>>> Engineer
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>> > >>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> --
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> Jarek Potiuk
>> > >>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
>> > >>>>> Engineer
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>> > >>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> --
>> > >>>>>>>>> *Kaxil Naik*
>> > >>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>> > >>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
>> > >> Spark
>> > >>>> &
>> > >>>>>>>> Neo4j
>> > >>>>>>>>> Developer
>> > >>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>> > >>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> --
>> > >>>>>>>> *Kaxil Naik*
>> > >>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>> > >>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
>> Spark
>> > >> &
>> > >>>>>>>> Neo4j
>> > >>>>>>>> Developer
>> > >>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>> > >>>>>>>
>> > >>>>>>
>> > >>>>>
>> > >>>>
>> > >>>
>> > >>
>> > >> --
>> > >>
>> > >> Jarek Potiuk
>> > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
>> > >>
>> > >> M: +48 660 796 129 <+48660796129>
>> > >> [image: Polidea] <https://www.polidea.com/>
>> > >>
>> >
>> >
>>
>> --
>> *Kaxil Naik*
>> *Big Data Consultant | DevOps Data Engineer*
>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
>> Developer
>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Changes in import paths
Posted by Jarek Potiuk <Ja...@polidea.com>.
I think almost everyone voted and we have almost perfect consensus. We all
agree amongst other on moving all operators out of contrib (Great!).
The only doubts are for *Case 3* (Cloud provider prefix) and *Case 4*
(Using Namespaces).
I think there was actually an overlap between those two. Also Ash's
comment/veto on that is quite clear.
So I have final (I hope!) proposal to join those two cases and have one
voting (till Friday) only on that.
I will update the doc if you all agree with me that makes more sense as
single case to vote on:
*[Case 3 + Case 4]: Grouping Cloud Provider operators.*
*Option A*. Keep operators/sensors/hooks in airflow/operators(sensors,
hooks) and keep/add prefixes in file names.
-
*airflow/contrib/operators/sns_publish_operator.py becomes
airflow/operators/**aws_sns_publish_operator.py*
- *airflow/contrib/operators/dataproc_operator.py*
becomes *airflow/operators/gcp_dataproc_operator.py*
-
*airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/*
*hooks/gcp_bigtable_hook.py*
-
*airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
*operators/ssh_operator.py*
*Option B*. Group operators/sensors/hooks in airflow/operators(sensors,
hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
airflow/operators(sensors/hooks), Drop the prefix.
-
*airflow/contrib/operators/sns_publish_operator.py
becomes airflow/operators/**aws/sns_publish_operator.py*
- *airflow/contrib/operators/dataproc_operator.py*
becomes *airflow/operators/gcp/dataproc_operator.py*
-
*airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
*operators/gcp/bigtable_operator.py*
-
*airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
*operators/ssh_operator.py*
*Option C*. Group operators/sensors/hooks in airflow/operators(sensors,
hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
airflow/operators(sensors/hooks), Keep the prefix.
-
*airflow/contrib/operators/sns_publish_operator.py
becomes airflow/operators/**aws/aws_sns_publish_operator.py*
- *airflow/contrib/operators/dataproc_operator.py*
becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
-
*airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
*operators/gcp/gcp_bigtable_operator.py*
-
*airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
*operators/ssh_operator.py*
*Option D.* Group operators/sensors/hooks in
*airflow/<PROVIDER>*/operators(sensors,
hooks). Non-cloud provider ones are moved to
airflow/operators(sensors/hooks). Drop the prefix.
-
*airflow/contrib/operators/sns_publish_operator.py
becomes airflow/aws/operators/**sns_publish_operator.py*
- *airflow/contrib/operators/dataproc_operator.py*
becomes *airflow/gcp/operators/dataproc_operator.py*
-
*airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
*gcp/operators/bigtable_operator.py*
-
*airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
*operators/ssh_operator.py*
*Option E.* Group operators/sensors/hooks in
*airflow/<PROVIDER>*/operators(sensors,
hooks). Non-cloud provider ones are moved to
airflow/operators(sensors/hooks).* Keep the prefix*.
-
*airflow/contrib/operators/sns_publish_operator.py
becomes airflow/aws/operators/aws_**sns_publish_operator.py*
- *airflow/contrib/operators/dataproc_operator.py*
becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
-
*airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/*
*gcp/operators/gcp_bigtable_operator.py*
-
*airflow/contrib/operators/ssh_operator.py *becomes *airflow/*
*operators/ssh_operator.py*
Shall I proceed with updating the page ?
J.
On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <ka...@gmail.com> wrote:
> Thanks Jarek, adding my vote.
>
> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > Thanks Jarek, much clearer. I have voted there too.
> >
> > -ash
> >
> >
> > > On 29 Jul 2019, at 08:13, Driesprong, Fokko <fo...@driesprong.frl>
> > wrote:
> > >
> > > Thanks Jarek, the Wiki works much better for me. I've added my vote
> there
> > > as well.
> > >
> > > You've convinced me on the second case. I've changed my vote on Case 2
> > from
> > > A to B :-)
> > >
> > > Maybe we should leave the vote open a bit longer since it involves
> quite
> > > some reading, and I would like to get some proper feedback and opinions
> > on
> > > this. Plus I only see two votes right now. If we don't think this
> > through,
> > > it might be that we're having this vote again in the near future :-)
> > >
> > > Cheers, Fokko
> > >
> > > Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>:
> > >
> > >> I updated the AIP-21 proposal in the way that should answer all the
> > >> concerns in this thread.
> > >>
> > >> NOTE! There is an action for all of those who are interested: Please
> > >> fill-in your voting in the voting table till Tuesday 30th of July 6pm
> > CEST
> > >> (5pm BST, 9 am PST).
> > >>
> > >> I created a summary of the proposal
> > >> <
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > >>>
> > >> in table form - which contains also examples for each case.
> > >>
> > >> I added a voting table
> > >> <
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> > >>>
> > >> where you should add your votes:
> > >>
> > >> I propose this voting mechanism:
> > >>
> > >> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 pm BST,
> > 9am
> > >> PST) .
> > >>
> > >> After the choice, the final consistent set of choices will be
> announced
> > >> (taking into account majority of binding vote, also including
> potential
> > >> vetos and consistency between the choices. Non-binding votes will be
> > taken
> > >> into account in case there is a draw. The final set of choices will be
> > >> announced at devlist thread
> > >> <
> > >>
> >
> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> > >>>
> > >> after
> > >> the voting completes.
> > >>
> > >> Unless there is a veto raised to the final proposal, the final
> proposal
> > is
> > >> accepted by Lazy Consensus
> > >> <https://community.apache.org/committers/lazyConsensus.html> on
> > *Friday*
> > >> 02
> > >> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
> > >>
> > >> Please let me know if what I proposed can be further improved to avoid
> > >> chaos.
> > >>
> > >> J.
> > >>
> > >> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >
> > >> wrote:
> > >>
> > >>> Ok. I see that indeed voting could have been organised a bit better -
> > dev
> > >>> list does not seem to cope well with such a complex case :).
> > >>>
> > >>> As Kamil noticed - we already have a formal AIP-21
> > >>> <
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > >>>
> > >>> in the Wiki - which I should have mentioned before. I have not much
> > time
> > >>> today - but tomorrow I will try to summarise the options - based on
> > some
> > >>> real code examples (to make it easier to digest). I think the best
> > >> approach
> > >>> will be to make a voting matrix in the AIP-21 Wiki page where people
> > will
> > >>> be able to state their preferences for each case (by adding a row to
> > the
> > >>> table). I think that might provide much better clarity and a single
> > place
> > >>> which will show the current aggregated state of voting.
> > >>>
> > >>> However - as Ash also stated - some of the options are
> inter-dependent
> > so
> > >>> at the end we will have to adjust the results in case they are not
> > >>> consistent - and make final voting on aggregated set of cases - but
> I
> > >>> think in this case following "lasy consensus" - i,e. if noone objects
> > to
> > >>> final set of cases, we will proceed with it..
> > >>>
> > >>> Do you think that will work better ?
> > >>>
> > >>> J.
> > >>>
> > >>> Principal Software Engineer
> > >>> Phone: +48660796129
> > >>>
> > >>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> > >>> kamil.bregula@polidea.com> napisał:
> > >>>
> > >>>> Document is also available as AIP on wiki:
> > >>>>
> > >>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > >>>>
> > >>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
> <fokko@driesprong.frl
> > >
> > >>>> wrote:
> > >>>>
> > >>>>> I agree with all, this is a bit chaotic. For the sake of clarity, I
> > >>>> would
> > >>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a
> > >> structural
> > >>>>> code improvement AIP and then and add a matrix of users/cases where
> > >>>>> everybody can add his vote per case. This gives also the
> opportunity
> > >> for
> > >>>>> changing your vote on a specific case. WDYT?
> > >>>>>
> > >>>>> Cheers, Fokko
> > >>>>>
> > >>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <ci...@gmail.com>:
> > >>>>>
> > >>>>>> I agree with Ash. It is clear to vote on each case rather than all
> > >>>>>> together.
> > >>>>>>
> > >>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <ash@apache.org
> >
> > >>>>> wrote:
> > >>>>>>
> > >>>>>>> A number of proposals overlap or add on to each other, so I think
> > >>>> (?) a
> > >>>>>>> single vote makes sense.
> > >>>>>>>
> > >>>>>>> To be clear, my vote is -1 until it's clear exactly what we are
> > >>>> voting
> > >>>>> on.
> > >>>>>>>
> > >>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <ka...@gmail.com>
> > >>>> wrote:
> > >>>>>>>> I agree with Ash and instead of voting on all 7 Cases together,
> > >> how
> > >>>>>>>> about
> > >>>>>>>> separate vote emails for all the cases? Each email can have the
> > >>>>>>>> description
> > >>>>>>>> copied over from the doc.
> > >>>>>>>>
> > >>>>>>>> It would probably be a bit easier to track a decision in the
> > >> future
> > >>>> as
> > >>>>>>>> well.
> > >>>>>>>>
> > >>>>>>>> What do you guys think? @Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > >>>>>>>> @Driesprong,
> > >>>>>>>> Fokko <fo...@driesprong.frl>
> > >>>>>>>>
> > >>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <kaxilnaik@gmail.com
> >
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
> > >>>>>>>>>
> > >>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
> > >>>>>>>>>
> > >>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
> > >>>>>>>>>
> > >>>>>>>>> *Case #4* Separate namespace for resources :
> > >>>>>>>>> *Solution A*
> > >>>>>>>>>
> > >>>>>>>>> *Case #5* *Operator : *Solution A*
> > >>>>>>>>>
> > >>>>>>>>> *Case #7* : *Solution A*
> > >>>>>>>>>
> > >>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> > >>>>>>>> <dp...@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> I have tried to add some clarification to Jarek's summary, but
> > >> I
> > >>>> am
> > >>>>>>>> a
> > >>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully Jarek
> > >> can
> > >>>>>>>> further
> > >>>>>>>>>> clarify.
> > >>>>>>>>>>
> > >>>>>>>>>> According to my reading, in this motion cases 4,5,6 are all
> > >>>> either
> > >>>>>>>>>> proposal
> > >>>>>>>>>> *rejections* or otherwise have no effect, so attention can be
> > >>>>>>>> focused on
> > >>>>>>>>>> cases 1,2,3 and 7.
> > >>>>>>>>>>
> > >>>>>>>>>> *Motion is to adopt the following:*
> > >>>>>>>>>>
> > >>>>>>>>>> Case 1 - Solution A - Get rid of contrib
> > >>>>>>>>>> All objects will be moved out of contrib folder and into
> > >>>> respective
> > >>>>>>>>>> non-contrib folder.
> > >>>>>>>>>>
> > >>>>>>>>>> Case 2 - Solution A - Remove object type suffix from modules
> > >>>>>>>>>> Example:
> > >>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
> > >>>>>>>>>> airflow.operators.foo.FooOperator
> > >>>>>>>>>> Example:
> > >>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
> > >>>> airflow.hooks.foo.FooHook
> > >>>>>>>>>>
> > >>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
> > >>>>>>>>>> *I am not sure exactly what the proposal is.*
> > >>>>>>>>>> Example case is
> > >>>> airflow/contrib/operators/gcp_bigtable_operator.py
> > >>>>>>>>>> Since motion adopts case 1 solution A, we can assume it is now
> > >>>> named
> > >>>>>>>> like
> > >>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
> > >>>>>>>>>> So what is proposed for this case?
> > >>>>>>>>>>
> > >>>>>>>>>> Case 4 - Solution B - *no change*
> > >>>>>>>>>> This proposal was about namespaces but the motion is to reject
> > >>>> the
> > >>>>>>>>>> proposal.
> > >>>>>>>>>>
> > >>>>>>>>>> Case 5 - Solution B - *no change*
> > >>>>>>>>>> The proposal was to remove class type suffix e.g. remove the
> > >>>>>>>> Operator in
> > >>>>>>>>>> FooOperator, but the motion is to reject this proposal -- i.e.
> > >>>>>>>> motion is
> > >>>>>>>>>> no
> > >>>>>>>>>> change on this case, i.e. we resolve to keep "Operator" (and
> > >>>>>>>> "Sensor")
> > >>>>>>>>>> suffixes on class names
> > >>>>>>>>>>
> > >>>>>>>>>> Case 6 - *No change*
> > >>>>>>>>>> This case concerns object naming inconsistencies, but motion
> > >> does
> > >>>>>>>> not
> > >>>>>>>>>> enact
> > >>>>>>>>>> any specific proposal.
> > >>>>>>>>>>
> > >>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
> > >> deprecation
> > >>>>>>>> warnings
> > >>>>>>>>>> (instead of zope)
> > >>>>>>>>>> Example:
> > >>>>>>>>>>
> > >>>>>>>>>> # in old_package/old_mod.py
> > >>>>>>>>>>
> > >>>>>>>>>> import warnings
> > >>>>>>>>>> from new_package.new_mod import *
> > >>>>>>>>>> warnings.warn("old_package.old_mod has moved to
> > >>>> new_package.new_mod.
> > >>>>>>>>>> Import
> > >>>>>>>>>> of "
> > >>>>>>>>>> "old_package.old_mod will become unsupported in version 2",
> > >>>>>>>>>> DeprecationWarning, 2)
> > >>>>>>>>>>
> > >>>>>>>>>> Reference implementation here:
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> > >>>>>>>>>>
> > >>>>>>>>>> I am very happy that this motion now gets rid of contrib.
> > >> Thank
> > >>>> you
> > >>>>>>>>>> Sergio
> > >>>>>>>>>> for turning the tide with your effective argumentation ;)
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
> > >>>> ash@apache.org>
> > >>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Could someone summarise what the proposed name changes are,
> > >>>> here,
> > >>>>>>>> in
> > >>>>>>>>>> this
> > >>>>>>>>>>> thread? Pointing at a google doc and only giving partial
> > >>>> examples
> > >>>>>>>> is 1)
> > >>>>>>>>>> a
> > >>>>>>>>>>> bit difficult to quickly work out what we are voting on, and
> > >> 2)
> > >>>>>>>> using a
> > >>>>>>>>>>> google doc isn't great for a longer term record.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>> Ash
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
> > >>>>>>>> <Ja...@polidea.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :) . I
> > >>>> would
> > >>>>>>>> like
> > >>>>>>>>>> to
> > >>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three +1
> > >>>>>>>> (binding)
> > >>>>>>>>>> votes
> > >>>>>>>>>>> are
> > >>>>>>>>>>>> needed to pass. I marked in red the modifications to the
> > >>>>>>>> previous
> > >>>>>>>>>>> proposal.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Here is the proposal (details here
> > >>>>>>>>>>>> <
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> )
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
> > >>>>>>>>>>>> - *Case 2: A: Airflow.operators.foo_operator.FooOperator
> > >>>> could
> > >>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > >>>>>>>>>>>> - *Case 3: C:
> > >>>>>>>>>>>>
> > >>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > >>>>>>>>>> could
> > >>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> > >>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > >>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
> > >>>> class
> > >>>>>>>>>> names*
> > >>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
> > >>>> (and
> > >>>>>>>> my team
> > >>>>>>>>>>> can
> > >>>>>>>>>>>> do the job of GCP-related operators)*
> > >>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
> > >>>>>>>> integration)*
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This is my binding (+1) vote.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> > >>>>>>>>>> Jarek.Potiuk@polidea.com>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hey Felix,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as well
> > >> (I
> > >>>>>>>> use IDE
> > >>>>>>>>>>> quite
> > >>>>>>>>>>>>> often ;) )
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> J.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> > >>>>>>>>>> <fokko@driesprong.frl
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks Kamil for putting the document together. I wasn't
> > >>>> able
> > >>>>>>>> to
> > >>>>>>>>>>> respond
> > >>>>>>>>>>>>>> earlier since I was giving Airflow workshops throughout
> > >>>> Europe
> > >>>>>>>> :-)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from contrib
> > >> into
> > >>>> a
> > >>>>>>>> single
> > >>>>>>>>>>>>>> package.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> To make it easier to the end-user, my preference would be
> > >>>> to
> > >>>>>>>> get
> > >>>>>>>>>> rid of
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>> contrib package altogether. What's in contrib or not is a
> > >>>>>>>> tricky
> > >>>>>>>>>>> question,
> > >>>>>>>>>>>>>> I think this is already reflected in the document since
> > >>>>>>>> "maturity"
> > >>>>>>>>>> is
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>>> quotes. What would be the moment to transfer the operator
> > >>>> to
> > >>>>>>>> the
> > >>>>>>>>>> core
> > >>>>>>>>>>> or
> > >>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
> > >> incubating
> > >>>>>>>> will
> > >>>>>>>>>> confuse
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>> user even more. For people who aren't as known with
> > >>>> Airflow it
> > >>>>>>>> isn't
> > >>>>>>>>>>> that
> > >>>>>>>>>>>>>> transparent which operator lives where, especially if you
> > >>>>>>>> don't use
> > >>>>>>>>>> an
> > >>>>>>>>>>> IDE
> > >>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is worth
> > >>>> changing
> > >>>>>>>> the
> > >>>>>>>>>>> public
> > >>>>>>>>>>>>>> API just for a different name.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with this. It
> > >>>> will
> > >>>>>>>> make
> > >>>>>>>>>>> easier
> > >>>>>>>>>>>>> as we will not have to define other rules as those that we
> > >>>>>>>> already
> > >>>>>>>>>> have.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's say a
> > >> +0)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I like to search on Github with t, and then search for
> > >>>>>>>> BashOperator.
> > >>>>>>>>>>> After
> > >>>>>>>>>>>>>> the change, you should search for Operator/Bash.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from your
> > >> mail
> > >>>> you
> > >>>>>>>>>> state: -
> > >>>>>>>>>>>>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator
> > >>>> could
> > >>>>>>>> become
> > >>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B solution is
> > >>>>>>>> keeping it
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
> > >>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would stay
> > >>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> My mistake. It was of course A. I think there is a little
> > >>>>>>>> confusion
> > >>>>>>>>>>> here.
> > >>>>>>>>>>>>> This case is not for changing BashOperator into Bash but
> > >> for
> > >>>>>>>> changing
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>> *module* name not the *class* name. So
> > >>>>>>>> bash_operator.BashOperator ->
> > >>>>>>>>>>>>> bash.BashOperator :) . you will still search for
> > >>>> BashOperator
> > >>>>>>>> in this
> > >>>>>>>>>>> case.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it gets
> > >>>> its
> > >>>>>>>> own
> > >>>>>>>>>>> package
> > >>>>>>>>>>>>>> or not. Besides the cloud providers, there are companies
> > >>>> like
> > >>>>>>>>>>> Databricks
> > >>>>>>>>>>>>>> and Qubole which might also end up with their own package
> > >>>>>>>> because
> > >>>>>>>>>> their
> > >>>>>>>>>>>>>> integration is getting richer over time. Moving this is a
> > >>>> bit
> > >>>>>>>>>> painful
> > >>>>>>>>>>>>>> because we need to deprecate the old path and move
> > >>>> everything
> > >>>>>>>> which
> > >>>>>>>>>>>>>> introduces noise into version control etc.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not as
> > >>>>>>>> difficult as
> > >>>>>>>>>> it
> > >>>>>>>>>>>>> seems to group operators in packages and it has numerous
> > >>>>>>>> advantages.
> > >>>>>>>>>> The
> > >>>>>>>>>>>>> nice thing about deprecations - we can do them in the
> > >>>>>>>> __init__.py of
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> packages which causes the noise fairly small (Git will
> > >>>> actually
> > >>>>>>>>>> realise
> > >>>>>>>>>>>>> that the file has been moved and you can follow that.
> > >> Maybe
> > >>>> not
> > >>>>>>>> as
> > >>>>>>>>>> super
> > >>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's not a
> > >> rocket
> > >>>>>>>> science
> > >>>>>>>>>>> either.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> *Case 7:* Python native solution
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Yes.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> ---
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Apart from all, great work!
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Cheers, Fokko
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> > >>>>>>>>>>>>>> <feluelle@pm.me.invalid
> > >>>>>>>>>>>>>>> :
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I have no strong "No" against any proposed change of
> > >> these
> > >>>>>>>> cases.
> > >>>>>>>>>> So I
> > >>>>>>>>>>>>>> go
> > >>>>>>>>>>>>>>> with +1 (non-binding).
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and your
> > >>>> intense
> > >>>>>>>> work
> > >>>>>>>>>>>>>> towards
> > >>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
> > >> creating
> > >>>>>>>> this
> > >>>>>>>>>>>>>> document. I
> > >>>>>>>>>>>>>>> like how the code is getting more and more consistent
> > >> and
> > >>>>>>>> clean :)
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Kind regards,
> > >>>>>>>>>>>>>>> Felix
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Sent from ProtonMail mobile
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> -------- Original Message --------
> > >>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> This email is calling a vote on the changes in import
> > >>>> paths.
> > >>>>>>>> It's
> > >>>>>>>>>>> been
> > >>>>>>>>>>>>>>>> discussed in
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th 6pm
> > >>>> CEST),
> > >>>>>>>> and
> > >>>>>>>>>> at
> > >>>>>>>>>>>>>> least
> > >>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The proposal to vote is based on the document from
> > >> Kamil
> > >>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The proposed solution is:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are
> > >> "well"
> > >>>>>>>> tested
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
> > >>>>>>>>>>>>>>>> - *Case 2: B:
> > >> Airflow.operators.foo_operator.FooOperator
> > >>>>>>>> could
> > >>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > >>>>>>>>>>>>>>>> - *Case 3: C:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > >>>>>>>>>>> could
> > >>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> > >>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > >>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes
> > >> on
> > >>>>>>>> class
> > >>>>>>>>>> names*
> > >>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
> > >>>> (and
> > >>>>>>>> my
> > >>>>>>>>>> team
> > >>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> This is my (binding) +1 vote.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> J.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jarek Potiuk
> > >>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> > >> Software
> > >>>>>>>> Engineer
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
> > >>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
> > >>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> --
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jarek Potiuk
> > >>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> > >>>>>>>> Engineer
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > >>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Jarek Potiuk
> > >>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> > >>>>> Engineer
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > >>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> *Kaxil Naik*
> > >>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > >>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> > >> Spark
> > >>>> &
> > >>>>>>>> Neo4j
> > >>>>>>>>> Developer
> > >>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> *Kaxil Naik*
> > >>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > >>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> Spark
> > >> &
> > >>>>>>>> Neo4j
> > >>>>>>>> Developer
> > >>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >> --
> > >>
> > >> Jarek Potiuk
> > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>
> > >> M: +48 660 796 129 <+48660796129>
> > >> [image: Polidea] <https://www.polidea.com/>
> > >>
> >
> >
>
> --
> *Kaxil Naik*
> *Big Data Consultant | DevOps Data Engineer*
> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> Developer
> *LinkedIn*: https://www.linkedin.com/in/kaxil
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Changes in import paths
Posted by Kaxil Naik <ka...@gmail.com>.
Thanks Jarek, adding my vote.
On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> Thanks Jarek, much clearer. I have voted there too.
>
> -ash
>
>
> > On 29 Jul 2019, at 08:13, Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
> >
> > Thanks Jarek, the Wiki works much better for me. I've added my vote there
> > as well.
> >
> > You've convinced me on the second case. I've changed my vote on Case 2
> from
> > A to B :-)
> >
> > Maybe we should leave the vote open a bit longer since it involves quite
> > some reading, and I would like to get some proper feedback and opinions
> on
> > this. Plus I only see two votes right now. If we don't think this
> through,
> > it might be that we're having this vote again in the near future :-)
> >
> > Cheers, Fokko
> >
> > Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
> Jarek.Potiuk@polidea.com>:
> >
> >> I updated the AIP-21 proposal in the way that should answer all the
> >> concerns in this thread.
> >>
> >> NOTE! There is an action for all of those who are interested: Please
> >> fill-in your voting in the voting table till Tuesday 30th of July 6pm
> CEST
> >> (5pm BST, 9 am PST).
> >>
> >> I created a summary of the proposal
> >> <
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> >>>
> >> in table form - which contains also examples for each case.
> >>
> >> I added a voting table
> >> <
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> >>>
> >> where you should add your votes:
> >>
> >> I propose this voting mechanism:
> >>
> >> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 pm BST,
> 9am
> >> PST) .
> >>
> >> After the choice, the final consistent set of choices will be announced
> >> (taking into account majority of binding vote, also including potential
> >> vetos and consistency between the choices. Non-binding votes will be
> taken
> >> into account in case there is a draw. The final set of choices will be
> >> announced at devlist thread
> >> <
> >>
> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> >>>
> >> after
> >> the voting completes.
> >>
> >> Unless there is a veto raised to the final proposal, the final proposal
> is
> >> accepted by Lazy Consensus
> >> <https://community.apache.org/committers/lazyConsensus.html> on
> *Friday*
> >> 02
> >> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
> >>
> >> Please let me know if what I proposed can be further improved to avoid
> >> chaos.
> >>
> >> J.
> >>
> >> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> >> wrote:
> >>
> >>> Ok. I see that indeed voting could have been organised a bit better -
> dev
> >>> list does not seem to cope well with such a complex case :).
> >>>
> >>> As Kamil noticed - we already have a formal AIP-21
> >>> <
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >>>
> >>> in the Wiki - which I should have mentioned before. I have not much
> time
> >>> today - but tomorrow I will try to summarise the options - based on
> some
> >>> real code examples (to make it easier to digest). I think the best
> >> approach
> >>> will be to make a voting matrix in the AIP-21 Wiki page where people
> will
> >>> be able to state their preferences for each case (by adding a row to
> the
> >>> table). I think that might provide much better clarity and a single
> place
> >>> which will show the current aggregated state of voting.
> >>>
> >>> However - as Ash also stated - some of the options are inter-dependent
> so
> >>> at the end we will have to adjust the results in case they are not
> >>> consistent - and make final voting on aggregated set of cases - but I
> >>> think in this case following "lasy consensus" - i,e. if noone objects
> to
> >>> final set of cases, we will proceed with it..
> >>>
> >>> Do you think that will work better ?
> >>>
> >>> J.
> >>>
> >>> Principal Software Engineer
> >>> Phone: +48660796129
> >>>
> >>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> >>> kamil.bregula@polidea.com> napisał:
> >>>
> >>>> Document is also available as AIP on wiki:
> >>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >>>>
> >>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko <fokko@driesprong.frl
> >
> >>>> wrote:
> >>>>
> >>>>> I agree with all, this is a bit chaotic. For the sake of clarity, I
> >>>> would
> >>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a
> >> structural
> >>>>> code improvement AIP and then and add a matrix of users/cases where
> >>>>> everybody can add his vote per case. This gives also the opportunity
> >> for
> >>>>> changing your vote on a specific case. WDYT?
> >>>>>
> >>>>> Cheers, Fokko
> >>>>>
> >>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <ci...@gmail.com>:
> >>>>>
> >>>>>> I agree with Ash. It is clear to vote on each case rather than all
> >>>>>> together.
> >>>>>>
> >>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <as...@apache.org>
> >>>>> wrote:
> >>>>>>
> >>>>>>> A number of proposals overlap or add on to each other, so I think
> >>>> (?) a
> >>>>>>> single vote makes sense.
> >>>>>>>
> >>>>>>> To be clear, my vote is -1 until it's clear exactly what we are
> >>>> voting
> >>>>> on.
> >>>>>>>
> >>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <ka...@gmail.com>
> >>>> wrote:
> >>>>>>>> I agree with Ash and instead of voting on all 7 Cases together,
> >> how
> >>>>>>>> about
> >>>>>>>> separate vote emails for all the cases? Each email can have the
> >>>>>>>> description
> >>>>>>>> copied over from the doc.
> >>>>>>>>
> >>>>>>>> It would probably be a bit easier to track a decision in the
> >> future
> >>>> as
> >>>>>>>> well.
> >>>>>>>>
> >>>>>>>> What do you guys think? @Jarek Potiuk <Ja...@polidea.com>
> >>>>>>>> @Driesprong,
> >>>>>>>> Fokko <fo...@driesprong.frl>
> >>>>>>>>
> >>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <ka...@gmail.com>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
> >>>>>>>>>
> >>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
> >>>>>>>>>
> >>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
> >>>>>>>>>
> >>>>>>>>> *Case #4* Separate namespace for resources :
> >>>>>>>>> *Solution A*
> >>>>>>>>>
> >>>>>>>>> *Case #5* *Operator : *Solution A*
> >>>>>>>>>
> >>>>>>>>> *Case #7* : *Solution A*
> >>>>>>>>>
> >>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> >>>>>>>> <dp...@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I have tried to add some clarification to Jarek's summary, but
> >> I
> >>>> am
> >>>>>>>> a
> >>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully Jarek
> >> can
> >>>>>>>> further
> >>>>>>>>>> clarify.
> >>>>>>>>>>
> >>>>>>>>>> According to my reading, in this motion cases 4,5,6 are all
> >>>> either
> >>>>>>>>>> proposal
> >>>>>>>>>> *rejections* or otherwise have no effect, so attention can be
> >>>>>>>> focused on
> >>>>>>>>>> cases 1,2,3 and 7.
> >>>>>>>>>>
> >>>>>>>>>> *Motion is to adopt the following:*
> >>>>>>>>>>
> >>>>>>>>>> Case 1 - Solution A - Get rid of contrib
> >>>>>>>>>> All objects will be moved out of contrib folder and into
> >>>> respective
> >>>>>>>>>> non-contrib folder.
> >>>>>>>>>>
> >>>>>>>>>> Case 2 - Solution A - Remove object type suffix from modules
> >>>>>>>>>> Example:
> >>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
> >>>>>>>>>> airflow.operators.foo.FooOperator
> >>>>>>>>>> Example:
> >>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
> >>>> airflow.hooks.foo.FooHook
> >>>>>>>>>>
> >>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
> >>>>>>>>>> *I am not sure exactly what the proposal is.*
> >>>>>>>>>> Example case is
> >>>> airflow/contrib/operators/gcp_bigtable_operator.py
> >>>>>>>>>> Since motion adopts case 1 solution A, we can assume it is now
> >>>> named
> >>>>>>>> like
> >>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
> >>>>>>>>>> So what is proposed for this case?
> >>>>>>>>>>
> >>>>>>>>>> Case 4 - Solution B - *no change*
> >>>>>>>>>> This proposal was about namespaces but the motion is to reject
> >>>> the
> >>>>>>>>>> proposal.
> >>>>>>>>>>
> >>>>>>>>>> Case 5 - Solution B - *no change*
> >>>>>>>>>> The proposal was to remove class type suffix e.g. remove the
> >>>>>>>> Operator in
> >>>>>>>>>> FooOperator, but the motion is to reject this proposal -- i.e.
> >>>>>>>> motion is
> >>>>>>>>>> no
> >>>>>>>>>> change on this case, i.e. we resolve to keep "Operator" (and
> >>>>>>>> "Sensor")
> >>>>>>>>>> suffixes on class names
> >>>>>>>>>>
> >>>>>>>>>> Case 6 - *No change*
> >>>>>>>>>> This case concerns object naming inconsistencies, but motion
> >> does
> >>>>>>>> not
> >>>>>>>>>> enact
> >>>>>>>>>> any specific proposal.
> >>>>>>>>>>
> >>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
> >> deprecation
> >>>>>>>> warnings
> >>>>>>>>>> (instead of zope)
> >>>>>>>>>> Example:
> >>>>>>>>>>
> >>>>>>>>>> # in old_package/old_mod.py
> >>>>>>>>>>
> >>>>>>>>>> import warnings
> >>>>>>>>>> from new_package.new_mod import *
> >>>>>>>>>> warnings.warn("old_package.old_mod has moved to
> >>>> new_package.new_mod.
> >>>>>>>>>> Import
> >>>>>>>>>> of "
> >>>>>>>>>> "old_package.old_mod will become unsupported in version 2",
> >>>>>>>>>> DeprecationWarning, 2)
> >>>>>>>>>>
> >>>>>>>>>> Reference implementation here:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> >>>>>>>>>>
> >>>>>>>>>> I am very happy that this motion now gets rid of contrib.
> >> Thank
> >>>> you
> >>>>>>>>>> Sergio
> >>>>>>>>>> for turning the tide with your effective argumentation ;)
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
> >>>> ash@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Could someone summarise what the proposed name changes are,
> >>>> here,
> >>>>>>>> in
> >>>>>>>>>> this
> >>>>>>>>>>> thread? Pointing at a google doc and only giving partial
> >>>> examples
> >>>>>>>> is 1)
> >>>>>>>>>> a
> >>>>>>>>>>> bit difficult to quickly work out what we are voting on, and
> >> 2)
> >>>>>>>> using a
> >>>>>>>>>>> google doc isn't great for a longer term record.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Ash
> >>>>>>>>>>>
> >>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
> >>>>>>>> <Ja...@polidea.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :) . I
> >>>> would
> >>>>>>>> like
> >>>>>>>>>> to
> >>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three +1
> >>>>>>>> (binding)
> >>>>>>>>>> votes
> >>>>>>>>>>> are
> >>>>>>>>>>>> needed to pass. I marked in red the modifications to the
> >>>>>>>> previous
> >>>>>>>>>>> proposal.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Here is the proposal (details here
> >>>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> >>>>>>>>>>>>
> >>>>>>>>>>>> )
> >>>>>>>>>>>>
> >>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
> >>>>>>>>>>>> - *Case 2: A: Airflow.operators.foo_operator.FooOperator
> >>>> could
> >>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> >>>>>>>>>>>> - *Case 3: C:
> >>>>>>>>>>>>
> >>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >>>>>>>>>> could
> >>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> >>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> >>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
> >>>> class
> >>>>>>>>>> names*
> >>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
> >>>> (and
> >>>>>>>> my team
> >>>>>>>>>>> can
> >>>>>>>>>>>> do the job of GCP-related operators)*
> >>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
> >>>>>>>> integration)*
> >>>>>>>>>>>>
> >>>>>>>>>>>> This is my binding (+1) vote.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> >>>>>>>>>> Jarek.Potiuk@polidea.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hey Felix,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as well
> >> (I
> >>>>>>>> use IDE
> >>>>>>>>>>> quite
> >>>>>>>>>>>>> often ;) )
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> J.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> >>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks Kamil for putting the document together. I wasn't
> >>>> able
> >>>>>>>> to
> >>>>>>>>>>> respond
> >>>>>>>>>>>>>> earlier since I was giving Airflow workshops throughout
> >>>> Europe
> >>>>>>>> :-)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from contrib
> >> into
> >>>> a
> >>>>>>>> single
> >>>>>>>>>>>>>> package.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> To make it easier to the end-user, my preference would be
> >>>> to
> >>>>>>>> get
> >>>>>>>>>> rid of
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>> contrib package altogether. What's in contrib or not is a
> >>>>>>>> tricky
> >>>>>>>>>>> question,
> >>>>>>>>>>>>>> I think this is already reflected in the document since
> >>>>>>>> "maturity"
> >>>>>>>>>> is
> >>>>>>>>>>> in
> >>>>>>>>>>>>>> quotes. What would be the moment to transfer the operator
> >>>> to
> >>>>>>>> the
> >>>>>>>>>> core
> >>>>>>>>>>> or
> >>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
> >> incubating
> >>>>>>>> will
> >>>>>>>>>> confuse
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>> user even more. For people who aren't as known with
> >>>> Airflow it
> >>>>>>>> isn't
> >>>>>>>>>>> that
> >>>>>>>>>>>>>> transparent which operator lives where, especially if you
> >>>>>>>> don't use
> >>>>>>>>>> an
> >>>>>>>>>>> IDE
> >>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is worth
> >>>> changing
> >>>>>>>> the
> >>>>>>>>>>> public
> >>>>>>>>>>>>>> API just for a different name.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with this. It
> >>>> will
> >>>>>>>> make
> >>>>>>>>>>> easier
> >>>>>>>>>>>>> as we will not have to define other rules as those that we
> >>>>>>>> already
> >>>>>>>>>> have.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's say a
> >> +0)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I like to search on Github with t, and then search for
> >>>>>>>> BashOperator.
> >>>>>>>>>>> After
> >>>>>>>>>>>>>> the change, you should search for Operator/Bash.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from your
> >> mail
> >>>> you
> >>>>>>>>>> state: -
> >>>>>>>>>>>>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator
> >>>> could
> >>>>>>>> become
> >>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B solution is
> >>>>>>>> keeping it
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
> >>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would stay
> >>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> My mistake. It was of course A. I think there is a little
> >>>>>>>> confusion
> >>>>>>>>>>> here.
> >>>>>>>>>>>>> This case is not for changing BashOperator into Bash but
> >> for
> >>>>>>>> changing
> >>>>>>>>>>> the
> >>>>>>>>>>>>> *module* name not the *class* name. So
> >>>>>>>> bash_operator.BashOperator ->
> >>>>>>>>>>>>> bash.BashOperator :) . you will still search for
> >>>> BashOperator
> >>>>>>>> in this
> >>>>>>>>>>> case.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it gets
> >>>> its
> >>>>>>>> own
> >>>>>>>>>>> package
> >>>>>>>>>>>>>> or not. Besides the cloud providers, there are companies
> >>>> like
> >>>>>>>>>>> Databricks
> >>>>>>>>>>>>>> and Qubole which might also end up with their own package
> >>>>>>>> because
> >>>>>>>>>> their
> >>>>>>>>>>>>>> integration is getting richer over time. Moving this is a
> >>>> bit
> >>>>>>>>>> painful
> >>>>>>>>>>>>>> because we need to deprecate the old path and move
> >>>> everything
> >>>>>>>> which
> >>>>>>>>>>>>>> introduces noise into version control etc.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not as
> >>>>>>>> difficult as
> >>>>>>>>>> it
> >>>>>>>>>>>>> seems to group operators in packages and it has numerous
> >>>>>>>> advantages.
> >>>>>>>>>> The
> >>>>>>>>>>>>> nice thing about deprecations - we can do them in the
> >>>>>>>> __init__.py of
> >>>>>>>>>> the
> >>>>>>>>>>>>> packages which causes the noise fairly small (Git will
> >>>> actually
> >>>>>>>>>> realise
> >>>>>>>>>>>>> that the file has been moved and you can follow that.
> >> Maybe
> >>>> not
> >>>>>>>> as
> >>>>>>>>>> super
> >>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's not a
> >> rocket
> >>>>>>>> science
> >>>>>>>>>>> either.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Case 7:* Python native solution
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yes.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Apart from all, great work!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> >>>>>>>>>>>>>> <feluelle@pm.me.invalid
> >>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I have no strong "No" against any proposed change of
> >> these
> >>>>>>>> cases.
> >>>>>>>>>> So I
> >>>>>>>>>>>>>> go
> >>>>>>>>>>>>>>> with +1 (non-binding).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and your
> >>>> intense
> >>>>>>>> work
> >>>>>>>>>>>>>> towards
> >>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
> >> creating
> >>>>>>>> this
> >>>>>>>>>>>>>> document. I
> >>>>>>>>>>>>>>> like how the code is getting more and more consistent
> >> and
> >>>>>>>> clean :)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Kind regards,
> >>>>>>>>>>>>>>> Felix
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Sent from ProtonMail mobile
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -------- Original Message --------
> >>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This email is calling a vote on the changes in import
> >>>> paths.
> >>>>>>>> It's
> >>>>>>>>>>> been
> >>>>>>>>>>>>>>>> discussed in
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th 6pm
> >>>> CEST),
> >>>>>>>> and
> >>>>>>>>>> at
> >>>>>>>>>>>>>> least
> >>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The proposal to vote is based on the document from
> >> Kamil
> >>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The proposed solution is:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are
> >> "well"
> >>>>>>>> tested
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
> >>>>>>>>>>>>>>>> - *Case 2: B:
> >> Airflow.operators.foo_operator.FooOperator
> >>>>>>>> could
> >>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> >>>>>>>>>>>>>>>> - *Case 3: C:
> >>>>>>>>>>>>>>>>
> >>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >>>>>>>>>>> could
> >>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> >>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> >>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes
> >> on
> >>>>>>>> class
> >>>>>>>>>> names*
> >>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
> >>>> (and
> >>>>>>>> my
> >>>>>>>>>> team
> >>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This is my (binding) +1 vote.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> J.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jarek Potiuk
> >>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> >> Software
> >>>>>>>> Engineer
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
> >>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
> >>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jarek Potiuk
> >>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> >>>>>>>> Engineer
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>>
> >>>>>>>>>>>> Jarek Potiuk
> >>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> >>>>> Engineer
> >>>>>>>>>>>>
> >>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> *Kaxil Naik*
> >>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> >>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> >> Spark
> >>>> &
> >>>>>>>> Neo4j
> >>>>>>>>> Developer
> >>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> *Kaxil Naik*
> >>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> >>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark
> >> &
> >>>>>>>> Neo4j
> >>>>>>>> Developer
> >>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>
> >> M: +48 660 796 129 <+48660796129>
> >> [image: Polidea] <https://www.polidea.com/>
> >>
>
>
--
*Kaxil Naik*
*Big Data Consultant | DevOps Data Engineer*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*LinkedIn*: https://www.linkedin.com/in/kaxil
Re: [VOTE] Changes in import paths
Posted by Ash Berlin-Taylor <as...@apache.org>.
Thanks Jarek, much clearer. I have voted there too.
-ash
> On 29 Jul 2019, at 08:13, Driesprong, Fokko <fo...@driesprong.frl> wrote:
>
> Thanks Jarek, the Wiki works much better for me. I've added my vote there
> as well.
>
> You've convinced me on the second case. I've changed my vote on Case 2 from
> A to B :-)
>
> Maybe we should leave the vote open a bit longer since it involves quite
> some reading, and I would like to get some proper feedback and opinions on
> this. Plus I only see two votes right now. If we don't think this through,
> it might be that we're having this vote again in the near future :-)
>
> Cheers, Fokko
>
> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <Ja...@polidea.com>:
>
>> I updated the AIP-21 proposal in the way that should answer all the
>> concerns in this thread.
>>
>> NOTE! There is an action for all of those who are interested: Please
>> fill-in your voting in the voting table till Tuesday 30th of July 6pm CEST
>> (5pm BST, 9 am PST).
>>
>> I created a summary of the proposal
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
>>>
>> in table form - which contains also examples for each case.
>>
>> I added a voting table
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
>>>
>> where you should add your votes:
>>
>> I propose this voting mechanism:
>>
>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 pm BST, 9am
>> PST) .
>>
>> After the choice, the final consistent set of choices will be announced
>> (taking into account majority of binding vote, also including potential
>> vetos and consistency between the choices. Non-binding votes will be taken
>> into account in case there is a draw. The final set of choices will be
>> announced at devlist thread
>> <
>> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
>>>
>> after
>> the voting completes.
>>
>> Unless there is a veto raised to the final proposal, the final proposal is
>> accepted by Lazy Consensus
>> <https://community.apache.org/committers/lazyConsensus.html> on *Friday*
>> 02
>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
>>
>> Please let me know if what I proposed can be further improved to avoid
>> chaos.
>>
>> J.
>>
>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> Ok. I see that indeed voting could have been organised a bit better - dev
>>> list does not seem to cope well with such a complex case :).
>>>
>>> As Kamil noticed - we already have a formal AIP-21
>>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>>>
>>> in the Wiki - which I should have mentioned before. I have not much time
>>> today - but tomorrow I will try to summarise the options - based on some
>>> real code examples (to make it easier to digest). I think the best
>> approach
>>> will be to make a voting matrix in the AIP-21 Wiki page where people will
>>> be able to state their preferences for each case (by adding a row to the
>>> table). I think that might provide much better clarity and a single place
>>> which will show the current aggregated state of voting.
>>>
>>> However - as Ash also stated - some of the options are inter-dependent so
>>> at the end we will have to adjust the results in case they are not
>>> consistent - and make final voting on aggregated set of cases - but I
>>> think in this case following "lasy consensus" - i,e. if noone objects to
>>> final set of cases, we will proceed with it..
>>>
>>> Do you think that will work better ?
>>>
>>> J.
>>>
>>> Principal Software Engineer
>>> Phone: +48660796129
>>>
>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
>>> kamil.bregula@polidea.com> napisał:
>>>
>>>> Document is also available as AIP on wiki:
>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>>>>
>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko <fo...@driesprong.frl>
>>>> wrote:
>>>>
>>>>> I agree with all, this is a bit chaotic. For the sake of clarity, I
>>>> would
>>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a
>> structural
>>>>> code improvement AIP and then and add a matrix of users/cases where
>>>>> everybody can add his vote per case. This gives also the opportunity
>> for
>>>>> changing your vote on a specific case. WDYT?
>>>>>
>>>>> Cheers, Fokko
>>>>>
>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <ci...@gmail.com>:
>>>>>
>>>>>> I agree with Ash. It is clear to vote on each case rather than all
>>>>>> together.
>>>>>>
>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <as...@apache.org>
>>>>> wrote:
>>>>>>
>>>>>>> A number of proposals overlap or add on to each other, so I think
>>>> (?) a
>>>>>>> single vote makes sense.
>>>>>>>
>>>>>>> To be clear, my vote is -1 until it's clear exactly what we are
>>>> voting
>>>>> on.
>>>>>>>
>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <ka...@gmail.com>
>>>> wrote:
>>>>>>>> I agree with Ash and instead of voting on all 7 Cases together,
>> how
>>>>>>>> about
>>>>>>>> separate vote emails for all the cases? Each email can have the
>>>>>>>> description
>>>>>>>> copied over from the doc.
>>>>>>>>
>>>>>>>> It would probably be a bit easier to track a decision in the
>> future
>>>> as
>>>>>>>> well.
>>>>>>>>
>>>>>>>> What do you guys think? @Jarek Potiuk <Ja...@polidea.com>
>>>>>>>> @Driesprong,
>>>>>>>> Fokko <fo...@driesprong.frl>
>>>>>>>>
>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <ka...@gmail.com>
>>>>> wrote:
>>>>>>>>
>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
>>>>>>>>>
>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
>>>>>>>>>
>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
>>>>>>>>>
>>>>>>>>> *Case #4* Separate namespace for resources :
>>>>>>>>> *Solution A*
>>>>>>>>>
>>>>>>>>> *Case #5* *Operator : *Solution A*
>>>>>>>>>
>>>>>>>>> *Case #7* : *Solution A*
>>>>>>>>>
>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
>>>>>>>> <dp...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I have tried to add some clarification to Jarek's summary, but
>> I
>>>> am
>>>>>>>> a
>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully Jarek
>> can
>>>>>>>> further
>>>>>>>>>> clarify.
>>>>>>>>>>
>>>>>>>>>> According to my reading, in this motion cases 4,5,6 are all
>>>> either
>>>>>>>>>> proposal
>>>>>>>>>> *rejections* or otherwise have no effect, so attention can be
>>>>>>>> focused on
>>>>>>>>>> cases 1,2,3 and 7.
>>>>>>>>>>
>>>>>>>>>> *Motion is to adopt the following:*
>>>>>>>>>>
>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
>>>>>>>>>> All objects will be moved out of contrib folder and into
>>>> respective
>>>>>>>>>> non-contrib folder.
>>>>>>>>>>
>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from modules
>>>>>>>>>> Example:
>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
>>>>>>>>>> airflow.operators.foo.FooOperator
>>>>>>>>>> Example:
>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
>>>> airflow.hooks.foo.FooHook
>>>>>>>>>>
>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
>>>>>>>>>> *I am not sure exactly what the proposal is.*
>>>>>>>>>> Example case is
>>>> airflow/contrib/operators/gcp_bigtable_operator.py
>>>>>>>>>> Since motion adopts case 1 solution A, we can assume it is now
>>>> named
>>>>>>>> like
>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
>>>>>>>>>> So what is proposed for this case?
>>>>>>>>>>
>>>>>>>>>> Case 4 - Solution B - *no change*
>>>>>>>>>> This proposal was about namespaces but the motion is to reject
>>>> the
>>>>>>>>>> proposal.
>>>>>>>>>>
>>>>>>>>>> Case 5 - Solution B - *no change*
>>>>>>>>>> The proposal was to remove class type suffix e.g. remove the
>>>>>>>> Operator in
>>>>>>>>>> FooOperator, but the motion is to reject this proposal -- i.e.
>>>>>>>> motion is
>>>>>>>>>> no
>>>>>>>>>> change on this case, i.e. we resolve to keep "Operator" (and
>>>>>>>> "Sensor")
>>>>>>>>>> suffixes on class names
>>>>>>>>>>
>>>>>>>>>> Case 6 - *No change*
>>>>>>>>>> This case concerns object naming inconsistencies, but motion
>> does
>>>>>>>> not
>>>>>>>>>> enact
>>>>>>>>>> any specific proposal.
>>>>>>>>>>
>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
>> deprecation
>>>>>>>> warnings
>>>>>>>>>> (instead of zope)
>>>>>>>>>> Example:
>>>>>>>>>>
>>>>>>>>>> # in old_package/old_mod.py
>>>>>>>>>>
>>>>>>>>>> import warnings
>>>>>>>>>> from new_package.new_mod import *
>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
>>>> new_package.new_mod.
>>>>>>>>>> Import
>>>>>>>>>> of "
>>>>>>>>>> "old_package.old_mod will become unsupported in version 2",
>>>>>>>>>> DeprecationWarning, 2)
>>>>>>>>>>
>>>>>>>>>> Reference implementation here:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
>>>>>>>>>>
>>>>>>>>>> I am very happy that this motion now gets rid of contrib.
>> Thank
>>>> you
>>>>>>>>>> Sergio
>>>>>>>>>> for turning the tide with your effective argumentation ;)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
>>>> ash@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Could someone summarise what the proposed name changes are,
>>>> here,
>>>>>>>> in
>>>>>>>>>> this
>>>>>>>>>>> thread? Pointing at a google doc and only giving partial
>>>> examples
>>>>>>>> is 1)
>>>>>>>>>> a
>>>>>>>>>>> bit difficult to quickly work out what we are voting on, and
>> 2)
>>>>>>>> using a
>>>>>>>>>>> google doc isn't great for a longer term record.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Ash
>>>>>>>>>>>
>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
>>>>>>>> <Ja...@polidea.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :) . I
>>>> would
>>>>>>>> like
>>>>>>>>>> to
>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three +1
>>>>>>>> (binding)
>>>>>>>>>> votes
>>>>>>>>>>> are
>>>>>>>>>>>> needed to pass. I marked in red the modifications to the
>>>>>>>> previous
>>>>>>>>>>> proposal.
>>>>>>>>>>>>
>>>>>>>>>>>> Here is the proposal (details here
>>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
>>>>>>>>>>>>
>>>>>>>>>>>> )
>>>>>>>>>>>>
>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
>>>>>>>>>>>> - *Case 2: A: Airflow.operators.foo_operator.FooOperator
>>>> could
>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
>>>>>>>>>>>> - *Case 3: C:
>>>>>>>>>>>>
>>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>>>>>>>>>> could
>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
>>>> class
>>>>>>>>>> names*
>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
>>>> (and
>>>>>>>> my team
>>>>>>>>>>> can
>>>>>>>>>>>> do the job of GCP-related operators)*
>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
>>>>>>>> integration)*
>>>>>>>>>>>>
>>>>>>>>>>>> This is my binding (+1) vote.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
>>>>>>>>>> Jarek.Potiuk@polidea.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey Felix,
>>>>>>>>>>>>>
>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as well
>> (I
>>>>>>>> use IDE
>>>>>>>>>>> quite
>>>>>>>>>>>>> often ;) )
>>>>>>>>>>>>>
>>>>>>>>>>>>> J.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I wasn't
>>>> able
>>>>>>>> to
>>>>>>>>>>> respond
>>>>>>>>>>>>>> earlier since I was giving Airflow workshops throughout
>>>> Europe
>>>>>>>> :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from contrib
>> into
>>>> a
>>>>>>>> single
>>>>>>>>>>>>>> package.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To make it easier to the end-user, my preference would be
>>>> to
>>>>>>>> get
>>>>>>>>>> rid of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> contrib package altogether. What's in contrib or not is a
>>>>>>>> tricky
>>>>>>>>>>> question,
>>>>>>>>>>>>>> I think this is already reflected in the document since
>>>>>>>> "maturity"
>>>>>>>>>> is
>>>>>>>>>>> in
>>>>>>>>>>>>>> quotes. What would be the moment to transfer the operator
>>>> to
>>>>>>>> the
>>>>>>>>>> core
>>>>>>>>>>> or
>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
>> incubating
>>>>>>>> will
>>>>>>>>>> confuse
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> user even more. For people who aren't as known with
>>>> Airflow it
>>>>>>>> isn't
>>>>>>>>>>> that
>>>>>>>>>>>>>> transparent which operator lives where, especially if you
>>>>>>>> don't use
>>>>>>>>>> an
>>>>>>>>>>> IDE
>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is worth
>>>> changing
>>>>>>>> the
>>>>>>>>>>> public
>>>>>>>>>>>>>> API just for a different name.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with this. It
>>>> will
>>>>>>>> make
>>>>>>>>>>> easier
>>>>>>>>>>>>> as we will not have to define other rules as those that we
>>>>>>>> already
>>>>>>>>>> have.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's say a
>> +0)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I like to search on Github with t, and then search for
>>>>>>>> BashOperator.
>>>>>>>>>>> After
>>>>>>>>>>>>>> the change, you should search for Operator/Bash.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from your
>> mail
>>>> you
>>>>>>>>>> state: -
>>>>>>>>>>>>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator
>>>> could
>>>>>>>> become
>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B solution is
>>>>>>>> keeping it
>>>>>>>>>>> the
>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would stay
>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a little
>>>>>>>> confusion
>>>>>>>>>>> here.
>>>>>>>>>>>>> This case is not for changing BashOperator into Bash but
>> for
>>>>>>>> changing
>>>>>>>>>>> the
>>>>>>>>>>>>> *module* name not the *class* name. So
>>>>>>>> bash_operator.BashOperator ->
>>>>>>>>>>>>> bash.BashOperator :) . you will still search for
>>>> BashOperator
>>>>>>>> in this
>>>>>>>>>>> case.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it gets
>>>> its
>>>>>>>> own
>>>>>>>>>>> package
>>>>>>>>>>>>>> or not. Besides the cloud providers, there are companies
>>>> like
>>>>>>>>>>> Databricks
>>>>>>>>>>>>>> and Qubole which might also end up with their own package
>>>>>>>> because
>>>>>>>>>> their
>>>>>>>>>>>>>> integration is getting richer over time. Moving this is a
>>>> bit
>>>>>>>>>> painful
>>>>>>>>>>>>>> because we need to deprecate the old path and move
>>>> everything
>>>>>>>> which
>>>>>>>>>>>>>> introduces noise into version control etc.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not as
>>>>>>>> difficult as
>>>>>>>>>> it
>>>>>>>>>>>>> seems to group operators in packages and it has numerous
>>>>>>>> advantages.
>>>>>>>>>> The
>>>>>>>>>>>>> nice thing about deprecations - we can do them in the
>>>>>>>> __init__.py of
>>>>>>>>>> the
>>>>>>>>>>>>> packages which causes the noise fairly small (Git will
>>>> actually
>>>>>>>>>> realise
>>>>>>>>>>>>> that the file has been moved and you can follow that.
>> Maybe
>>>> not
>>>>>>>> as
>>>>>>>>>> super
>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's not a
>> rocket
>>>>>>>> science
>>>>>>>>>>> either.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Case 7:* Python native solution
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Apart from all, great work!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
>>>>>>>>>>>>>> <feluelle@pm.me.invalid
>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have no strong "No" against any proposed change of
>> these
>>>>>>>> cases.
>>>>>>>>>> So I
>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>> with +1 (non-binding).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and your
>>>> intense
>>>>>>>> work
>>>>>>>>>>>>>> towards
>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
>> creating
>>>>>>>> this
>>>>>>>>>>>>>> document. I
>>>>>>>>>>>>>>> like how the code is getting more and more consistent
>> and
>>>>>>>> clean :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>> Felix
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sent from ProtonMail mobile
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -------- Original Message --------
>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This email is calling a vote on the changes in import
>>>> paths.
>>>>>>>> It's
>>>>>>>>>>> been
>>>>>>>>>>>>>>>> discussed in
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th 6pm
>>>> CEST),
>>>>>>>> and
>>>>>>>>>> at
>>>>>>>>>>>>>> least
>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The proposal to vote is based on the document from
>> Kamil
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The proposed solution is:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are
>> "well"
>>>>>>>> tested
>>>>>>>>>> and
>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
>>>>>>>>>>>>>>>> - *Case 2: B:
>> Airflow.operators.foo_operator.FooOperator
>>>>>>>> could
>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
>>>>>>>>>>>>>>>> - *Case 3: C:
>>>>>>>>>>>>>>>>
>>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>>>>>>>>>>> could
>>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes
>> on
>>>>>>>> class
>>>>>>>>>> names*
>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
>>>> (and
>>>>>>>> my
>>>>>>>>>> team
>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
>> Software
>>>>>>>> Engineer
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
>>>>>>>> Engineer
>>>>>>>>>>>>>
>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
>>>>> Engineer
>>>>>>>>>>>>
>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Kaxil Naik*
>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
>> Spark
>>>> &
>>>>>>>> Neo4j
>>>>>>>>> Developer
>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Kaxil Naik*
>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark
>> &
>>>>>>>> Neo4j
>>>>>>>> Developer
>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
Re: [VOTE] Changes in import paths
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Thanks Jarek, the Wiki works much better for me. I've added my vote there
as well.
You've convinced me on the second case. I've changed my vote on Case 2 from
A to B :-)
Maybe we should leave the vote open a bit longer since it involves quite
some reading, and I would like to get some proper feedback and opinions on
this. Plus I only see two votes right now. If we don't think this through,
it might be that we're having this vote again in the near future :-)
Cheers, Fokko
Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <Ja...@polidea.com>:
> I updated the AIP-21 proposal in the way that should answer all the
> concerns in this thread.
>
> NOTE! There is an action for all of those who are interested: Please
> fill-in your voting in the voting table till Tuesday 30th of July 6pm CEST
> (5pm BST, 9 am PST).
>
> I created a summary of the proposal
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> >
> in table form - which contains also examples for each case.
>
> I added a voting table
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> >
> where you should add your votes:
>
> I propose this voting mechanism:
>
> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 pm BST, 9am
> PST) .
>
> After the choice, the final consistent set of choices will be announced
> (taking into account majority of binding vote, also including potential
> vetos and consistency between the choices. Non-binding votes will be taken
> into account in case there is a draw. The final set of choices will be
> announced at devlist thread
> <
> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> >
> after
> the voting completes.
>
> Unless there is a veto raised to the final proposal, the final proposal is
> accepted by Lazy Consensus
> <https://community.apache.org/committers/lazyConsensus.html> on *Friday*
> 02
> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
>
> Please let me know if what I proposed can be further improved to avoid
> chaos.
>
> J.
>
> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > Ok. I see that indeed voting could have been organised a bit better - dev
> > list does not seem to cope well with such a complex case :).
> >
> > As Kamil noticed - we already have a formal AIP-21
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >
> > in the Wiki - which I should have mentioned before. I have not much time
> > today - but tomorrow I will try to summarise the options - based on some
> > real code examples (to make it easier to digest). I think the best
> approach
> > will be to make a voting matrix in the AIP-21 Wiki page where people will
> > be able to state their preferences for each case (by adding a row to the
> > table). I think that might provide much better clarity and a single place
> > which will show the current aggregated state of voting.
> >
> > However - as Ash also stated - some of the options are inter-dependent so
> > at the end we will have to adjust the results in case they are not
> > consistent - and make final voting on aggregated set of cases - but I
> > think in this case following "lasy consensus" - i,e. if noone objects to
> > final set of cases, we will proceed with it..
> >
> > Do you think that will work better ?
> >
> > J.
> >
> > Principal Software Engineer
> > Phone: +48660796129
> >
> > sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> > kamil.bregula@polidea.com> napisał:
> >
> >> Document is also available as AIP on wiki:
> >>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >>
> >> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko <fo...@driesprong.frl>
> >> wrote:
> >>
> >> > I agree with all, this is a bit chaotic. For the sake of clarity, I
> >> would
> >> > suggest moving the Google Doc to the Wiki as an AIP. Create a
> structural
> >> > code improvement AIP and then and add a matrix of users/cases where
> >> > everybody can add his vote per case. This gives also the opportunity
> for
> >> > changing your vote on a specific case. WDYT?
> >> >
> >> > Cheers, Fokko
> >> >
> >> > Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <ci...@gmail.com>:
> >> >
> >> > > I agree with Ash. It is clear to vote on each case rather than all
> >> > > together.
> >> > >
> >> > > On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <as...@apache.org>
> >> > wrote:
> >> > >
> >> > >> A number of proposals overlap or add on to each other, so I think
> >> (?) a
> >> > >> single vote makes sense.
> >> > >>
> >> > >> To be clear, my vote is -1 until it's clear exactly what we are
> >> voting
> >> > on.
> >> > >>
> >> > >> On 26 July 2019 20:39:56 BST, Kaxil Naik <ka...@gmail.com>
> >> wrote:
> >> > >> >I agree with Ash and instead of voting on all 7 Cases together,
> how
> >> > >> >about
> >> > >> >separate vote emails for all the cases? Each email can have the
> >> > >> >description
> >> > >> >copied over from the doc.
> >> > >> >
> >> > >> >It would probably be a bit easier to track a decision in the
> future
> >> as
> >> > >> >well.
> >> > >> >
> >> > >> >What do you guys think? @Jarek Potiuk <Ja...@polidea.com>
> >> > >> >@Driesprong,
> >> > >> >Fokko <fo...@driesprong.frl>
> >> > >> >
> >> > >> >On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <ka...@gmail.com>
> >> > wrote:
> >> > >> >
> >> > >> >> *Case #1* airflow.contrib.{resources} : *Solution A *
> >> > >> >>
> >> > >> >> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
> >> > >> >>
> >> > >> >> *Case #3* {aws/azure/gcp}_* : *Solution C*
> >> > >> >>
> >> > >> >> *Case #4* Separate namespace for resources :
> >> > >> >> *Solution A*
> >> > >> >>
> >> > >> >> *Case #5* *Operator : *Solution A*
> >> > >> >>
> >> > >> >> *Case #7* : *Solution A*
> >> > >> >>
> >> > >> >> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> >> > >> ><dp...@gmail.com>
> >> > >> >> wrote:
> >> > >> >>
> >> > >> >>> I have tried to add some clarification to Jarek's summary, but
> I
> >> am
> >> > >> >a
> >> > >> >>> little fuzzy on exact proposal for case 3 so hopefully Jarek
> can
> >> > >> >further
> >> > >> >>> clarify.
> >> > >> >>>
> >> > >> >>> According to my reading, in this motion cases 4,5,6 are all
> >> either
> >> > >> >>> proposal
> >> > >> >>> *rejections* or otherwise have no effect, so attention can be
> >> > >> >focused on
> >> > >> >>> cases 1,2,3 and 7.
> >> > >> >>>
> >> > >> >>> *Motion is to adopt the following:*
> >> > >> >>>
> >> > >> >>> Case 1 - Solution A - Get rid of contrib
> >> > >> >>> All objects will be moved out of contrib folder and into
> >> respective
> >> > >> >>> non-contrib folder.
> >> > >> >>>
> >> > >> >>> Case 2 - Solution A - Remove object type suffix from modules
> >> > >> >>> Example:
> >> > >> >>> Airflow.operators.foo_operator.FooOperator becomes
> >> > >> >>> airflow.operators.foo.FooOperator
> >> > >> >>> Example:
> >> > >> >>> Airflow.hooks.foo_hook.FooOperator becomes
> >> airflow.hooks.foo.FooHook
> >> > >> >>>
> >> > >> >>> Case 3 - conventions for cloud provider etc prefixes
> >> > >> >>> *I am not sure exactly what the proposal is.*
> >> > >> >>> Example case is
> >> airflow/contrib/operators/gcp_bigtable_operator.py
> >> > >> >>> Since motion adopts case 1 solution A, we can assume it is now
> >> named
> >> > >> >like
> >> > >> >>> so: airflow/operators/gcp_bigtable_operator.py
> >> > >> >>> So what is proposed for this case?
> >> > >> >>>
> >> > >> >>> Case 4 - Solution B - *no change*
> >> > >> >>> This proposal was about namespaces but the motion is to reject
> >> the
> >> > >> >>> proposal.
> >> > >> >>>
> >> > >> >>> Case 5 - Solution B - *no change*
> >> > >> >>> The proposal was to remove class type suffix e.g. remove the
> >> > >> >Operator in
> >> > >> >>> FooOperator, but the motion is to reject this proposal -- i.e.
> >> > >> >motion is
> >> > >> >>> no
> >> > >> >>> change on this case, i.e. we resolve to keep "Operator" (and
> >> > >> >"Sensor")
> >> > >> >>> suffixes on class names
> >> > >> >>>
> >> > >> >>> Case 6 - *No change*
> >> > >> >>> This case concerns object naming inconsistencies, but motion
> does
> >> > >> >not
> >> > >> >>> enact
> >> > >> >>> any specific proposal.
> >> > >> >>>
> >> > >> >>> Case 7 - Solution A - use warnings.warn template for
> deprecation
> >> > >> >warnings
> >> > >> >>> (instead of zope)
> >> > >> >>> Example:
> >> > >> >>>
> >> > >> >>> # in old_package/old_mod.py
> >> > >> >>>
> >> > >> >>> import warnings
> >> > >> >>> from new_package.new_mod import *
> >> > >> >>> warnings.warn("old_package.old_mod has moved to
> >> new_package.new_mod.
> >> > >> >>> Import
> >> > >> >>> of "
> >> > >> >>> "old_package.old_mod will become unsupported in version 2",
> >> > >> >>> DeprecationWarning, 2)
> >> > >> >>>
> >> > >> >>> Reference implementation here:
> >> > >> >>>
> >> > >> >>>
> >> > >> >
> >> > >>
> >> >
> >>
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> >> > >> >>>
> >> > >> >>>
> >> > >> >>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> >> > >> >>>
> >> > >> >>> I am very happy that this motion now gets rid of contrib.
> Thank
> >> you
> >> > >> >>> Sergio
> >> > >> >>> for turning the tide with your effective argumentation ;)
> >> > >> >>>
> >> > >> >>>
> >> > >> >>>
> >> > >> >>>
> >> > >> >>>
> >> > >> >>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
> >> ash@apache.org>
> >> > >> >wrote:
> >> > >> >>>
> >> > >> >>> > Could someone summarise what the proposed name changes are,
> >> here,
> >> > >> >in
> >> > >> >>> this
> >> > >> >>> > thread? Pointing at a google doc and only giving partial
> >> examples
> >> > >> >is 1)
> >> > >> >>> a
> >> > >> >>> > bit difficult to quickly work out what we are voting on, and
> 2)
> >> > >> >using a
> >> > >> >>> > google doc isn't great for a longer term record.
> >> > >> >>> >
> >> > >> >>> > Thanks,
> >> > >> >>> > Ash
> >> > >> >>> >
> >> > >> >>> > > On 26 Jul 2019, at 09:14, Jarek Potiuk
> >> > >> ><Ja...@polidea.com>
> >> > >> >>> wrote:
> >> > >> >>> > >
> >> > >> >>> > > I would like to recast the vote. Let's start from 0 :) . I
> >> would
> >> > >> >like
> >> > >> >>> to
> >> > >> >>> > > keep the July 30th 6pm CEST date, and at least three +1
> >> > >> >(binding)
> >> > >> >>> votes
> >> > >> >>> > are
> >> > >> >>> > > needed to pass. I marked in red the modifications to the
> >> > >> >previous
> >> > >> >>> > proposal.
> >> > >> >>> > >
> >> > >> >>> > > Here is the proposal (details here
> >> > >> >>> > > <
> >> > >> >>> >
> >> > >> >>>
> >> > >> >
> >> > >>
> >> >
> >>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> >> > >> >>> > >
> >> > >> >>> > > )
> >> > >> >>> > >
> >> > >> >>> > > - *Case 1: A: Get rid of contrib,*
> >> > >> >>> > > - *Case 2: A: Airflow.operators.foo_operator.FooOperator
> >> could
> >> > >> >>> > > become airflow.operators.foo.FooOperator*
> >> > >> >>> > > - *Case 3: C:
> >> > >> >>> > >
> >> > >> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >> > >> >>> could
> >> > >> >>> > > become airflow.gcp.operators.bigtable.BigTableOperator*
> >> > >> >>> > > - *Case 4: B: no namespace introduction*
> >> > >> >>> > > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
> >> class
> >> > >> >>> names*
> >> > >> >>> > > - *Case 6: We will treat isolated cases on case-by-case
> >> (and
> >> > >> >my team
> >> > >> >>> > can
> >> > >> >>> > > do the job of GCP-related operators)*
> >> > >> >>> > > - *Case 7: Native python solution (with better IDE
> >> > >> >integration)*
> >> > >> >>> > >
> >> > >> >>> > > This is my binding (+1) vote.
> >> > >> >>> > >
> >> > >> >>> > >
> >> > >> >>> > >
> >> > >> >>> > > On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> >> > >> >>> Jarek.Potiuk@polidea.com>
> >> > >> >>> > > wrote:
> >> > >> >>> > >
> >> > >> >>> > >> Hey Felix,
> >> > >> >>> > >>
> >> > >> >>> > >> For 7* -> I am in favour of trying the native one as well
> (I
> >> > >> >use IDE
> >> > >> >>> > quite
> >> > >> >>> > >> often ;) )
> >> > >> >>> > >>
> >> > >> >>> > >> J.
> >> > >> >>> > >>
> >> > >> >>> > >> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> >> > >> >>> <fokko@driesprong.frl
> >> > >> >>> > >
> >> > >> >>> > >> wrote:
> >> > >> >>> > >>
> >> > >> >>> > >>> Thanks Kamil for putting the document together. I wasn't
> >> able
> >> > >> >to
> >> > >> >>> > respond
> >> > >> >>> > >>> earlier since I was giving Airflow workshops throughout
> >> Europe
> >> > >> >:-)
> >> > >> >>> > >>>
> >> > >> >>> > >>> *Case 1: *Solution A → Remove everything from contrib
> into
> >> a
> >> > >> >single
> >> > >> >>> > >>> package.
> >> > >> >>> > >>>
> >> > >> >>> > >>> To make it easier to the end-user, my preference would be
> >> to
> >> > >> >get
> >> > >> >>> rid of
> >> > >> >>> > >>> the
> >> > >> >>> > >>> contrib package altogether. What's in contrib or not is a
> >> > >> >tricky
> >> > >> >>> > question,
> >> > >> >>> > >>> I think this is already reflected in the document since
> >> > >> >"maturity"
> >> > >> >>> is
> >> > >> >>> > in
> >> > >> >>> > >>> quotes. What would be the moment to transfer the operator
> >> to
> >> > >> >the
> >> > >> >>> core
> >> > >> >>> > or
> >> > >> >>> > >>> vice versa? I'm afraid that renaming contrib to
> incubating
> >> > >> >will
> >> > >> >>> confuse
> >> > >> >>> > >>> the
> >> > >> >>> > >>> user even more. For people who aren't as known with
> >> Airflow it
> >> > >> >isn't
> >> > >> >>> > that
> >> > >> >>> > >>> transparent which operator lives where, especially if you
> >> > >> >don't use
> >> > >> >>> an
> >> > >> >>> > IDE
> >> > >> >>> > >>> such as Ash ;) Besides that I don't think it is worth
> >> changing
> >> > >> >the
> >> > >> >>> > public
> >> > >> >>> > >>> API just for a different name.
> >> > >> >>> > >>>
> >> > >> >>> > >>
> >> > >> >>> > >> Ok. I am convinced. I will re-cast the vote with this. It
> >> will
> >> > >> >make
> >> > >> >>> > easier
> >> > >> >>> > >> as we will not have to define other rules as those that we
> >> > >> >already
> >> > >> >>> have.
> >> > >> >>> > >>
> >> > >> >>> > >>
> >> > >> >>> > >>> *Case 2:* Slight preference to Solution B (let's say a
> +0)
> >> > >> >>> > >>>
> >> > >> >>> > >>> I like to search on Github with t, and then search for
> >> > >> >BashOperator.
> >> > >> >>> > After
> >> > >> >>> > >>> the change, you should search for Operator/Bash.
> >> > >> >>> > >>>
> >> > >> >>> > >>> Jarek, I'm confused with your vote on this, from your
> mail
> >> you
> >> > >> >>> state: -
> >> > >> >>> > >>> *Case 2: B: Airflow.operators.foo_operator.FooOperator
> >> could
> >> > >> >become
> >> > >> >>> > >>> airflow.operators.foo.FooOperator*. But the B solution is
> >> > >> >keeping it
> >> > >> >>> > the
> >> > >> >>> > >>> same, so that would mean - *Case 2: B:
> >> > >> >>> > >>> Airflow.operators.foo_operator.FooOperator *would stay
> >> > >> >>> > >>> airflow.operators.foo_operator*.FooOperator*
> >> > >> >>> > >>>
> >> > >> >>> > >>> My mistake. It was of course A. I think there is a little
> >> > >> >confusion
> >> > >> >>> > here.
> >> > >> >>> > >> This case is not for changing BashOperator into Bash but
> for
> >> > >> >changing
> >> > >> >>> > the
> >> > >> >>> > >> *module* name not the *class* name. So
> >> > >> >bash_operator.BashOperator ->
> >> > >> >>> > >> bash.BashOperator :) . you will still search for
> >> BashOperator
> >> > >> >in this
> >> > >> >>> > case.
> >> > >> >>> > >>
> >> > >> >>> > >>
> >> > >> >>> > >>> *Case 3:* I'm leaning towards B
> >> > >> >>> > >>>
> >> > >> >>> > >>> Mostly because it isn't as trivial to decide when it gets
> >> its
> >> > >> >own
> >> > >> >>> > package
> >> > >> >>> > >>> or not. Besides the cloud providers, there are companies
> >> like
> >> > >> >>> > Databricks
> >> > >> >>> > >>> and Qubole which might also end up with their own package
> >> > >> >because
> >> > >> >>> their
> >> > >> >>> > >>> integration is getting richer over time. Moving this is a
> >> bit
> >> > >> >>> painful
> >> > >> >>> > >>> because we need to deprecate the old path and move
> >> everything
> >> > >> >which
> >> > >> >>> > >>> introduces noise into version control etc.
> >> > >> >>> > >>
> >> > >> >>> > >>
> >> > >> >>> > >> I'd say, we should go for C. As I proposed. It's not as
> >> > >> >difficult as
> >> > >> >>> it
> >> > >> >>> > >> seems to group operators in packages and it has numerous
> >> > >> >advantages.
> >> > >> >>> The
> >> > >> >>> > >> nice thing about deprecations - we can do them in the
> >> > >> >__init__.py of
> >> > >> >>> the
> >> > >> >>> > >> packages which causes the noise fairly small (Git will
> >> actually
> >> > >> >>> realise
> >> > >> >>> > >> that the file has been moved and you can follow that.
> Maybe
> >> not
> >> > >> >as
> >> > >> >>> super
> >> > >> >>> > >> easy to merge changes later to 1.10.4 but it's not a
> rocket
> >> > >> >science
> >> > >> >>> > either.
> >> > >> >>> > >>
> >> > >> >>> > >>
> >> > >> >>> > >>> *Case 7:* Python native solution
> >> > >> >>> > >>>
> >> > >> >>> > >>
> >> > >> >>> > >> Yes.
> >> > >> >>> > >>
> >> > >> >>> > >>
> >> > >> >>> > >>>
> >> > >> >>> > >>> ---
> >> > >> >>> > >>>
> >> > >> >>> > >>> Apart from all, great work!
> >> > >> >>> > >>>
> >> > >> >>> > >>> Cheers, Fokko
> >> > >> >>> > >>>
> >> > >> >>> > >>>
> >> > >> >>> > >>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> >> > >> >>> > >>> <feluelle@pm.me.invalid
> >> > >> >>> > >>>> :
> >> > >> >>> > >>>
> >> > >> >>> > >>>> I have no strong "No" against any proposed change of
> these
> >> > >> >cases.
> >> > >> >>> So I
> >> > >> >>> > >>> go
> >> > >> >>> > >>>> with +1 (non-binding).
> >> > >> >>> > >>>>
> >> > >> >>> > >>>> P.S. Thanks Jarek for bringing this up again and your
> >> intense
> >> > >> >work
> >> > >> >>> > >>> towards
> >> > >> >>> > >>>> airflow currently :) and thanks to Kamil for even
> creating
> >> > >> >this
> >> > >> >>> > >>> document. I
> >> > >> >>> > >>>> like how the code is getting more and more consistent
> and
> >> > >> >clean :)
> >> > >> >>> > >>>>
> >> > >> >>> > >>>> Kind regards,
> >> > >> >>> > >>>> Felix
> >> > >> >>> > >>>>
> >> > >> >>> > >>>> Sent from ProtonMail mobile
> >> > >> >>> > >>>>
> >> > >> >>> > >>>> -------- Original Message --------
> >> > >> >>> > >>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> >> > >> >>> > >>>>
> >> > >> >>> > >>>>> Hello everyone,
> >> > >> >>> > >>>>>
> >> > >> >>> > >>>>> This email is calling a vote on the changes in import
> >> paths.
> >> > >> >It's
> >> > >> >>> > been
> >> > >> >>> > >>>>> discussed in
> >> > >> >>> > >>>>>
> >> > >> >>> > >>>>
> >> > >> >>> > >>>
> >> > >> >>> >
> >> > >> >>>
> >> > >> >
> >> > >>
> >> >
> >>
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> >> > >> >>> > >>>>>
> >> > >> >>> > >>>>> The vote will last for at least 1 week (July 30th 6pm
> >> CEST),
> >> > >> >and
> >> > >> >>> at
> >> > >> >>> > >>> least
> >> > >> >>> > >>>>> three +1 (binding) votes have been cast.
> >> > >> >>> > >>>>>
> >> > >> >>> > >>>>> The proposal to vote is based on the document from
> Kamil
> >> > >> >>> > >>>>> <
> >> > >> >>> > >>>>
> >> > >> >>> > >>>
> >> > >> >>> >
> >> > >> >>>
> >> > >> >
> >> > >>
> >> >
> >>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> >> > >> >>> > >>>>>
> >> > >> >>> > >>>>>
> >> > >> >>> > >>>>> The proposed solution is:
> >> > >> >>> > >>>>>
> >> > >> >>> > >>>>> - *Case 1: B: Contrib vs not: we move all that are
> "well"
> >> > >> >tested
> >> > >> >>> and
> >> > >> >>> > >>>>> rename contrib to "incubating" or similar.*
> >> > >> >>> > >>>>> - *Case 2: B:
> Airflow.operators.foo_operator.FooOperator
> >> > >> >could
> >> > >> >>> > >>>>> become airflow.operators.foo.FooOperator*
> >> > >> >>> > >>>>> - *Case 3: C:
> >> > >> >>> > >>>>>
> >> > >> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >> > >> >>> > could
> >> > >> >>> > >>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> >> > >> >>> > >>>>> - *Case 4: B: no namespace introduction*
> >> > >> >>> > >>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes
> on
> >> > >> >class
> >> > >> >>> names*
> >> > >> >>> > >>>>> - *Case 6: We will treat isolated cases on case-by-case
> >> (and
> >> > >> >my
> >> > >> >>> team
> >> > >> >>> > >>> can
> >> > >> >>> > >>>>> do the job of GCP-related operators)*
> >> > >> >>> > >>>>>
> >> > >> >>> > >>>>> This is my (binding) +1 vote.
> >> > >> >>> > >>>>>
> >> > >> >>> > >>>>> Best regards,
> >> > >> >>> > >>>>>
> >> > >> >>> > >>>>> J.
> >> > >> >>> > >>>>>
> >> > >> >>> > >>>>> --
> >> > >> >>> > >>>>>
> >> > >> >>> > >>>>> Jarek Potiuk
> >> > >> >>> > >>>>> Polidea <https://www.polidea.com/> | Principal
> Software
> >> > >> >Engineer
> >> > >> >>> > >>>>>
> >> > >> >>> > >>>>> M: [+48 660 796 129](tel:+48660796129)
> >> > >> >>> > >>> <[+48660796129](tel:+48660796129)>
> >> > >> >>> > >>>>> [image: Polidea] <https://www.polidea.com/>
> >> > >> >>> > >>>
> >> > >> >>> > >>
> >> > >> >>> > >>
> >> > >> >>> > >> --
> >> > >> >>> > >>
> >> > >> >>> > >> Jarek Potiuk
> >> > >> >>> > >> Polidea <https://www.polidea.com/> | Principal Software
> >> > >> >Engineer
> >> > >> >>> > >>
> >> > >> >>> > >> M: +48 660 796 129 <+48660796129>
> >> > >> >>> > >> [image: Polidea] <https://www.polidea.com/>
> >> > >> >>> > >>
> >> > >> >>> > >>
> >> > >> >>> > >
> >> > >> >>> > > --
> >> > >> >>> > >
> >> > >> >>> > > Jarek Potiuk
> >> > >> >>> > > Polidea <https://www.polidea.com/> | Principal Software
> >> > Engineer
> >> > >> >>> > >
> >> > >> >>> > > M: +48 660 796 129 <+48660796129>
> >> > >> >>> > > [image: Polidea] <https://www.polidea.com/>
> >> > >> >>> >
> >> > >> >>> >
> >> > >> >>>
> >> > >> >>
> >> > >> >>
> >> > >> >> --
> >> > >> >> *Kaxil Naik*
> >> > >> >> *Big Data Consultant | DevOps Data Engineer*
> >> > >> >> *Certified *Google Cloud Data Engineer | *Certified* Apache
> Spark
> >> &
> >> > >> >Neo4j
> >> > >> >> Developer
> >> > >> >> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >> > >> >>
> >> > >> >
> >> > >> >
> >> > >> >--
> >> > >> >*Kaxil Naik*
> >> > >> >*Big Data Consultant | DevOps Data Engineer*
> >> > >> >*Certified *Google Cloud Data Engineer | *Certified* Apache Spark
> &
> >> > >> >Neo4j
> >> > >> >Developer
> >> > >> >*LinkedIn*: https://www.linkedin.com/in/kaxil
> >> > >>
> >> > >
> >> >
> >>
> >
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
Re: [VOTE] Changes in import paths
Posted by Jarek Potiuk <Ja...@polidea.com>.
I updated the AIP-21 proposal in the way that should answer all the
concerns in this thread.
NOTE! There is an action for all of those who are interested: Please
fill-in your voting in the voting table till Tuesday 30th of July 6pm CEST
(5pm BST, 9 am PST).
I created a summary of the proposal
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal>
in table form - which contains also examples for each case.
I added a voting table
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting>
where you should add your votes:
I propose this voting mechanism:
Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 pm BST, 9am
PST) .
After the choice, the final consistent set of choices will be announced
(taking into account majority of binding vote, also including potential
vetos and consistency between the choices. Non-binding votes will be taken
into account in case there is a draw. The final set of choices will be
announced at devlist thread
<https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E>
after
the voting completes.
Unless there is a veto raised to the final proposal, the final proposal is
accepted by Lazy Consensus
<https://community.apache.org/committers/lazyConsensus.html> on *Friday* 02
Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
Please let me know if what I proposed can be further improved to avoid
chaos.
J.
On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Ok. I see that indeed voting could have been organised a bit better - dev
> list does not seem to cope well with such a complex case :).
>
> As Kamil noticed - we already have a formal AIP-21
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths>
> in the Wiki - which I should have mentioned before. I have not much time
> today - but tomorrow I will try to summarise the options - based on some
> real code examples (to make it easier to digest). I think the best approach
> will be to make a voting matrix in the AIP-21 Wiki page where people will
> be able to state their preferences for each case (by adding a row to the
> table). I think that might provide much better clarity and a single place
> which will show the current aggregated state of voting.
>
> However - as Ash also stated - some of the options are inter-dependent so
> at the end we will have to adjust the results in case they are not
> consistent - and make final voting on aggregated set of cases - but I
> think in this case following "lasy consensus" - i,e. if noone objects to
> final set of cases, we will proceed with it..
>
> Do you think that will work better ?
>
> J.
>
> Principal Software Engineer
> Phone: +48660796129
>
> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> kamil.bregula@polidea.com> napisał:
>
>> Document is also available as AIP on wiki:
>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>>
>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko <fo...@driesprong.frl>
>> wrote:
>>
>> > I agree with all, this is a bit chaotic. For the sake of clarity, I
>> would
>> > suggest moving the Google Doc to the Wiki as an AIP. Create a structural
>> > code improvement AIP and then and add a matrix of users/cases where
>> > everybody can add his vote per case. This gives also the opportunity for
>> > changing your vote on a specific case. WDYT?
>> >
>> > Cheers, Fokko
>> >
>> > Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <ci...@gmail.com>:
>> >
>> > > I agree with Ash. It is clear to vote on each case rather than all
>> > > together.
>> > >
>> > > On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <as...@apache.org>
>> > wrote:
>> > >
>> > >> A number of proposals overlap or add on to each other, so I think
>> (?) a
>> > >> single vote makes sense.
>> > >>
>> > >> To be clear, my vote is -1 until it's clear exactly what we are
>> voting
>> > on.
>> > >>
>> > >> On 26 July 2019 20:39:56 BST, Kaxil Naik <ka...@gmail.com>
>> wrote:
>> > >> >I agree with Ash and instead of voting on all 7 Cases together, how
>> > >> >about
>> > >> >separate vote emails for all the cases? Each email can have the
>> > >> >description
>> > >> >copied over from the doc.
>> > >> >
>> > >> >It would probably be a bit easier to track a decision in the future
>> as
>> > >> >well.
>> > >> >
>> > >> >What do you guys think? @Jarek Potiuk <Ja...@polidea.com>
>> > >> >@Driesprong,
>> > >> >Fokko <fo...@driesprong.frl>
>> > >> >
>> > >> >On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <ka...@gmail.com>
>> > wrote:
>> > >> >
>> > >> >> *Case #1* airflow.contrib.{resources} : *Solution A *
>> > >> >>
>> > >> >> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
>> > >> >>
>> > >> >> *Case #3* {aws/azure/gcp}_* : *Solution C*
>> > >> >>
>> > >> >> *Case #4* Separate namespace for resources :
>> > >> >> *Solution A*
>> > >> >>
>> > >> >> *Case #5* *Operator : *Solution A*
>> > >> >>
>> > >> >> *Case #7* : *Solution A*
>> > >> >>
>> > >> >> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
>> > >> ><dp...@gmail.com>
>> > >> >> wrote:
>> > >> >>
>> > >> >>> I have tried to add some clarification to Jarek's summary, but I
>> am
>> > >> >a
>> > >> >>> little fuzzy on exact proposal for case 3 so hopefully Jarek can
>> > >> >further
>> > >> >>> clarify.
>> > >> >>>
>> > >> >>> According to my reading, in this motion cases 4,5,6 are all
>> either
>> > >> >>> proposal
>> > >> >>> *rejections* or otherwise have no effect, so attention can be
>> > >> >focused on
>> > >> >>> cases 1,2,3 and 7.
>> > >> >>>
>> > >> >>> *Motion is to adopt the following:*
>> > >> >>>
>> > >> >>> Case 1 - Solution A - Get rid of contrib
>> > >> >>> All objects will be moved out of contrib folder and into
>> respective
>> > >> >>> non-contrib folder.
>> > >> >>>
>> > >> >>> Case 2 - Solution A - Remove object type suffix from modules
>> > >> >>> Example:
>> > >> >>> Airflow.operators.foo_operator.FooOperator becomes
>> > >> >>> airflow.operators.foo.FooOperator
>> > >> >>> Example:
>> > >> >>> Airflow.hooks.foo_hook.FooOperator becomes
>> airflow.hooks.foo.FooHook
>> > >> >>>
>> > >> >>> Case 3 - conventions for cloud provider etc prefixes
>> > >> >>> *I am not sure exactly what the proposal is.*
>> > >> >>> Example case is
>> airflow/contrib/operators/gcp_bigtable_operator.py
>> > >> >>> Since motion adopts case 1 solution A, we can assume it is now
>> named
>> > >> >like
>> > >> >>> so: airflow/operators/gcp_bigtable_operator.py
>> > >> >>> So what is proposed for this case?
>> > >> >>>
>> > >> >>> Case 4 - Solution B - *no change*
>> > >> >>> This proposal was about namespaces but the motion is to reject
>> the
>> > >> >>> proposal.
>> > >> >>>
>> > >> >>> Case 5 - Solution B - *no change*
>> > >> >>> The proposal was to remove class type suffix e.g. remove the
>> > >> >Operator in
>> > >> >>> FooOperator, but the motion is to reject this proposal -- i.e.
>> > >> >motion is
>> > >> >>> no
>> > >> >>> change on this case, i.e. we resolve to keep "Operator" (and
>> > >> >"Sensor")
>> > >> >>> suffixes on class names
>> > >> >>>
>> > >> >>> Case 6 - *No change*
>> > >> >>> This case concerns object naming inconsistencies, but motion does
>> > >> >not
>> > >> >>> enact
>> > >> >>> any specific proposal.
>> > >> >>>
>> > >> >>> Case 7 - Solution A - use warnings.warn template for deprecation
>> > >> >warnings
>> > >> >>> (instead of zope)
>> > >> >>> Example:
>> > >> >>>
>> > >> >>> # in old_package/old_mod.py
>> > >> >>>
>> > >> >>> import warnings
>> > >> >>> from new_package.new_mod import *
>> > >> >>> warnings.warn("old_package.old_mod has moved to
>> new_package.new_mod.
>> > >> >>> Import
>> > >> >>> of "
>> > >> >>> "old_package.old_mod will become unsupported in version 2",
>> > >> >>> DeprecationWarning, 2)
>> > >> >>>
>> > >> >>> Reference implementation here:
>> > >> >>>
>> > >> >>>
>> > >> >
>> > >>
>> >
>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
>> > >> >>>
>> > >> >>>
>> > >> >>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
>> > >> >>>
>> > >> >>> I am very happy that this motion now gets rid of contrib. Thank
>> you
>> > >> >>> Sergio
>> > >> >>> for turning the tide with your effective argumentation ;)
>> > >> >>>
>> > >> >>>
>> > >> >>>
>> > >> >>>
>> > >> >>>
>> > >> >>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
>> ash@apache.org>
>> > >> >wrote:
>> > >> >>>
>> > >> >>> > Could someone summarise what the proposed name changes are,
>> here,
>> > >> >in
>> > >> >>> this
>> > >> >>> > thread? Pointing at a google doc and only giving partial
>> examples
>> > >> >is 1)
>> > >> >>> a
>> > >> >>> > bit difficult to quickly work out what we are voting on, and 2)
>> > >> >using a
>> > >> >>> > google doc isn't great for a longer term record.
>> > >> >>> >
>> > >> >>> > Thanks,
>> > >> >>> > Ash
>> > >> >>> >
>> > >> >>> > > On 26 Jul 2019, at 09:14, Jarek Potiuk
>> > >> ><Ja...@polidea.com>
>> > >> >>> wrote:
>> > >> >>> > >
>> > >> >>> > > I would like to recast the vote. Let's start from 0 :) . I
>> would
>> > >> >like
>> > >> >>> to
>> > >> >>> > > keep the July 30th 6pm CEST date, and at least three +1
>> > >> >(binding)
>> > >> >>> votes
>> > >> >>> > are
>> > >> >>> > > needed to pass. I marked in red the modifications to the
>> > >> >previous
>> > >> >>> > proposal.
>> > >> >>> > >
>> > >> >>> > > Here is the proposal (details here
>> > >> >>> > > <
>> > >> >>> >
>> > >> >>>
>> > >> >
>> > >>
>> >
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
>> > >> >>> > >
>> > >> >>> > > )
>> > >> >>> > >
>> > >> >>> > > - *Case 1: A: Get rid of contrib,*
>> > >> >>> > > - *Case 2: A: Airflow.operators.foo_operator.FooOperator
>> could
>> > >> >>> > > become airflow.operators.foo.FooOperator*
>> > >> >>> > > - *Case 3: C:
>> > >> >>> > >
>> > >> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>> > >> >>> could
>> > >> >>> > > become airflow.gcp.operators.bigtable.BigTableOperator*
>> > >> >>> > > - *Case 4: B: no namespace introduction*
>> > >> >>> > > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
>> class
>> > >> >>> names*
>> > >> >>> > > - *Case 6: We will treat isolated cases on case-by-case
>> (and
>> > >> >my team
>> > >> >>> > can
>> > >> >>> > > do the job of GCP-related operators)*
>> > >> >>> > > - *Case 7: Native python solution (with better IDE
>> > >> >integration)*
>> > >> >>> > >
>> > >> >>> > > This is my binding (+1) vote.
>> > >> >>> > >
>> > >> >>> > >
>> > >> >>> > >
>> > >> >>> > > On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
>> > >> >>> Jarek.Potiuk@polidea.com>
>> > >> >>> > > wrote:
>> > >> >>> > >
>> > >> >>> > >> Hey Felix,
>> > >> >>> > >>
>> > >> >>> > >> For 7* -> I am in favour of trying the native one as well (I
>> > >> >use IDE
>> > >> >>> > quite
>> > >> >>> > >> often ;) )
>> > >> >>> > >>
>> > >> >>> > >> J.
>> > >> >>> > >>
>> > >> >>> > >> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
>> > >> >>> <fokko@driesprong.frl
>> > >> >>> > >
>> > >> >>> > >> wrote:
>> > >> >>> > >>
>> > >> >>> > >>> Thanks Kamil for putting the document together. I wasn't
>> able
>> > >> >to
>> > >> >>> > respond
>> > >> >>> > >>> earlier since I was giving Airflow workshops throughout
>> Europe
>> > >> >:-)
>> > >> >>> > >>>
>> > >> >>> > >>> *Case 1: *Solution A → Remove everything from contrib into
>> a
>> > >> >single
>> > >> >>> > >>> package.
>> > >> >>> > >>>
>> > >> >>> > >>> To make it easier to the end-user, my preference would be
>> to
>> > >> >get
>> > >> >>> rid of
>> > >> >>> > >>> the
>> > >> >>> > >>> contrib package altogether. What's in contrib or not is a
>> > >> >tricky
>> > >> >>> > question,
>> > >> >>> > >>> I think this is already reflected in the document since
>> > >> >"maturity"
>> > >> >>> is
>> > >> >>> > in
>> > >> >>> > >>> quotes. What would be the moment to transfer the operator
>> to
>> > >> >the
>> > >> >>> core
>> > >> >>> > or
>> > >> >>> > >>> vice versa? I'm afraid that renaming contrib to incubating
>> > >> >will
>> > >> >>> confuse
>> > >> >>> > >>> the
>> > >> >>> > >>> user even more. For people who aren't as known with
>> Airflow it
>> > >> >isn't
>> > >> >>> > that
>> > >> >>> > >>> transparent which operator lives where, especially if you
>> > >> >don't use
>> > >> >>> an
>> > >> >>> > IDE
>> > >> >>> > >>> such as Ash ;) Besides that I don't think it is worth
>> changing
>> > >> >the
>> > >> >>> > public
>> > >> >>> > >>> API just for a different name.
>> > >> >>> > >>>
>> > >> >>> > >>
>> > >> >>> > >> Ok. I am convinced. I will re-cast the vote with this. It
>> will
>> > >> >make
>> > >> >>> > easier
>> > >> >>> > >> as we will not have to define other rules as those that we
>> > >> >already
>> > >> >>> have.
>> > >> >>> > >>
>> > >> >>> > >>
>> > >> >>> > >>> *Case 2:* Slight preference to Solution B (let's say a +0)
>> > >> >>> > >>>
>> > >> >>> > >>> I like to search on Github with t, and then search for
>> > >> >BashOperator.
>> > >> >>> > After
>> > >> >>> > >>> the change, you should search for Operator/Bash.
>> > >> >>> > >>>
>> > >> >>> > >>> Jarek, I'm confused with your vote on this, from your mail
>> you
>> > >> >>> state: -
>> > >> >>> > >>> *Case 2: B: Airflow.operators.foo_operator.FooOperator
>> could
>> > >> >become
>> > >> >>> > >>> airflow.operators.foo.FooOperator*. But the B solution is
>> > >> >keeping it
>> > >> >>> > the
>> > >> >>> > >>> same, so that would mean - *Case 2: B:
>> > >> >>> > >>> Airflow.operators.foo_operator.FooOperator *would stay
>> > >> >>> > >>> airflow.operators.foo_operator*.FooOperator*
>> > >> >>> > >>>
>> > >> >>> > >>> My mistake. It was of course A. I think there is a little
>> > >> >confusion
>> > >> >>> > here.
>> > >> >>> > >> This case is not for changing BashOperator into Bash but for
>> > >> >changing
>> > >> >>> > the
>> > >> >>> > >> *module* name not the *class* name. So
>> > >> >bash_operator.BashOperator ->
>> > >> >>> > >> bash.BashOperator :) . you will still search for
>> BashOperator
>> > >> >in this
>> > >> >>> > case.
>> > >> >>> > >>
>> > >> >>> > >>
>> > >> >>> > >>> *Case 3:* I'm leaning towards B
>> > >> >>> > >>>
>> > >> >>> > >>> Mostly because it isn't as trivial to decide when it gets
>> its
>> > >> >own
>> > >> >>> > package
>> > >> >>> > >>> or not. Besides the cloud providers, there are companies
>> like
>> > >> >>> > Databricks
>> > >> >>> > >>> and Qubole which might also end up with their own package
>> > >> >because
>> > >> >>> their
>> > >> >>> > >>> integration is getting richer over time. Moving this is a
>> bit
>> > >> >>> painful
>> > >> >>> > >>> because we need to deprecate the old path and move
>> everything
>> > >> >which
>> > >> >>> > >>> introduces noise into version control etc.
>> > >> >>> > >>
>> > >> >>> > >>
>> > >> >>> > >> I'd say, we should go for C. As I proposed. It's not as
>> > >> >difficult as
>> > >> >>> it
>> > >> >>> > >> seems to group operators in packages and it has numerous
>> > >> >advantages.
>> > >> >>> The
>> > >> >>> > >> nice thing about deprecations - we can do them in the
>> > >> >__init__.py of
>> > >> >>> the
>> > >> >>> > >> packages which causes the noise fairly small (Git will
>> actually
>> > >> >>> realise
>> > >> >>> > >> that the file has been moved and you can follow that. Maybe
>> not
>> > >> >as
>> > >> >>> super
>> > >> >>> > >> easy to merge changes later to 1.10.4 but it's not a rocket
>> > >> >science
>> > >> >>> > either.
>> > >> >>> > >>
>> > >> >>> > >>
>> > >> >>> > >>> *Case 7:* Python native solution
>> > >> >>> > >>>
>> > >> >>> > >>
>> > >> >>> > >> Yes.
>> > >> >>> > >>
>> > >> >>> > >>
>> > >> >>> > >>>
>> > >> >>> > >>> ---
>> > >> >>> > >>>
>> > >> >>> > >>> Apart from all, great work!
>> > >> >>> > >>>
>> > >> >>> > >>> Cheers, Fokko
>> > >> >>> > >>>
>> > >> >>> > >>>
>> > >> >>> > >>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
>> > >> >>> > >>> <feluelle@pm.me.invalid
>> > >> >>> > >>>> :
>> > >> >>> > >>>
>> > >> >>> > >>>> I have no strong "No" against any proposed change of these
>> > >> >cases.
>> > >> >>> So I
>> > >> >>> > >>> go
>> > >> >>> > >>>> with +1 (non-binding).
>> > >> >>> > >>>>
>> > >> >>> > >>>> P.S. Thanks Jarek for bringing this up again and your
>> intense
>> > >> >work
>> > >> >>> > >>> towards
>> > >> >>> > >>>> airflow currently :) and thanks to Kamil for even creating
>> > >> >this
>> > >> >>> > >>> document. I
>> > >> >>> > >>>> like how the code is getting more and more consistent and
>> > >> >clean :)
>> > >> >>> > >>>>
>> > >> >>> > >>>> Kind regards,
>> > >> >>> > >>>> Felix
>> > >> >>> > >>>>
>> > >> >>> > >>>> Sent from ProtonMail mobile
>> > >> >>> > >>>>
>> > >> >>> > >>>> -------- Original Message --------
>> > >> >>> > >>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
>> > >> >>> > >>>>
>> > >> >>> > >>>>> Hello everyone,
>> > >> >>> > >>>>>
>> > >> >>> > >>>>> This email is calling a vote on the changes in import
>> paths.
>> > >> >It's
>> > >> >>> > been
>> > >> >>> > >>>>> discussed in
>> > >> >>> > >>>>>
>> > >> >>> > >>>>
>> > >> >>> > >>>
>> > >> >>> >
>> > >> >>>
>> > >> >
>> > >>
>> >
>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
>> > >> >>> > >>>>>
>> > >> >>> > >>>>> The vote will last for at least 1 week (July 30th 6pm
>> CEST),
>> > >> >and
>> > >> >>> at
>> > >> >>> > >>> least
>> > >> >>> > >>>>> three +1 (binding) votes have been cast.
>> > >> >>> > >>>>>
>> > >> >>> > >>>>> The proposal to vote is based on the document from Kamil
>> > >> >>> > >>>>> <
>> > >> >>> > >>>>
>> > >> >>> > >>>
>> > >> >>> >
>> > >> >>>
>> > >> >
>> > >>
>> >
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
>> > >> >>> > >>>>>
>> > >> >>> > >>>>>
>> > >> >>> > >>>>> The proposed solution is:
>> > >> >>> > >>>>>
>> > >> >>> > >>>>> - *Case 1: B: Contrib vs not: we move all that are "well"
>> > >> >tested
>> > >> >>> and
>> > >> >>> > >>>>> rename contrib to "incubating" or similar.*
>> > >> >>> > >>>>> - *Case 2: B: Airflow.operators.foo_operator.FooOperator
>> > >> >could
>> > >> >>> > >>>>> become airflow.operators.foo.FooOperator*
>> > >> >>> > >>>>> - *Case 3: C:
>> > >> >>> > >>>>>
>> > >> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>> > >> >>> > could
>> > >> >>> > >>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
>> > >> >>> > >>>>> - *Case 4: B: no namespace introduction*
>> > >> >>> > >>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
>> > >> >class
>> > >> >>> names*
>> > >> >>> > >>>>> - *Case 6: We will treat isolated cases on case-by-case
>> (and
>> > >> >my
>> > >> >>> team
>> > >> >>> > >>> can
>> > >> >>> > >>>>> do the job of GCP-related operators)*
>> > >> >>> > >>>>>
>> > >> >>> > >>>>> This is my (binding) +1 vote.
>> > >> >>> > >>>>>
>> > >> >>> > >>>>> Best regards,
>> > >> >>> > >>>>>
>> > >> >>> > >>>>> J.
>> > >> >>> > >>>>>
>> > >> >>> > >>>>> --
>> > >> >>> > >>>>>
>> > >> >>> > >>>>> Jarek Potiuk
>> > >> >>> > >>>>> Polidea <https://www.polidea.com/> | Principal Software
>> > >> >Engineer
>> > >> >>> > >>>>>
>> > >> >>> > >>>>> M: [+48 660 796 129](tel:+48660796129)
>> > >> >>> > >>> <[+48660796129](tel:+48660796129)>
>> > >> >>> > >>>>> [image: Polidea] <https://www.polidea.com/>
>> > >> >>> > >>>
>> > >> >>> > >>
>> > >> >>> > >>
>> > >> >>> > >> --
>> > >> >>> > >>
>> > >> >>> > >> Jarek Potiuk
>> > >> >>> > >> Polidea <https://www.polidea.com/> | Principal Software
>> > >> >Engineer
>> > >> >>> > >>
>> > >> >>> > >> M: +48 660 796 129 <+48660796129>
>> > >> >>> > >> [image: Polidea] <https://www.polidea.com/>
>> > >> >>> > >>
>> > >> >>> > >>
>> > >> >>> > >
>> > >> >>> > > --
>> > >> >>> > >
>> > >> >>> > > Jarek Potiuk
>> > >> >>> > > Polidea <https://www.polidea.com/> | Principal Software
>> > Engineer
>> > >> >>> > >
>> > >> >>> > > M: +48 660 796 129 <+48660796129>
>> > >> >>> > > [image: Polidea] <https://www.polidea.com/>
>> > >> >>> >
>> > >> >>> >
>> > >> >>>
>> > >> >>
>> > >> >>
>> > >> >> --
>> > >> >> *Kaxil Naik*
>> > >> >> *Big Data Consultant | DevOps Data Engineer*
>> > >> >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark
>> &
>> > >> >Neo4j
>> > >> >> Developer
>> > >> >> *LinkedIn*: https://www.linkedin.com/in/kaxil
>> > >> >>
>> > >> >
>> > >> >
>> > >> >--
>> > >> >*Kaxil Naik*
>> > >> >*Big Data Consultant | DevOps Data Engineer*
>> > >> >*Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
>> > >> >Neo4j
>> > >> >Developer
>> > >> >*LinkedIn*: https://www.linkedin.com/in/kaxil
>> > >>
>> > >
>> >
>>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Changes in import paths
Posted by Jarek Potiuk <Ja...@polidea.com>.
Ok. I see that indeed voting could have been organised a bit better - dev
list does not seem to cope well with such a complex case :).
As Kamil noticed - we already have a formal AIP-21
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths>
in the Wiki - which I should have mentioned before. I have not much time
today - but tomorrow I will try to summarise the options - based on some
real code examples (to make it easier to digest). I think the best approach
will be to make a voting matrix in the AIP-21 Wiki page where people will
be able to state their preferences for each case (by adding a row to the
table). I think that might provide much better clarity and a single place
which will show the current aggregated state of voting.
However - as Ash also stated - some of the options are inter-dependent so
at the end we will have to adjust the results in case they are not
consistent - and make final voting on aggregated set of cases - but I
think in this case following "lasy consensus" - i,e. if noone objects to
final set of cases, we will proceed with it..
Do you think that will work better ?
J.
Principal Software Engineer
Phone: +48660796129
sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <ka...@polidea.com>
napisał:
> Document is also available as AIP on wiki:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>
> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
> > I agree with all, this is a bit chaotic. For the sake of clarity, I would
> > suggest moving the Google Doc to the Wiki as an AIP. Create a structural
> > code improvement AIP and then and add a matrix of users/cases where
> > everybody can add his vote per case. This gives also the opportunity for
> > changing your vote on a specific case. WDYT?
> >
> > Cheers, Fokko
> >
> > Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <ci...@gmail.com>:
> >
> > > I agree with Ash. It is clear to vote on each case rather than all
> > > together.
> > >
> > > On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > >
> > >> A number of proposals overlap or add on to each other, so I think (?)
> a
> > >> single vote makes sense.
> > >>
> > >> To be clear, my vote is -1 until it's clear exactly what we are voting
> > on.
> > >>
> > >> On 26 July 2019 20:39:56 BST, Kaxil Naik <ka...@gmail.com> wrote:
> > >> >I agree with Ash and instead of voting on all 7 Cases together, how
> > >> >about
> > >> >separate vote emails for all the cases? Each email can have the
> > >> >description
> > >> >copied over from the doc.
> > >> >
> > >> >It would probably be a bit easier to track a decision in the future
> as
> > >> >well.
> > >> >
> > >> >What do you guys think? @Jarek Potiuk <Ja...@polidea.com>
> > >> >@Driesprong,
> > >> >Fokko <fo...@driesprong.frl>
> > >> >
> > >> >On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <ka...@gmail.com>
> > wrote:
> > >> >
> > >> >> *Case #1* airflow.contrib.{resources} : *Solution A *
> > >> >>
> > >> >> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
> > >> >>
> > >> >> *Case #3* {aws/azure/gcp}_* : *Solution C*
> > >> >>
> > >> >> *Case #4* Separate namespace for resources :
> > >> >> *Solution A*
> > >> >>
> > >> >> *Case #5* *Operator : *Solution A*
> > >> >>
> > >> >> *Case #7* : *Solution A*
> > >> >>
> > >> >> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> > >> ><dp...@gmail.com>
> > >> >> wrote:
> > >> >>
> > >> >>> I have tried to add some clarification to Jarek's summary, but I
> am
> > >> >a
> > >> >>> little fuzzy on exact proposal for case 3 so hopefully Jarek can
> > >> >further
> > >> >>> clarify.
> > >> >>>
> > >> >>> According to my reading, in this motion cases 4,5,6 are all either
> > >> >>> proposal
> > >> >>> *rejections* or otherwise have no effect, so attention can be
> > >> >focused on
> > >> >>> cases 1,2,3 and 7.
> > >> >>>
> > >> >>> *Motion is to adopt the following:*
> > >> >>>
> > >> >>> Case 1 - Solution A - Get rid of contrib
> > >> >>> All objects will be moved out of contrib folder and into
> respective
> > >> >>> non-contrib folder.
> > >> >>>
> > >> >>> Case 2 - Solution A - Remove object type suffix from modules
> > >> >>> Example:
> > >> >>> Airflow.operators.foo_operator.FooOperator becomes
> > >> >>> airflow.operators.foo.FooOperator
> > >> >>> Example:
> > >> >>> Airflow.hooks.foo_hook.FooOperator becomes
> airflow.hooks.foo.FooHook
> > >> >>>
> > >> >>> Case 3 - conventions for cloud provider etc prefixes
> > >> >>> *I am not sure exactly what the proposal is.*
> > >> >>> Example case is airflow/contrib/operators/gcp_bigtable_operator.py
> > >> >>> Since motion adopts case 1 solution A, we can assume it is now
> named
> > >> >like
> > >> >>> so: airflow/operators/gcp_bigtable_operator.py
> > >> >>> So what is proposed for this case?
> > >> >>>
> > >> >>> Case 4 - Solution B - *no change*
> > >> >>> This proposal was about namespaces but the motion is to reject the
> > >> >>> proposal.
> > >> >>>
> > >> >>> Case 5 - Solution B - *no change*
> > >> >>> The proposal was to remove class type suffix e.g. remove the
> > >> >Operator in
> > >> >>> FooOperator, but the motion is to reject this proposal -- i.e.
> > >> >motion is
> > >> >>> no
> > >> >>> change on this case, i.e. we resolve to keep "Operator" (and
> > >> >"Sensor")
> > >> >>> suffixes on class names
> > >> >>>
> > >> >>> Case 6 - *No change*
> > >> >>> This case concerns object naming inconsistencies, but motion does
> > >> >not
> > >> >>> enact
> > >> >>> any specific proposal.
> > >> >>>
> > >> >>> Case 7 - Solution A - use warnings.warn template for deprecation
> > >> >warnings
> > >> >>> (instead of zope)
> > >> >>> Example:
> > >> >>>
> > >> >>> # in old_package/old_mod.py
> > >> >>>
> > >> >>> import warnings
> > >> >>> from new_package.new_mod import *
> > >> >>> warnings.warn("old_package.old_mod has moved to
> new_package.new_mod.
> > >> >>> Import
> > >> >>> of "
> > >> >>> "old_package.old_mod will become unsupported in version 2",
> > >> >>> DeprecationWarning, 2)
> > >> >>>
> > >> >>> Reference implementation here:
> > >> >>>
> > >> >>>
> > >> >
> > >>
> >
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> > >> >>>
> > >> >>>
> > >> >>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> > >> >>>
> > >> >>> I am very happy that this motion now gets rid of contrib. Thank
> you
> > >> >>> Sergio
> > >> >>> for turning the tide with your effective argumentation ;)
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <ash@apache.org
> >
> > >> >wrote:
> > >> >>>
> > >> >>> > Could someone summarise what the proposed name changes are,
> here,
> > >> >in
> > >> >>> this
> > >> >>> > thread? Pointing at a google doc and only giving partial
> examples
> > >> >is 1)
> > >> >>> a
> > >> >>> > bit difficult to quickly work out what we are voting on, and 2)
> > >> >using a
> > >> >>> > google doc isn't great for a longer term record.
> > >> >>> >
> > >> >>> > Thanks,
> > >> >>> > Ash
> > >> >>> >
> > >> >>> > > On 26 Jul 2019, at 09:14, Jarek Potiuk
> > >> ><Ja...@polidea.com>
> > >> >>> wrote:
> > >> >>> > >
> > >> >>> > > I would like to recast the vote. Let's start from 0 :) . I
> would
> > >> >like
> > >> >>> to
> > >> >>> > > keep the July 30th 6pm CEST date, and at least three +1
> > >> >(binding)
> > >> >>> votes
> > >> >>> > are
> > >> >>> > > needed to pass. I marked in red the modifications to the
> > >> >previous
> > >> >>> > proposal.
> > >> >>> > >
> > >> >>> > > Here is the proposal (details here
> > >> >>> > > <
> > >> >>> >
> > >> >>>
> > >> >
> > >>
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> > >> >>> > >
> > >> >>> > > )
> > >> >>> > >
> > >> >>> > > - *Case 1: A: Get rid of contrib,*
> > >> >>> > > - *Case 2: A: Airflow.operators.foo_operator.FooOperator
> could
> > >> >>> > > become airflow.operators.foo.FooOperator*
> > >> >>> > > - *Case 3: C:
> > >> >>> > >
> > >> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > >> >>> could
> > >> >>> > > become airflow.gcp.operators.bigtable.BigTableOperator*
> > >> >>> > > - *Case 4: B: no namespace introduction*
> > >> >>> > > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
> class
> > >> >>> names*
> > >> >>> > > - *Case 6: We will treat isolated cases on case-by-case (and
> > >> >my team
> > >> >>> > can
> > >> >>> > > do the job of GCP-related operators)*
> > >> >>> > > - *Case 7: Native python solution (with better IDE
> > >> >integration)*
> > >> >>> > >
> > >> >>> > > This is my binding (+1) vote.
> > >> >>> > >
> > >> >>> > >
> > >> >>> > >
> > >> >>> > > On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> > >> >>> Jarek.Potiuk@polidea.com>
> > >> >>> > > wrote:
> > >> >>> > >
> > >> >>> > >> Hey Felix,
> > >> >>> > >>
> > >> >>> > >> For 7* -> I am in favour of trying the native one as well (I
> > >> >use IDE
> > >> >>> > quite
> > >> >>> > >> often ;) )
> > >> >>> > >>
> > >> >>> > >> J.
> > >> >>> > >>
> > >> >>> > >> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> > >> >>> <fokko@driesprong.frl
> > >> >>> > >
> > >> >>> > >> wrote:
> > >> >>> > >>
> > >> >>> > >>> Thanks Kamil for putting the document together. I wasn't
> able
> > >> >to
> > >> >>> > respond
> > >> >>> > >>> earlier since I was giving Airflow workshops throughout
> Europe
> > >> >:-)
> > >> >>> > >>>
> > >> >>> > >>> *Case 1: *Solution A → Remove everything from contrib into a
> > >> >single
> > >> >>> > >>> package.
> > >> >>> > >>>
> > >> >>> > >>> To make it easier to the end-user, my preference would be to
> > >> >get
> > >> >>> rid of
> > >> >>> > >>> the
> > >> >>> > >>> contrib package altogether. What's in contrib or not is a
> > >> >tricky
> > >> >>> > question,
> > >> >>> > >>> I think this is already reflected in the document since
> > >> >"maturity"
> > >> >>> is
> > >> >>> > in
> > >> >>> > >>> quotes. What would be the moment to transfer the operator to
> > >> >the
> > >> >>> core
> > >> >>> > or
> > >> >>> > >>> vice versa? I'm afraid that renaming contrib to incubating
> > >> >will
> > >> >>> confuse
> > >> >>> > >>> the
> > >> >>> > >>> user even more. For people who aren't as known with Airflow
> it
> > >> >isn't
> > >> >>> > that
> > >> >>> > >>> transparent which operator lives where, especially if you
> > >> >don't use
> > >> >>> an
> > >> >>> > IDE
> > >> >>> > >>> such as Ash ;) Besides that I don't think it is worth
> changing
> > >> >the
> > >> >>> > public
> > >> >>> > >>> API just for a different name.
> > >> >>> > >>>
> > >> >>> > >>
> > >> >>> > >> Ok. I am convinced. I will re-cast the vote with this. It
> will
> > >> >make
> > >> >>> > easier
> > >> >>> > >> as we will not have to define other rules as those that we
> > >> >already
> > >> >>> have.
> > >> >>> > >>
> > >> >>> > >>
> > >> >>> > >>> *Case 2:* Slight preference to Solution B (let's say a +0)
> > >> >>> > >>>
> > >> >>> > >>> I like to search on Github with t, and then search for
> > >> >BashOperator.
> > >> >>> > After
> > >> >>> > >>> the change, you should search for Operator/Bash.
> > >> >>> > >>>
> > >> >>> > >>> Jarek, I'm confused with your vote on this, from your mail
> you
> > >> >>> state: -
> > >> >>> > >>> *Case 2: B: Airflow.operators.foo_operator.FooOperator could
> > >> >become
> > >> >>> > >>> airflow.operators.foo.FooOperator*. But the B solution is
> > >> >keeping it
> > >> >>> > the
> > >> >>> > >>> same, so that would mean - *Case 2: B:
> > >> >>> > >>> Airflow.operators.foo_operator.FooOperator *would stay
> > >> >>> > >>> airflow.operators.foo_operator*.FooOperator*
> > >> >>> > >>>
> > >> >>> > >>> My mistake. It was of course A. I think there is a little
> > >> >confusion
> > >> >>> > here.
> > >> >>> > >> This case is not for changing BashOperator into Bash but for
> > >> >changing
> > >> >>> > the
> > >> >>> > >> *module* name not the *class* name. So
> > >> >bash_operator.BashOperator ->
> > >> >>> > >> bash.BashOperator :) . you will still search for BashOperator
> > >> >in this
> > >> >>> > case.
> > >> >>> > >>
> > >> >>> > >>
> > >> >>> > >>> *Case 3:* I'm leaning towards B
> > >> >>> > >>>
> > >> >>> > >>> Mostly because it isn't as trivial to decide when it gets
> its
> > >> >own
> > >> >>> > package
> > >> >>> > >>> or not. Besides the cloud providers, there are companies
> like
> > >> >>> > Databricks
> > >> >>> > >>> and Qubole which might also end up with their own package
> > >> >because
> > >> >>> their
> > >> >>> > >>> integration is getting richer over time. Moving this is a
> bit
> > >> >>> painful
> > >> >>> > >>> because we need to deprecate the old path and move
> everything
> > >> >which
> > >> >>> > >>> introduces noise into version control etc.
> > >> >>> > >>
> > >> >>> > >>
> > >> >>> > >> I'd say, we should go for C. As I proposed. It's not as
> > >> >difficult as
> > >> >>> it
> > >> >>> > >> seems to group operators in packages and it has numerous
> > >> >advantages.
> > >> >>> The
> > >> >>> > >> nice thing about deprecations - we can do them in the
> > >> >__init__.py of
> > >> >>> the
> > >> >>> > >> packages which causes the noise fairly small (Git will
> actually
> > >> >>> realise
> > >> >>> > >> that the file has been moved and you can follow that. Maybe
> not
> > >> >as
> > >> >>> super
> > >> >>> > >> easy to merge changes later to 1.10.4 but it's not a rocket
> > >> >science
> > >> >>> > either.
> > >> >>> > >>
> > >> >>> > >>
> > >> >>> > >>> *Case 7:* Python native solution
> > >> >>> > >>>
> > >> >>> > >>
> > >> >>> > >> Yes.
> > >> >>> > >>
> > >> >>> > >>
> > >> >>> > >>>
> > >> >>> > >>> ---
> > >> >>> > >>>
> > >> >>> > >>> Apart from all, great work!
> > >> >>> > >>>
> > >> >>> > >>> Cheers, Fokko
> > >> >>> > >>>
> > >> >>> > >>>
> > >> >>> > >>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> > >> >>> > >>> <feluelle@pm.me.invalid
> > >> >>> > >>>> :
> > >> >>> > >>>
> > >> >>> > >>>> I have no strong "No" against any proposed change of these
> > >> >cases.
> > >> >>> So I
> > >> >>> > >>> go
> > >> >>> > >>>> with +1 (non-binding).
> > >> >>> > >>>>
> > >> >>> > >>>> P.S. Thanks Jarek for bringing this up again and your
> intense
> > >> >work
> > >> >>> > >>> towards
> > >> >>> > >>>> airflow currently :) and thanks to Kamil for even creating
> > >> >this
> > >> >>> > >>> document. I
> > >> >>> > >>>> like how the code is getting more and more consistent and
> > >> >clean :)
> > >> >>> > >>>>
> > >> >>> > >>>> Kind regards,
> > >> >>> > >>>> Felix
> > >> >>> > >>>>
> > >> >>> > >>>> Sent from ProtonMail mobile
> > >> >>> > >>>>
> > >> >>> > >>>> -------- Original Message --------
> > >> >>> > >>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> > >> >>> > >>>>
> > >> >>> > >>>>> Hello everyone,
> > >> >>> > >>>>>
> > >> >>> > >>>>> This email is calling a vote on the changes in import
> paths.
> > >> >It's
> > >> >>> > been
> > >> >>> > >>>>> discussed in
> > >> >>> > >>>>>
> > >> >>> > >>>>
> > >> >>> > >>>
> > >> >>> >
> > >> >>>
> > >> >
> > >>
> >
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> > >> >>> > >>>>>
> > >> >>> > >>>>> The vote will last for at least 1 week (July 30th 6pm
> CEST),
> > >> >and
> > >> >>> at
> > >> >>> > >>> least
> > >> >>> > >>>>> three +1 (binding) votes have been cast.
> > >> >>> > >>>>>
> > >> >>> > >>>>> The proposal to vote is based on the document from Kamil
> > >> >>> > >>>>> <
> > >> >>> > >>>>
> > >> >>> > >>>
> > >> >>> >
> > >> >>>
> > >> >
> > >>
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> > >> >>> > >>>>>
> > >> >>> > >>>>>
> > >> >>> > >>>>> The proposed solution is:
> > >> >>> > >>>>>
> > >> >>> > >>>>> - *Case 1: B: Contrib vs not: we move all that are "well"
> > >> >tested
> > >> >>> and
> > >> >>> > >>>>> rename contrib to "incubating" or similar.*
> > >> >>> > >>>>> - *Case 2: B: Airflow.operators.foo_operator.FooOperator
> > >> >could
> > >> >>> > >>>>> become airflow.operators.foo.FooOperator*
> > >> >>> > >>>>> - *Case 3: C:
> > >> >>> > >>>>>
> > >> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > >> >>> > could
> > >> >>> > >>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> > >> >>> > >>>>> - *Case 4: B: no namespace introduction*
> > >> >>> > >>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
> > >> >class
> > >> >>> names*
> > >> >>> > >>>>> - *Case 6: We will treat isolated cases on case-by-case
> (and
> > >> >my
> > >> >>> team
> > >> >>> > >>> can
> > >> >>> > >>>>> do the job of GCP-related operators)*
> > >> >>> > >>>>>
> > >> >>> > >>>>> This is my (binding) +1 vote.
> > >> >>> > >>>>>
> > >> >>> > >>>>> Best regards,
> > >> >>> > >>>>>
> > >> >>> > >>>>> J.
> > >> >>> > >>>>>
> > >> >>> > >>>>> --
> > >> >>> > >>>>>
> > >> >>> > >>>>> Jarek Potiuk
> > >> >>> > >>>>> Polidea <https://www.polidea.com/> | Principal Software
> > >> >Engineer
> > >> >>> > >>>>>
> > >> >>> > >>>>> M: [+48 660 796 129](tel:+48660796129)
> > >> >>> > >>> <[+48660796129](tel:+48660796129)>
> > >> >>> > >>>>> [image: Polidea] <https://www.polidea.com/>
> > >> >>> > >>>
> > >> >>> > >>
> > >> >>> > >>
> > >> >>> > >> --
> > >> >>> > >>
> > >> >>> > >> Jarek Potiuk
> > >> >>> > >> Polidea <https://www.polidea.com/> | Principal Software
> > >> >Engineer
> > >> >>> > >>
> > >> >>> > >> M: +48 660 796 129 <+48660796129>
> > >> >>> > >> [image: Polidea] <https://www.polidea.com/>
> > >> >>> > >>
> > >> >>> > >>
> > >> >>> > >
> > >> >>> > > --
> > >> >>> > >
> > >> >>> > > Jarek Potiuk
> > >> >>> > > Polidea <https://www.polidea.com/> | Principal Software
> > Engineer
> > >> >>> > >
> > >> >>> > > M: +48 660 796 129 <+48660796129>
> > >> >>> > > [image: Polidea] <https://www.polidea.com/>
> > >> >>> >
> > >> >>> >
> > >> >>>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> *Kaxil Naik*
> > >> >> *Big Data Consultant | DevOps Data Engineer*
> > >> >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> > >> >Neo4j
> > >> >> Developer
> > >> >> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >> >>
> > >> >
> > >> >
> > >> >--
> > >> >*Kaxil Naik*
> > >> >*Big Data Consultant | DevOps Data Engineer*
> > >> >*Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> > >> >Neo4j
> > >> >Developer
> > >> >*LinkedIn*: https://www.linkedin.com/in/kaxil
> > >>
> > >
> >
>
Re: [VOTE] Changes in import paths
Posted by Kamil Breguła <ka...@polidea.com>.
Document is also available as AIP on wiki:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> I agree with all, this is a bit chaotic. For the sake of clarity, I would
> suggest moving the Google Doc to the Wiki as an AIP. Create a structural
> code improvement AIP and then and add a matrix of users/cases where
> everybody can add his vote per case. This gives also the opportunity for
> changing your vote on a specific case. WDYT?
>
> Cheers, Fokko
>
> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <ci...@gmail.com>:
>
> > I agree with Ash. It is clear to vote on each case rather than all
> > together.
> >
> > On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> >> A number of proposals overlap or add on to each other, so I think (?) a
> >> single vote makes sense.
> >>
> >> To be clear, my vote is -1 until it's clear exactly what we are voting
> on.
> >>
> >> On 26 July 2019 20:39:56 BST, Kaxil Naik <ka...@gmail.com> wrote:
> >> >I agree with Ash and instead of voting on all 7 Cases together, how
> >> >about
> >> >separate vote emails for all the cases? Each email can have the
> >> >description
> >> >copied over from the doc.
> >> >
> >> >It would probably be a bit easier to track a decision in the future as
> >> >well.
> >> >
> >> >What do you guys think? @Jarek Potiuk <Ja...@polidea.com>
> >> >@Driesprong,
> >> >Fokko <fo...@driesprong.frl>
> >> >
> >> >On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <ka...@gmail.com>
> wrote:
> >> >
> >> >> *Case #1* airflow.contrib.{resources} : *Solution A *
> >> >>
> >> >> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
> >> >>
> >> >> *Case #3* {aws/azure/gcp}_* : *Solution C*
> >> >>
> >> >> *Case #4* Separate namespace for resources :
> >> >> *Solution A*
> >> >>
> >> >> *Case #5* *Operator : *Solution A*
> >> >>
> >> >> *Case #7* : *Solution A*
> >> >>
> >> >> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> >> ><dp...@gmail.com>
> >> >> wrote:
> >> >>
> >> >>> I have tried to add some clarification to Jarek's summary, but I am
> >> >a
> >> >>> little fuzzy on exact proposal for case 3 so hopefully Jarek can
> >> >further
> >> >>> clarify.
> >> >>>
> >> >>> According to my reading, in this motion cases 4,5,6 are all either
> >> >>> proposal
> >> >>> *rejections* or otherwise have no effect, so attention can be
> >> >focused on
> >> >>> cases 1,2,3 and 7.
> >> >>>
> >> >>> *Motion is to adopt the following:*
> >> >>>
> >> >>> Case 1 - Solution A - Get rid of contrib
> >> >>> All objects will be moved out of contrib folder and into respective
> >> >>> non-contrib folder.
> >> >>>
> >> >>> Case 2 - Solution A - Remove object type suffix from modules
> >> >>> Example:
> >> >>> Airflow.operators.foo_operator.FooOperator becomes
> >> >>> airflow.operators.foo.FooOperator
> >> >>> Example:
> >> >>> Airflow.hooks.foo_hook.FooOperator becomes airflow.hooks.foo.FooHook
> >> >>>
> >> >>> Case 3 - conventions for cloud provider etc prefixes
> >> >>> *I am not sure exactly what the proposal is.*
> >> >>> Example case is airflow/contrib/operators/gcp_bigtable_operator.py
> >> >>> Since motion adopts case 1 solution A, we can assume it is now named
> >> >like
> >> >>> so: airflow/operators/gcp_bigtable_operator.py
> >> >>> So what is proposed for this case?
> >> >>>
> >> >>> Case 4 - Solution B - *no change*
> >> >>> This proposal was about namespaces but the motion is to reject the
> >> >>> proposal.
> >> >>>
> >> >>> Case 5 - Solution B - *no change*
> >> >>> The proposal was to remove class type suffix e.g. remove the
> >> >Operator in
> >> >>> FooOperator, but the motion is to reject this proposal -- i.e.
> >> >motion is
> >> >>> no
> >> >>> change on this case, i.e. we resolve to keep "Operator" (and
> >> >"Sensor")
> >> >>> suffixes on class names
> >> >>>
> >> >>> Case 6 - *No change*
> >> >>> This case concerns object naming inconsistencies, but motion does
> >> >not
> >> >>> enact
> >> >>> any specific proposal.
> >> >>>
> >> >>> Case 7 - Solution A - use warnings.warn template for deprecation
> >> >warnings
> >> >>> (instead of zope)
> >> >>> Example:
> >> >>>
> >> >>> # in old_package/old_mod.py
> >> >>>
> >> >>> import warnings
> >> >>> from new_package.new_mod import *
> >> >>> warnings.warn("old_package.old_mod has moved to new_package.new_mod.
> >> >>> Import
> >> >>> of "
> >> >>> "old_package.old_mod will become unsupported in version 2",
> >> >>> DeprecationWarning, 2)
> >> >>>
> >> >>> Reference implementation here:
> >> >>>
> >> >>>
> >> >
> >>
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> >> >>>
> >> >>>
> >> >>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> >> >>>
> >> >>> I am very happy that this motion now gets rid of contrib. Thank you
> >> >>> Sergio
> >> >>> for turning the tide with your effective argumentation ;)
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <as...@apache.org>
> >> >wrote:
> >> >>>
> >> >>> > Could someone summarise what the proposed name changes are, here,
> >> >in
> >> >>> this
> >> >>> > thread? Pointing at a google doc and only giving partial examples
> >> >is 1)
> >> >>> a
> >> >>> > bit difficult to quickly work out what we are voting on, and 2)
> >> >using a
> >> >>> > google doc isn't great for a longer term record.
> >> >>> >
> >> >>> > Thanks,
> >> >>> > Ash
> >> >>> >
> >> >>> > > On 26 Jul 2019, at 09:14, Jarek Potiuk
> >> ><Ja...@polidea.com>
> >> >>> wrote:
> >> >>> > >
> >> >>> > > I would like to recast the vote. Let's start from 0 :) . I would
> >> >like
> >> >>> to
> >> >>> > > keep the July 30th 6pm CEST date, and at least three +1
> >> >(binding)
> >> >>> votes
> >> >>> > are
> >> >>> > > needed to pass. I marked in red the modifications to the
> >> >previous
> >> >>> > proposal.
> >> >>> > >
> >> >>> > > Here is the proposal (details here
> >> >>> > > <
> >> >>> >
> >> >>>
> >> >
> >>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> >> >>> > >
> >> >>> > > )
> >> >>> > >
> >> >>> > > - *Case 1: A: Get rid of contrib,*
> >> >>> > > - *Case 2: A: Airflow.operators.foo_operator.FooOperator could
> >> >>> > > become airflow.operators.foo.FooOperator*
> >> >>> > > - *Case 3: C:
> >> >>> > >
> >> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >> >>> could
> >> >>> > > become airflow.gcp.operators.bigtable.BigTableOperator*
> >> >>> > > - *Case 4: B: no namespace introduction*
> >> >>> > > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class
> >> >>> names*
> >> >>> > > - *Case 6: We will treat isolated cases on case-by-case (and
> >> >my team
> >> >>> > can
> >> >>> > > do the job of GCP-related operators)*
> >> >>> > > - *Case 7: Native python solution (with better IDE
> >> >integration)*
> >> >>> > >
> >> >>> > > This is my binding (+1) vote.
> >> >>> > >
> >> >>> > >
> >> >>> > >
> >> >>> > > On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> >> >>> Jarek.Potiuk@polidea.com>
> >> >>> > > wrote:
> >> >>> > >
> >> >>> > >> Hey Felix,
> >> >>> > >>
> >> >>> > >> For 7* -> I am in favour of trying the native one as well (I
> >> >use IDE
> >> >>> > quite
> >> >>> > >> often ;) )
> >> >>> > >>
> >> >>> > >> J.
> >> >>> > >>
> >> >>> > >> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> >> >>> <fokko@driesprong.frl
> >> >>> > >
> >> >>> > >> wrote:
> >> >>> > >>
> >> >>> > >>> Thanks Kamil for putting the document together. I wasn't able
> >> >to
> >> >>> > respond
> >> >>> > >>> earlier since I was giving Airflow workshops throughout Europe
> >> >:-)
> >> >>> > >>>
> >> >>> > >>> *Case 1: *Solution A → Remove everything from contrib into a
> >> >single
> >> >>> > >>> package.
> >> >>> > >>>
> >> >>> > >>> To make it easier to the end-user, my preference would be to
> >> >get
> >> >>> rid of
> >> >>> > >>> the
> >> >>> > >>> contrib package altogether. What's in contrib or not is a
> >> >tricky
> >> >>> > question,
> >> >>> > >>> I think this is already reflected in the document since
> >> >"maturity"
> >> >>> is
> >> >>> > in
> >> >>> > >>> quotes. What would be the moment to transfer the operator to
> >> >the
> >> >>> core
> >> >>> > or
> >> >>> > >>> vice versa? I'm afraid that renaming contrib to incubating
> >> >will
> >> >>> confuse
> >> >>> > >>> the
> >> >>> > >>> user even more. For people who aren't as known with Airflow it
> >> >isn't
> >> >>> > that
> >> >>> > >>> transparent which operator lives where, especially if you
> >> >don't use
> >> >>> an
> >> >>> > IDE
> >> >>> > >>> such as Ash ;) Besides that I don't think it is worth changing
> >> >the
> >> >>> > public
> >> >>> > >>> API just for a different name.
> >> >>> > >>>
> >> >>> > >>
> >> >>> > >> Ok. I am convinced. I will re-cast the vote with this. It will
> >> >make
> >> >>> > easier
> >> >>> > >> as we will not have to define other rules as those that we
> >> >already
> >> >>> have.
> >> >>> > >>
> >> >>> > >>
> >> >>> > >>> *Case 2:* Slight preference to Solution B (let's say a +0)
> >> >>> > >>>
> >> >>> > >>> I like to search on Github with t, and then search for
> >> >BashOperator.
> >> >>> > After
> >> >>> > >>> the change, you should search for Operator/Bash.
> >> >>> > >>>
> >> >>> > >>> Jarek, I'm confused with your vote on this, from your mail you
> >> >>> state: -
> >> >>> > >>> *Case 2: B: Airflow.operators.foo_operator.FooOperator could
> >> >become
> >> >>> > >>> airflow.operators.foo.FooOperator*. But the B solution is
> >> >keeping it
> >> >>> > the
> >> >>> > >>> same, so that would mean - *Case 2: B:
> >> >>> > >>> Airflow.operators.foo_operator.FooOperator *would stay
> >> >>> > >>> airflow.operators.foo_operator*.FooOperator*
> >> >>> > >>>
> >> >>> > >>> My mistake. It was of course A. I think there is a little
> >> >confusion
> >> >>> > here.
> >> >>> > >> This case is not for changing BashOperator into Bash but for
> >> >changing
> >> >>> > the
> >> >>> > >> *module* name not the *class* name. So
> >> >bash_operator.BashOperator ->
> >> >>> > >> bash.BashOperator :) . you will still search for BashOperator
> >> >in this
> >> >>> > case.
> >> >>> > >>
> >> >>> > >>
> >> >>> > >>> *Case 3:* I'm leaning towards B
> >> >>> > >>>
> >> >>> > >>> Mostly because it isn't as trivial to decide when it gets its
> >> >own
> >> >>> > package
> >> >>> > >>> or not. Besides the cloud providers, there are companies like
> >> >>> > Databricks
> >> >>> > >>> and Qubole which might also end up with their own package
> >> >because
> >> >>> their
> >> >>> > >>> integration is getting richer over time. Moving this is a bit
> >> >>> painful
> >> >>> > >>> because we need to deprecate the old path and move everything
> >> >which
> >> >>> > >>> introduces noise into version control etc.
> >> >>> > >>
> >> >>> > >>
> >> >>> > >> I'd say, we should go for C. As I proposed. It's not as
> >> >difficult as
> >> >>> it
> >> >>> > >> seems to group operators in packages and it has numerous
> >> >advantages.
> >> >>> The
> >> >>> > >> nice thing about deprecations - we can do them in the
> >> >__init__.py of
> >> >>> the
> >> >>> > >> packages which causes the noise fairly small (Git will actually
> >> >>> realise
> >> >>> > >> that the file has been moved and you can follow that. Maybe not
> >> >as
> >> >>> super
> >> >>> > >> easy to merge changes later to 1.10.4 but it's not a rocket
> >> >science
> >> >>> > either.
> >> >>> > >>
> >> >>> > >>
> >> >>> > >>> *Case 7:* Python native solution
> >> >>> > >>>
> >> >>> > >>
> >> >>> > >> Yes.
> >> >>> > >>
> >> >>> > >>
> >> >>> > >>>
> >> >>> > >>> ---
> >> >>> > >>>
> >> >>> > >>> Apart from all, great work!
> >> >>> > >>>
> >> >>> > >>> Cheers, Fokko
> >> >>> > >>>
> >> >>> > >>>
> >> >>> > >>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> >> >>> > >>> <feluelle@pm.me.invalid
> >> >>> > >>>> :
> >> >>> > >>>
> >> >>> > >>>> I have no strong "No" against any proposed change of these
> >> >cases.
> >> >>> So I
> >> >>> > >>> go
> >> >>> > >>>> with +1 (non-binding).
> >> >>> > >>>>
> >> >>> > >>>> P.S. Thanks Jarek for bringing this up again and your intense
> >> >work
> >> >>> > >>> towards
> >> >>> > >>>> airflow currently :) and thanks to Kamil for even creating
> >> >this
> >> >>> > >>> document. I
> >> >>> > >>>> like how the code is getting more and more consistent and
> >> >clean :)
> >> >>> > >>>>
> >> >>> > >>>> Kind regards,
> >> >>> > >>>> Felix
> >> >>> > >>>>
> >> >>> > >>>> Sent from ProtonMail mobile
> >> >>> > >>>>
> >> >>> > >>>> -------- Original Message --------
> >> >>> > >>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> >> >>> > >>>>
> >> >>> > >>>>> Hello everyone,
> >> >>> > >>>>>
> >> >>> > >>>>> This email is calling a vote on the changes in import paths.
> >> >It's
> >> >>> > been
> >> >>> > >>>>> discussed in
> >> >>> > >>>>>
> >> >>> > >>>>
> >> >>> > >>>
> >> >>> >
> >> >>>
> >> >
> >>
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> >> >>> > >>>>>
> >> >>> > >>>>> The vote will last for at least 1 week (July 30th 6pm CEST),
> >> >and
> >> >>> at
> >> >>> > >>> least
> >> >>> > >>>>> three +1 (binding) votes have been cast.
> >> >>> > >>>>>
> >> >>> > >>>>> The proposal to vote is based on the document from Kamil
> >> >>> > >>>>> <
> >> >>> > >>>>
> >> >>> > >>>
> >> >>> >
> >> >>>
> >> >
> >>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> >> >>> > >>>>>
> >> >>> > >>>>>
> >> >>> > >>>>> The proposed solution is:
> >> >>> > >>>>>
> >> >>> > >>>>> - *Case 1: B: Contrib vs not: we move all that are "well"
> >> >tested
> >> >>> and
> >> >>> > >>>>> rename contrib to "incubating" or similar.*
> >> >>> > >>>>> - *Case 2: B: Airflow.operators.foo_operator.FooOperator
> >> >could
> >> >>> > >>>>> become airflow.operators.foo.FooOperator*
> >> >>> > >>>>> - *Case 3: C:
> >> >>> > >>>>>
> >> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >> >>> > could
> >> >>> > >>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> >> >>> > >>>>> - *Case 4: B: no namespace introduction*
> >> >>> > >>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
> >> >class
> >> >>> names*
> >> >>> > >>>>> - *Case 6: We will treat isolated cases on case-by-case (and
> >> >my
> >> >>> team
> >> >>> > >>> can
> >> >>> > >>>>> do the job of GCP-related operators)*
> >> >>> > >>>>>
> >> >>> > >>>>> This is my (binding) +1 vote.
> >> >>> > >>>>>
> >> >>> > >>>>> Best regards,
> >> >>> > >>>>>
> >> >>> > >>>>> J.
> >> >>> > >>>>>
> >> >>> > >>>>> --
> >> >>> > >>>>>
> >> >>> > >>>>> Jarek Potiuk
> >> >>> > >>>>> Polidea <https://www.polidea.com/> | Principal Software
> >> >Engineer
> >> >>> > >>>>>
> >> >>> > >>>>> M: [+48 660 796 129](tel:+48660796129)
> >> >>> > >>> <[+48660796129](tel:+48660796129)>
> >> >>> > >>>>> [image: Polidea] <https://www.polidea.com/>
> >> >>> > >>>
> >> >>> > >>
> >> >>> > >>
> >> >>> > >> --
> >> >>> > >>
> >> >>> > >> Jarek Potiuk
> >> >>> > >> Polidea <https://www.polidea.com/> | Principal Software
> >> >Engineer
> >> >>> > >>
> >> >>> > >> M: +48 660 796 129 <+48660796129>
> >> >>> > >> [image: Polidea] <https://www.polidea.com/>
> >> >>> > >>
> >> >>> > >>
> >> >>> > >
> >> >>> > > --
> >> >>> > >
> >> >>> > > Jarek Potiuk
> >> >>> > > Polidea <https://www.polidea.com/> | Principal Software
> Engineer
> >> >>> > >
> >> >>> > > M: +48 660 796 129 <+48660796129>
> >> >>> > > [image: Polidea] <https://www.polidea.com/>
> >> >>> >
> >> >>> >
> >> >>>
> >> >>
> >> >>
> >> >> --
> >> >> *Kaxil Naik*
> >> >> *Big Data Consultant | DevOps Data Engineer*
> >> >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> >> >Neo4j
> >> >> Developer
> >> >> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >> >>
> >> >
> >> >
> >> >--
> >> >*Kaxil Naik*
> >> >*Big Data Consultant | DevOps Data Engineer*
> >> >*Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> >> >Neo4j
> >> >Developer
> >> >*LinkedIn*: https://www.linkedin.com/in/kaxil
> >>
> >
>
Re: [VOTE] Changes in import paths
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
I agree with all, this is a bit chaotic. For the sake of clarity, I would
suggest moving the Google Doc to the Wiki as an AIP. Create a structural
code improvement AIP and then and add a matrix of users/cases where
everybody can add his vote per case. This gives also the opportunity for
changing your vote on a specific case. WDYT?
Cheers, Fokko
Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <ci...@gmail.com>:
> I agree with Ash. It is clear to vote on each case rather than all
> together.
>
> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> A number of proposals overlap or add on to each other, so I think (?) a
>> single vote makes sense.
>>
>> To be clear, my vote is -1 until it's clear exactly what we are voting on.
>>
>> On 26 July 2019 20:39:56 BST, Kaxil Naik <ka...@gmail.com> wrote:
>> >I agree with Ash and instead of voting on all 7 Cases together, how
>> >about
>> >separate vote emails for all the cases? Each email can have the
>> >description
>> >copied over from the doc.
>> >
>> >It would probably be a bit easier to track a decision in the future as
>> >well.
>> >
>> >What do you guys think? @Jarek Potiuk <Ja...@polidea.com>
>> >@Driesprong,
>> >Fokko <fo...@driesprong.frl>
>> >
>> >On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <ka...@gmail.com> wrote:
>> >
>> >> *Case #1* airflow.contrib.{resources} : *Solution A *
>> >>
>> >> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
>> >>
>> >> *Case #3* {aws/azure/gcp}_* : *Solution C*
>> >>
>> >> *Case #4* Separate namespace for resources :
>> >> *Solution A*
>> >>
>> >> *Case #5* *Operator : *Solution A*
>> >>
>> >> *Case #7* : *Solution A*
>> >>
>> >> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
>> ><dp...@gmail.com>
>> >> wrote:
>> >>
>> >>> I have tried to add some clarification to Jarek's summary, but I am
>> >a
>> >>> little fuzzy on exact proposal for case 3 so hopefully Jarek can
>> >further
>> >>> clarify.
>> >>>
>> >>> According to my reading, in this motion cases 4,5,6 are all either
>> >>> proposal
>> >>> *rejections* or otherwise have no effect, so attention can be
>> >focused on
>> >>> cases 1,2,3 and 7.
>> >>>
>> >>> *Motion is to adopt the following:*
>> >>>
>> >>> Case 1 - Solution A - Get rid of contrib
>> >>> All objects will be moved out of contrib folder and into respective
>> >>> non-contrib folder.
>> >>>
>> >>> Case 2 - Solution A - Remove object type suffix from modules
>> >>> Example:
>> >>> Airflow.operators.foo_operator.FooOperator becomes
>> >>> airflow.operators.foo.FooOperator
>> >>> Example:
>> >>> Airflow.hooks.foo_hook.FooOperator becomes airflow.hooks.foo.FooHook
>> >>>
>> >>> Case 3 - conventions for cloud provider etc prefixes
>> >>> *I am not sure exactly what the proposal is.*
>> >>> Example case is airflow/contrib/operators/gcp_bigtable_operator.py
>> >>> Since motion adopts case 1 solution A, we can assume it is now named
>> >like
>> >>> so: airflow/operators/gcp_bigtable_operator.py
>> >>> So what is proposed for this case?
>> >>>
>> >>> Case 4 - Solution B - *no change*
>> >>> This proposal was about namespaces but the motion is to reject the
>> >>> proposal.
>> >>>
>> >>> Case 5 - Solution B - *no change*
>> >>> The proposal was to remove class type suffix e.g. remove the
>> >Operator in
>> >>> FooOperator, but the motion is to reject this proposal -- i.e.
>> >motion is
>> >>> no
>> >>> change on this case, i.e. we resolve to keep "Operator" (and
>> >"Sensor")
>> >>> suffixes on class names
>> >>>
>> >>> Case 6 - *No change*
>> >>> This case concerns object naming inconsistencies, but motion does
>> >not
>> >>> enact
>> >>> any specific proposal.
>> >>>
>> >>> Case 7 - Solution A - use warnings.warn template for deprecation
>> >warnings
>> >>> (instead of zope)
>> >>> Example:
>> >>>
>> >>> # in old_package/old_mod.py
>> >>>
>> >>> import warnings
>> >>> from new_package.new_mod import *
>> >>> warnings.warn("old_package.old_mod has moved to new_package.new_mod.
>> >>> Import
>> >>> of "
>> >>> "old_package.old_mod will become unsupported in version 2",
>> >>> DeprecationWarning, 2)
>> >>>
>> >>> Reference implementation here:
>> >>>
>> >>>
>> >
>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
>> >>>
>> >>>
>> >>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
>> >>>
>> >>> I am very happy that this motion now gets rid of contrib. Thank you
>> >>> Sergio
>> >>> for turning the tide with your effective argumentation ;)
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <as...@apache.org>
>> >wrote:
>> >>>
>> >>> > Could someone summarise what the proposed name changes are, here,
>> >in
>> >>> this
>> >>> > thread? Pointing at a google doc and only giving partial examples
>> >is 1)
>> >>> a
>> >>> > bit difficult to quickly work out what we are voting on, and 2)
>> >using a
>> >>> > google doc isn't great for a longer term record.
>> >>> >
>> >>> > Thanks,
>> >>> > Ash
>> >>> >
>> >>> > > On 26 Jul 2019, at 09:14, Jarek Potiuk
>> ><Ja...@polidea.com>
>> >>> wrote:
>> >>> > >
>> >>> > > I would like to recast the vote. Let's start from 0 :) . I would
>> >like
>> >>> to
>> >>> > > keep the July 30th 6pm CEST date, and at least three +1
>> >(binding)
>> >>> votes
>> >>> > are
>> >>> > > needed to pass. I marked in red the modifications to the
>> >previous
>> >>> > proposal.
>> >>> > >
>> >>> > > Here is the proposal (details here
>> >>> > > <
>> >>> >
>> >>>
>> >
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
>> >>> > >
>> >>> > > )
>> >>> > >
>> >>> > > - *Case 1: A: Get rid of contrib,*
>> >>> > > - *Case 2: A: Airflow.operators.foo_operator.FooOperator could
>> >>> > > become airflow.operators.foo.FooOperator*
>> >>> > > - *Case 3: C:
>> >>> > >
>> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>> >>> could
>> >>> > > become airflow.gcp.operators.bigtable.BigTableOperator*
>> >>> > > - *Case 4: B: no namespace introduction*
>> >>> > > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class
>> >>> names*
>> >>> > > - *Case 6: We will treat isolated cases on case-by-case (and
>> >my team
>> >>> > can
>> >>> > > do the job of GCP-related operators)*
>> >>> > > - *Case 7: Native python solution (with better IDE
>> >integration)*
>> >>> > >
>> >>> > > This is my binding (+1) vote.
>> >>> > >
>> >>> > >
>> >>> > >
>> >>> > > On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
>> >>> Jarek.Potiuk@polidea.com>
>> >>> > > wrote:
>> >>> > >
>> >>> > >> Hey Felix,
>> >>> > >>
>> >>> > >> For 7* -> I am in favour of trying the native one as well (I
>> >use IDE
>> >>> > quite
>> >>> > >> often ;) )
>> >>> > >>
>> >>> > >> J.
>> >>> > >>
>> >>> > >> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
>> >>> <fokko@driesprong.frl
>> >>> > >
>> >>> > >> wrote:
>> >>> > >>
>> >>> > >>> Thanks Kamil for putting the document together. I wasn't able
>> >to
>> >>> > respond
>> >>> > >>> earlier since I was giving Airflow workshops throughout Europe
>> >:-)
>> >>> > >>>
>> >>> > >>> *Case 1: *Solution A → Remove everything from contrib into a
>> >single
>> >>> > >>> package.
>> >>> > >>>
>> >>> > >>> To make it easier to the end-user, my preference would be to
>> >get
>> >>> rid of
>> >>> > >>> the
>> >>> > >>> contrib package altogether. What's in contrib or not is a
>> >tricky
>> >>> > question,
>> >>> > >>> I think this is already reflected in the document since
>> >"maturity"
>> >>> is
>> >>> > in
>> >>> > >>> quotes. What would be the moment to transfer the operator to
>> >the
>> >>> core
>> >>> > or
>> >>> > >>> vice versa? I'm afraid that renaming contrib to incubating
>> >will
>> >>> confuse
>> >>> > >>> the
>> >>> > >>> user even more. For people who aren't as known with Airflow it
>> >isn't
>> >>> > that
>> >>> > >>> transparent which operator lives where, especially if you
>> >don't use
>> >>> an
>> >>> > IDE
>> >>> > >>> such as Ash ;) Besides that I don't think it is worth changing
>> >the
>> >>> > public
>> >>> > >>> API just for a different name.
>> >>> > >>>
>> >>> > >>
>> >>> > >> Ok. I am convinced. I will re-cast the vote with this. It will
>> >make
>> >>> > easier
>> >>> > >> as we will not have to define other rules as those that we
>> >already
>> >>> have.
>> >>> > >>
>> >>> > >>
>> >>> > >>> *Case 2:* Slight preference to Solution B (let's say a +0)
>> >>> > >>>
>> >>> > >>> I like to search on Github with t, and then search for
>> >BashOperator.
>> >>> > After
>> >>> > >>> the change, you should search for Operator/Bash.
>> >>> > >>>
>> >>> > >>> Jarek, I'm confused with your vote on this, from your mail you
>> >>> state: -
>> >>> > >>> *Case 2: B: Airflow.operators.foo_operator.FooOperator could
>> >become
>> >>> > >>> airflow.operators.foo.FooOperator*. But the B solution is
>> >keeping it
>> >>> > the
>> >>> > >>> same, so that would mean - *Case 2: B:
>> >>> > >>> Airflow.operators.foo_operator.FooOperator *would stay
>> >>> > >>> airflow.operators.foo_operator*.FooOperator*
>> >>> > >>>
>> >>> > >>> My mistake. It was of course A. I think there is a little
>> >confusion
>> >>> > here.
>> >>> > >> This case is not for changing BashOperator into Bash but for
>> >changing
>> >>> > the
>> >>> > >> *module* name not the *class* name. So
>> >bash_operator.BashOperator ->
>> >>> > >> bash.BashOperator :) . you will still search for BashOperator
>> >in this
>> >>> > case.
>> >>> > >>
>> >>> > >>
>> >>> > >>> *Case 3:* I'm leaning towards B
>> >>> > >>>
>> >>> > >>> Mostly because it isn't as trivial to decide when it gets its
>> >own
>> >>> > package
>> >>> > >>> or not. Besides the cloud providers, there are companies like
>> >>> > Databricks
>> >>> > >>> and Qubole which might also end up with their own package
>> >because
>> >>> their
>> >>> > >>> integration is getting richer over time. Moving this is a bit
>> >>> painful
>> >>> > >>> because we need to deprecate the old path and move everything
>> >which
>> >>> > >>> introduces noise into version control etc.
>> >>> > >>
>> >>> > >>
>> >>> > >> I'd say, we should go for C. As I proposed. It's not as
>> >difficult as
>> >>> it
>> >>> > >> seems to group operators in packages and it has numerous
>> >advantages.
>> >>> The
>> >>> > >> nice thing about deprecations - we can do them in the
>> >__init__.py of
>> >>> the
>> >>> > >> packages which causes the noise fairly small (Git will actually
>> >>> realise
>> >>> > >> that the file has been moved and you can follow that. Maybe not
>> >as
>> >>> super
>> >>> > >> easy to merge changes later to 1.10.4 but it's not a rocket
>> >science
>> >>> > either.
>> >>> > >>
>> >>> > >>
>> >>> > >>> *Case 7:* Python native solution
>> >>> > >>>
>> >>> > >>
>> >>> > >> Yes.
>> >>> > >>
>> >>> > >>
>> >>> > >>>
>> >>> > >>> ---
>> >>> > >>>
>> >>> > >>> Apart from all, great work!
>> >>> > >>>
>> >>> > >>> Cheers, Fokko
>> >>> > >>>
>> >>> > >>>
>> >>> > >>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
>> >>> > >>> <feluelle@pm.me.invalid
>> >>> > >>>> :
>> >>> > >>>
>> >>> > >>>> I have no strong "No" against any proposed change of these
>> >cases.
>> >>> So I
>> >>> > >>> go
>> >>> > >>>> with +1 (non-binding).
>> >>> > >>>>
>> >>> > >>>> P.S. Thanks Jarek for bringing this up again and your intense
>> >work
>> >>> > >>> towards
>> >>> > >>>> airflow currently :) and thanks to Kamil for even creating
>> >this
>> >>> > >>> document. I
>> >>> > >>>> like how the code is getting more and more consistent and
>> >clean :)
>> >>> > >>>>
>> >>> > >>>> Kind regards,
>> >>> > >>>> Felix
>> >>> > >>>>
>> >>> > >>>> Sent from ProtonMail mobile
>> >>> > >>>>
>> >>> > >>>> -------- Original Message --------
>> >>> > >>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
>> >>> > >>>>
>> >>> > >>>>> Hello everyone,
>> >>> > >>>>>
>> >>> > >>>>> This email is calling a vote on the changes in import paths.
>> >It's
>> >>> > been
>> >>> > >>>>> discussed in
>> >>> > >>>>>
>> >>> > >>>>
>> >>> > >>>
>> >>> >
>> >>>
>> >
>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
>> >>> > >>>>>
>> >>> > >>>>> The vote will last for at least 1 week (July 30th 6pm CEST),
>> >and
>> >>> at
>> >>> > >>> least
>> >>> > >>>>> three +1 (binding) votes have been cast.
>> >>> > >>>>>
>> >>> > >>>>> The proposal to vote is based on the document from Kamil
>> >>> > >>>>> <
>> >>> > >>>>
>> >>> > >>>
>> >>> >
>> >>>
>> >
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
>> >>> > >>>>>
>> >>> > >>>>>
>> >>> > >>>>> The proposed solution is:
>> >>> > >>>>>
>> >>> > >>>>> - *Case 1: B: Contrib vs not: we move all that are "well"
>> >tested
>> >>> and
>> >>> > >>>>> rename contrib to "incubating" or similar.*
>> >>> > >>>>> - *Case 2: B: Airflow.operators.foo_operator.FooOperator
>> >could
>> >>> > >>>>> become airflow.operators.foo.FooOperator*
>> >>> > >>>>> - *Case 3: C:
>> >>> > >>>>>
>> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>> >>> > could
>> >>> > >>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
>> >>> > >>>>> - *Case 4: B: no namespace introduction*
>> >>> > >>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
>> >class
>> >>> names*
>> >>> > >>>>> - *Case 6: We will treat isolated cases on case-by-case (and
>> >my
>> >>> team
>> >>> > >>> can
>> >>> > >>>>> do the job of GCP-related operators)*
>> >>> > >>>>>
>> >>> > >>>>> This is my (binding) +1 vote.
>> >>> > >>>>>
>> >>> > >>>>> Best regards,
>> >>> > >>>>>
>> >>> > >>>>> J.
>> >>> > >>>>>
>> >>> > >>>>> --
>> >>> > >>>>>
>> >>> > >>>>> Jarek Potiuk
>> >>> > >>>>> Polidea <https://www.polidea.com/> | Principal Software
>> >Engineer
>> >>> > >>>>>
>> >>> > >>>>> M: [+48 660 796 129](tel:+48660796129)
>> >>> > >>> <[+48660796129](tel:+48660796129)>
>> >>> > >>>>> [image: Polidea] <https://www.polidea.com/>
>> >>> > >>>
>> >>> > >>
>> >>> > >>
>> >>> > >> --
>> >>> > >>
>> >>> > >> Jarek Potiuk
>> >>> > >> Polidea <https://www.polidea.com/> | Principal Software
>> >Engineer
>> >>> > >>
>> >>> > >> M: +48 660 796 129 <+48660796129>
>> >>> > >> [image: Polidea] <https://www.polidea.com/>
>> >>> > >>
>> >>> > >>
>> >>> > >
>> >>> > > --
>> >>> > >
>> >>> > > Jarek Potiuk
>> >>> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> >>> > >
>> >>> > > M: +48 660 796 129 <+48660796129>
>> >>> > > [image: Polidea] <https://www.polidea.com/>
>> >>> >
>> >>> >
>> >>>
>> >>
>> >>
>> >> --
>> >> *Kaxil Naik*
>> >> *Big Data Consultant | DevOps Data Engineer*
>> >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
>> >Neo4j
>> >> Developer
>> >> *LinkedIn*: https://www.linkedin.com/in/kaxil
>> >>
>> >
>> >
>> >--
>> >*Kaxil Naik*
>> >*Big Data Consultant | DevOps Data Engineer*
>> >*Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
>> >Neo4j
>> >Developer
>> >*LinkedIn*: https://www.linkedin.com/in/kaxil
>>
>
Re: [VOTE] Changes in import paths
Posted by Chen Tong <ci...@gmail.com>.
I agree with Ash. It is clear to vote on each case rather than all
together.
On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> A number of proposals overlap or add on to each other, so I think (?) a
> single vote makes sense.
>
> To be clear, my vote is -1 until it's clear exactly what we are voting on.
>
> On 26 July 2019 20:39:56 BST, Kaxil Naik <ka...@gmail.com> wrote:
> >I agree with Ash and instead of voting on all 7 Cases together, how
> >about
> >separate vote emails for all the cases? Each email can have the
> >description
> >copied over from the doc.
> >
> >It would probably be a bit easier to track a decision in the future as
> >well.
> >
> >What do you guys think? @Jarek Potiuk <Ja...@polidea.com>
> >@Driesprong,
> >Fokko <fo...@driesprong.frl>
> >
> >On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> >> *Case #1* airflow.contrib.{resources} : *Solution A *
> >>
> >> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
> >>
> >> *Case #3* {aws/azure/gcp}_* : *Solution C*
> >>
> >> *Case #4* Separate namespace for resources :
> >> *Solution A*
> >>
> >> *Case #5* *Operator : *Solution A*
> >>
> >> *Case #7* : *Solution A*
> >>
> >> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> ><dp...@gmail.com>
> >> wrote:
> >>
> >>> I have tried to add some clarification to Jarek's summary, but I am
> >a
> >>> little fuzzy on exact proposal for case 3 so hopefully Jarek can
> >further
> >>> clarify.
> >>>
> >>> According to my reading, in this motion cases 4,5,6 are all either
> >>> proposal
> >>> *rejections* or otherwise have no effect, so attention can be
> >focused on
> >>> cases 1,2,3 and 7.
> >>>
> >>> *Motion is to adopt the following:*
> >>>
> >>> Case 1 - Solution A - Get rid of contrib
> >>> All objects will be moved out of contrib folder and into respective
> >>> non-contrib folder.
> >>>
> >>> Case 2 - Solution A - Remove object type suffix from modules
> >>> Example:
> >>> Airflow.operators.foo_operator.FooOperator becomes
> >>> airflow.operators.foo.FooOperator
> >>> Example:
> >>> Airflow.hooks.foo_hook.FooOperator becomes airflow.hooks.foo.FooHook
> >>>
> >>> Case 3 - conventions for cloud provider etc prefixes
> >>> *I am not sure exactly what the proposal is.*
> >>> Example case is airflow/contrib/operators/gcp_bigtable_operator.py
> >>> Since motion adopts case 1 solution A, we can assume it is now named
> >like
> >>> so: airflow/operators/gcp_bigtable_operator.py
> >>> So what is proposed for this case?
> >>>
> >>> Case 4 - Solution B - *no change*
> >>> This proposal was about namespaces but the motion is to reject the
> >>> proposal.
> >>>
> >>> Case 5 - Solution B - *no change*
> >>> The proposal was to remove class type suffix e.g. remove the
> >Operator in
> >>> FooOperator, but the motion is to reject this proposal -- i.e.
> >motion is
> >>> no
> >>> change on this case, i.e. we resolve to keep "Operator" (and
> >"Sensor")
> >>> suffixes on class names
> >>>
> >>> Case 6 - *No change*
> >>> This case concerns object naming inconsistencies, but motion does
> >not
> >>> enact
> >>> any specific proposal.
> >>>
> >>> Case 7 - Solution A - use warnings.warn template for deprecation
> >warnings
> >>> (instead of zope)
> >>> Example:
> >>>
> >>> # in old_package/old_mod.py
> >>>
> >>> import warnings
> >>> from new_package.new_mod import *
> >>> warnings.warn("old_package.old_mod has moved to new_package.new_mod.
> >>> Import
> >>> of "
> >>> "old_package.old_mod will become unsupported in version 2",
> >>> DeprecationWarning, 2)
> >>>
> >>> Reference implementation here:
> >>>
> >>>
> >
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> >>>
> >>>
> >>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> >>>
> >>> I am very happy that this motion now gets rid of contrib. Thank you
> >>> Sergio
> >>> for turning the tide with your effective argumentation ;)
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <as...@apache.org>
> >wrote:
> >>>
> >>> > Could someone summarise what the proposed name changes are, here,
> >in
> >>> this
> >>> > thread? Pointing at a google doc and only giving partial examples
> >is 1)
> >>> a
> >>> > bit difficult to quickly work out what we are voting on, and 2)
> >using a
> >>> > google doc isn't great for a longer term record.
> >>> >
> >>> > Thanks,
> >>> > Ash
> >>> >
> >>> > > On 26 Jul 2019, at 09:14, Jarek Potiuk
> ><Ja...@polidea.com>
> >>> wrote:
> >>> > >
> >>> > > I would like to recast the vote. Let's start from 0 :) . I would
> >like
> >>> to
> >>> > > keep the July 30th 6pm CEST date, and at least three +1
> >(binding)
> >>> votes
> >>> > are
> >>> > > needed to pass. I marked in red the modifications to the
> >previous
> >>> > proposal.
> >>> > >
> >>> > > Here is the proposal (details here
> >>> > > <
> >>> >
> >>>
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> >>> > >
> >>> > > )
> >>> > >
> >>> > > - *Case 1: A: Get rid of contrib,*
> >>> > > - *Case 2: A: Airflow.operators.foo_operator.FooOperator could
> >>> > > become airflow.operators.foo.FooOperator*
> >>> > > - *Case 3: C:
> >>> > >
> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >>> could
> >>> > > become airflow.gcp.operators.bigtable.BigTableOperator*
> >>> > > - *Case 4: B: no namespace introduction*
> >>> > > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class
> >>> names*
> >>> > > - *Case 6: We will treat isolated cases on case-by-case (and
> >my team
> >>> > can
> >>> > > do the job of GCP-related operators)*
> >>> > > - *Case 7: Native python solution (with better IDE
> >integration)*
> >>> > >
> >>> > > This is my binding (+1) vote.
> >>> > >
> >>> > >
> >>> > >
> >>> > > On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> >>> Jarek.Potiuk@polidea.com>
> >>> > > wrote:
> >>> > >
> >>> > >> Hey Felix,
> >>> > >>
> >>> > >> For 7* -> I am in favour of trying the native one as well (I
> >use IDE
> >>> > quite
> >>> > >> often ;) )
> >>> > >>
> >>> > >> J.
> >>> > >>
> >>> > >> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> >>> <fokko@driesprong.frl
> >>> > >
> >>> > >> wrote:
> >>> > >>
> >>> > >>> Thanks Kamil for putting the document together. I wasn't able
> >to
> >>> > respond
> >>> > >>> earlier since I was giving Airflow workshops throughout Europe
> >:-)
> >>> > >>>
> >>> > >>> *Case 1: *Solution A → Remove everything from contrib into a
> >single
> >>> > >>> package.
> >>> > >>>
> >>> > >>> To make it easier to the end-user, my preference would be to
> >get
> >>> rid of
> >>> > >>> the
> >>> > >>> contrib package altogether. What's in contrib or not is a
> >tricky
> >>> > question,
> >>> > >>> I think this is already reflected in the document since
> >"maturity"
> >>> is
> >>> > in
> >>> > >>> quotes. What would be the moment to transfer the operator to
> >the
> >>> core
> >>> > or
> >>> > >>> vice versa? I'm afraid that renaming contrib to incubating
> >will
> >>> confuse
> >>> > >>> the
> >>> > >>> user even more. For people who aren't as known with Airflow it
> >isn't
> >>> > that
> >>> > >>> transparent which operator lives where, especially if you
> >don't use
> >>> an
> >>> > IDE
> >>> > >>> such as Ash ;) Besides that I don't think it is worth changing
> >the
> >>> > public
> >>> > >>> API just for a different name.
> >>> > >>>
> >>> > >>
> >>> > >> Ok. I am convinced. I will re-cast the vote with this. It will
> >make
> >>> > easier
> >>> > >> as we will not have to define other rules as those that we
> >already
> >>> have.
> >>> > >>
> >>> > >>
> >>> > >>> *Case 2:* Slight preference to Solution B (let's say a +0)
> >>> > >>>
> >>> > >>> I like to search on Github with t, and then search for
> >BashOperator.
> >>> > After
> >>> > >>> the change, you should search for Operator/Bash.
> >>> > >>>
> >>> > >>> Jarek, I'm confused with your vote on this, from your mail you
> >>> state: -
> >>> > >>> *Case 2: B: Airflow.operators.foo_operator.FooOperator could
> >become
> >>> > >>> airflow.operators.foo.FooOperator*. But the B solution is
> >keeping it
> >>> > the
> >>> > >>> same, so that would mean - *Case 2: B:
> >>> > >>> Airflow.operators.foo_operator.FooOperator *would stay
> >>> > >>> airflow.operators.foo_operator*.FooOperator*
> >>> > >>>
> >>> > >>> My mistake. It was of course A. I think there is a little
> >confusion
> >>> > here.
> >>> > >> This case is not for changing BashOperator into Bash but for
> >changing
> >>> > the
> >>> > >> *module* name not the *class* name. So
> >bash_operator.BashOperator ->
> >>> > >> bash.BashOperator :) . you will still search for BashOperator
> >in this
> >>> > case.
> >>> > >>
> >>> > >>
> >>> > >>> *Case 3:* I'm leaning towards B
> >>> > >>>
> >>> > >>> Mostly because it isn't as trivial to decide when it gets its
> >own
> >>> > package
> >>> > >>> or not. Besides the cloud providers, there are companies like
> >>> > Databricks
> >>> > >>> and Qubole which might also end up with their own package
> >because
> >>> their
> >>> > >>> integration is getting richer over time. Moving this is a bit
> >>> painful
> >>> > >>> because we need to deprecate the old path and move everything
> >which
> >>> > >>> introduces noise into version control etc.
> >>> > >>
> >>> > >>
> >>> > >> I'd say, we should go for C. As I proposed. It's not as
> >difficult as
> >>> it
> >>> > >> seems to group operators in packages and it has numerous
> >advantages.
> >>> The
> >>> > >> nice thing about deprecations - we can do them in the
> >__init__.py of
> >>> the
> >>> > >> packages which causes the noise fairly small (Git will actually
> >>> realise
> >>> > >> that the file has been moved and you can follow that. Maybe not
> >as
> >>> super
> >>> > >> easy to merge changes later to 1.10.4 but it's not a rocket
> >science
> >>> > either.
> >>> > >>
> >>> > >>
> >>> > >>> *Case 7:* Python native solution
> >>> > >>>
> >>> > >>
> >>> > >> Yes.
> >>> > >>
> >>> > >>
> >>> > >>>
> >>> > >>> ---
> >>> > >>>
> >>> > >>> Apart from all, great work!
> >>> > >>>
> >>> > >>> Cheers, Fokko
> >>> > >>>
> >>> > >>>
> >>> > >>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> >>> > >>> <feluelle@pm.me.invalid
> >>> > >>>> :
> >>> > >>>
> >>> > >>>> I have no strong "No" against any proposed change of these
> >cases.
> >>> So I
> >>> > >>> go
> >>> > >>>> with +1 (non-binding).
> >>> > >>>>
> >>> > >>>> P.S. Thanks Jarek for bringing this up again and your intense
> >work
> >>> > >>> towards
> >>> > >>>> airflow currently :) and thanks to Kamil for even creating
> >this
> >>> > >>> document. I
> >>> > >>>> like how the code is getting more and more consistent and
> >clean :)
> >>> > >>>>
> >>> > >>>> Kind regards,
> >>> > >>>> Felix
> >>> > >>>>
> >>> > >>>> Sent from ProtonMail mobile
> >>> > >>>>
> >>> > >>>> -------- Original Message --------
> >>> > >>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> >>> > >>>>
> >>> > >>>>> Hello everyone,
> >>> > >>>>>
> >>> > >>>>> This email is calling a vote on the changes in import paths.
> >It's
> >>> > been
> >>> > >>>>> discussed in
> >>> > >>>>>
> >>> > >>>>
> >>> > >>>
> >>> >
> >>>
> >
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> >>> > >>>>>
> >>> > >>>>> The vote will last for at least 1 week (July 30th 6pm CEST),
> >and
> >>> at
> >>> > >>> least
> >>> > >>>>> three +1 (binding) votes have been cast.
> >>> > >>>>>
> >>> > >>>>> The proposal to vote is based on the document from Kamil
> >>> > >>>>> <
> >>> > >>>>
> >>> > >>>
> >>> >
> >>>
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> >>> > >>>>>
> >>> > >>>>>
> >>> > >>>>> The proposed solution is:
> >>> > >>>>>
> >>> > >>>>> - *Case 1: B: Contrib vs not: we move all that are "well"
> >tested
> >>> and
> >>> > >>>>> rename contrib to "incubating" or similar.*
> >>> > >>>>> - *Case 2: B: Airflow.operators.foo_operator.FooOperator
> >could
> >>> > >>>>> become airflow.operators.foo.FooOperator*
> >>> > >>>>> - *Case 3: C:
> >>> > >>>>>
> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >>> > could
> >>> > >>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> >>> > >>>>> - *Case 4: B: no namespace introduction*
> >>> > >>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
> >class
> >>> names*
> >>> > >>>>> - *Case 6: We will treat isolated cases on case-by-case (and
> >my
> >>> team
> >>> > >>> can
> >>> > >>>>> do the job of GCP-related operators)*
> >>> > >>>>>
> >>> > >>>>> This is my (binding) +1 vote.
> >>> > >>>>>
> >>> > >>>>> Best regards,
> >>> > >>>>>
> >>> > >>>>> J.
> >>> > >>>>>
> >>> > >>>>> --
> >>> > >>>>>
> >>> > >>>>> Jarek Potiuk
> >>> > >>>>> Polidea <https://www.polidea.com/> | Principal Software
> >Engineer
> >>> > >>>>>
> >>> > >>>>> M: [+48 660 796 129](tel:+48660796129)
> >>> > >>> <[+48660796129](tel:+48660796129)>
> >>> > >>>>> [image: Polidea] <https://www.polidea.com/>
> >>> > >>>
> >>> > >>
> >>> > >>
> >>> > >> --
> >>> > >>
> >>> > >> Jarek Potiuk
> >>> > >> Polidea <https://www.polidea.com/> | Principal Software
> >Engineer
> >>> > >>
> >>> > >> M: +48 660 796 129 <+48660796129>
> >>> > >> [image: Polidea] <https://www.polidea.com/>
> >>> > >>
> >>> > >>
> >>> > >
> >>> > > --
> >>> > >
> >>> > > Jarek Potiuk
> >>> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>> > >
> >>> > > M: +48 660 796 129 <+48660796129>
> >>> > > [image: Polidea] <https://www.polidea.com/>
> >>> >
> >>> >
> >>>
> >>
> >>
> >> --
> >> *Kaxil Naik*
> >> *Big Data Consultant | DevOps Data Engineer*
> >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> >Neo4j
> >> Developer
> >> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>
> >
> >
> >--
> >*Kaxil Naik*
> >*Big Data Consultant | DevOps Data Engineer*
> >*Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> >Neo4j
> >Developer
> >*LinkedIn*: https://www.linkedin.com/in/kaxil
>
Re: [VOTE] Changes in import paths
Posted by Ash Berlin-Taylor <as...@apache.org>.
A number of proposals overlap or add on to each other, so I think (?) a single vote makes sense.
To be clear, my vote is -1 until it's clear exactly what we are voting on.
On 26 July 2019 20:39:56 BST, Kaxil Naik <ka...@gmail.com> wrote:
>I agree with Ash and instead of voting on all 7 Cases together, how
>about
>separate vote emails for all the cases? Each email can have the
>description
>copied over from the doc.
>
>It would probably be a bit easier to track a decision in the future as
>well.
>
>What do you guys think? @Jarek Potiuk <Ja...@polidea.com>
>@Driesprong,
>Fokko <fo...@driesprong.frl>
>
>On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <ka...@gmail.com> wrote:
>
>> *Case #1* airflow.contrib.{resources} : *Solution A *
>>
>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
>>
>> *Case #3* {aws/azure/gcp}_* : *Solution C*
>>
>> *Case #4* Separate namespace for resources :
>> *Solution A*
>>
>> *Case #5* *Operator : *Solution A*
>>
>> *Case #7* : *Solution A*
>>
>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
><dp...@gmail.com>
>> wrote:
>>
>>> I have tried to add some clarification to Jarek's summary, but I am
>a
>>> little fuzzy on exact proposal for case 3 so hopefully Jarek can
>further
>>> clarify.
>>>
>>> According to my reading, in this motion cases 4,5,6 are all either
>>> proposal
>>> *rejections* or otherwise have no effect, so attention can be
>focused on
>>> cases 1,2,3 and 7.
>>>
>>> *Motion is to adopt the following:*
>>>
>>> Case 1 - Solution A - Get rid of contrib
>>> All objects will be moved out of contrib folder and into respective
>>> non-contrib folder.
>>>
>>> Case 2 - Solution A - Remove object type suffix from modules
>>> Example:
>>> Airflow.operators.foo_operator.FooOperator becomes
>>> airflow.operators.foo.FooOperator
>>> Example:
>>> Airflow.hooks.foo_hook.FooOperator becomes airflow.hooks.foo.FooHook
>>>
>>> Case 3 - conventions for cloud provider etc prefixes
>>> *I am not sure exactly what the proposal is.*
>>> Example case is airflow/contrib/operators/gcp_bigtable_operator.py
>>> Since motion adopts case 1 solution A, we can assume it is now named
>like
>>> so: airflow/operators/gcp_bigtable_operator.py
>>> So what is proposed for this case?
>>>
>>> Case 4 - Solution B - *no change*
>>> This proposal was about namespaces but the motion is to reject the
>>> proposal.
>>>
>>> Case 5 - Solution B - *no change*
>>> The proposal was to remove class type suffix e.g. remove the
>Operator in
>>> FooOperator, but the motion is to reject this proposal -- i.e.
>motion is
>>> no
>>> change on this case, i.e. we resolve to keep "Operator" (and
>"Sensor")
>>> suffixes on class names
>>>
>>> Case 6 - *No change*
>>> This case concerns object naming inconsistencies, but motion does
>not
>>> enact
>>> any specific proposal.
>>>
>>> Case 7 - Solution A - use warnings.warn template for deprecation
>warnings
>>> (instead of zope)
>>> Example:
>>>
>>> # in old_package/old_mod.py
>>>
>>> import warnings
>>> from new_package.new_mod import *
>>> warnings.warn("old_package.old_mod has moved to new_package.new_mod.
>>> Import
>>> of "
>>> "old_package.old_mod will become unsupported in version 2",
>>> DeprecationWarning, 2)
>>>
>>> Reference implementation here:
>>>
>>>
>https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
>>>
>>>
>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
>>>
>>> I am very happy that this motion now gets rid of contrib. Thank you
>>> Sergio
>>> for turning the tide with your effective argumentation ;)
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <as...@apache.org>
>wrote:
>>>
>>> > Could someone summarise what the proposed name changes are, here,
>in
>>> this
>>> > thread? Pointing at a google doc and only giving partial examples
>is 1)
>>> a
>>> > bit difficult to quickly work out what we are voting on, and 2)
>using a
>>> > google doc isn't great for a longer term record.
>>> >
>>> > Thanks,
>>> > Ash
>>> >
>>> > > On 26 Jul 2019, at 09:14, Jarek Potiuk
><Ja...@polidea.com>
>>> wrote:
>>> > >
>>> > > I would like to recast the vote. Let's start from 0 :) . I would
>like
>>> to
>>> > > keep the July 30th 6pm CEST date, and at least three +1
>(binding)
>>> votes
>>> > are
>>> > > needed to pass. I marked in red the modifications to the
>previous
>>> > proposal.
>>> > >
>>> > > Here is the proposal (details here
>>> > > <
>>> >
>>>
>https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
>>> > >
>>> > > )
>>> > >
>>> > > - *Case 1: A: Get rid of contrib,*
>>> > > - *Case 2: A: Airflow.operators.foo_operator.FooOperator could
>>> > > become airflow.operators.foo.FooOperator*
>>> > > - *Case 3: C:
>>> > >
>airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>>> could
>>> > > become airflow.gcp.operators.bigtable.BigTableOperator*
>>> > > - *Case 4: B: no namespace introduction*
>>> > > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class
>>> names*
>>> > > - *Case 6: We will treat isolated cases on case-by-case (and
>my team
>>> > can
>>> > > do the job of GCP-related operators)*
>>> > > - *Case 7: Native python solution (with better IDE
>integration)*
>>> > >
>>> > > This is my binding (+1) vote.
>>> > >
>>> > >
>>> > >
>>> > > On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
>>> Jarek.Potiuk@polidea.com>
>>> > > wrote:
>>> > >
>>> > >> Hey Felix,
>>> > >>
>>> > >> For 7* -> I am in favour of trying the native one as well (I
>use IDE
>>> > quite
>>> > >> often ;) )
>>> > >>
>>> > >> J.
>>> > >>
>>> > >> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
>>> <fokko@driesprong.frl
>>> > >
>>> > >> wrote:
>>> > >>
>>> > >>> Thanks Kamil for putting the document together. I wasn't able
>to
>>> > respond
>>> > >>> earlier since I was giving Airflow workshops throughout Europe
>:-)
>>> > >>>
>>> > >>> *Case 1: *Solution A → Remove everything from contrib into a
>single
>>> > >>> package.
>>> > >>>
>>> > >>> To make it easier to the end-user, my preference would be to
>get
>>> rid of
>>> > >>> the
>>> > >>> contrib package altogether. What's in contrib or not is a
>tricky
>>> > question,
>>> > >>> I think this is already reflected in the document since
>"maturity"
>>> is
>>> > in
>>> > >>> quotes. What would be the moment to transfer the operator to
>the
>>> core
>>> > or
>>> > >>> vice versa? I'm afraid that renaming contrib to incubating
>will
>>> confuse
>>> > >>> the
>>> > >>> user even more. For people who aren't as known with Airflow it
>isn't
>>> > that
>>> > >>> transparent which operator lives where, especially if you
>don't use
>>> an
>>> > IDE
>>> > >>> such as Ash ;) Besides that I don't think it is worth changing
>the
>>> > public
>>> > >>> API just for a different name.
>>> > >>>
>>> > >>
>>> > >> Ok. I am convinced. I will re-cast the vote with this. It will
>make
>>> > easier
>>> > >> as we will not have to define other rules as those that we
>already
>>> have.
>>> > >>
>>> > >>
>>> > >>> *Case 2:* Slight preference to Solution B (let's say a +0)
>>> > >>>
>>> > >>> I like to search on Github with t, and then search for
>BashOperator.
>>> > After
>>> > >>> the change, you should search for Operator/Bash.
>>> > >>>
>>> > >>> Jarek, I'm confused with your vote on this, from your mail you
>>> state: -
>>> > >>> *Case 2: B: Airflow.operators.foo_operator.FooOperator could
>become
>>> > >>> airflow.operators.foo.FooOperator*. But the B solution is
>keeping it
>>> > the
>>> > >>> same, so that would mean - *Case 2: B:
>>> > >>> Airflow.operators.foo_operator.FooOperator *would stay
>>> > >>> airflow.operators.foo_operator*.FooOperator*
>>> > >>>
>>> > >>> My mistake. It was of course A. I think there is a little
>confusion
>>> > here.
>>> > >> This case is not for changing BashOperator into Bash but for
>changing
>>> > the
>>> > >> *module* name not the *class* name. So
>bash_operator.BashOperator ->
>>> > >> bash.BashOperator :) . you will still search for BashOperator
>in this
>>> > case.
>>> > >>
>>> > >>
>>> > >>> *Case 3:* I'm leaning towards B
>>> > >>>
>>> > >>> Mostly because it isn't as trivial to decide when it gets its
>own
>>> > package
>>> > >>> or not. Besides the cloud providers, there are companies like
>>> > Databricks
>>> > >>> and Qubole which might also end up with their own package
>because
>>> their
>>> > >>> integration is getting richer over time. Moving this is a bit
>>> painful
>>> > >>> because we need to deprecate the old path and move everything
>which
>>> > >>> introduces noise into version control etc.
>>> > >>
>>> > >>
>>> > >> I'd say, we should go for C. As I proposed. It's not as
>difficult as
>>> it
>>> > >> seems to group operators in packages and it has numerous
>advantages.
>>> The
>>> > >> nice thing about deprecations - we can do them in the
>__init__.py of
>>> the
>>> > >> packages which causes the noise fairly small (Git will actually
>>> realise
>>> > >> that the file has been moved and you can follow that. Maybe not
>as
>>> super
>>> > >> easy to merge changes later to 1.10.4 but it's not a rocket
>science
>>> > either.
>>> > >>
>>> > >>
>>> > >>> *Case 7:* Python native solution
>>> > >>>
>>> > >>
>>> > >> Yes.
>>> > >>
>>> > >>
>>> > >>>
>>> > >>> ---
>>> > >>>
>>> > >>> Apart from all, great work!
>>> > >>>
>>> > >>> Cheers, Fokko
>>> > >>>
>>> > >>>
>>> > >>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
>>> > >>> <feluelle@pm.me.invalid
>>> > >>>> :
>>> > >>>
>>> > >>>> I have no strong "No" against any proposed change of these
>cases.
>>> So I
>>> > >>> go
>>> > >>>> with +1 (non-binding).
>>> > >>>>
>>> > >>>> P.S. Thanks Jarek for bringing this up again and your intense
>work
>>> > >>> towards
>>> > >>>> airflow currently :) and thanks to Kamil for even creating
>this
>>> > >>> document. I
>>> > >>>> like how the code is getting more and more consistent and
>clean :)
>>> > >>>>
>>> > >>>> Kind regards,
>>> > >>>> Felix
>>> > >>>>
>>> > >>>> Sent from ProtonMail mobile
>>> > >>>>
>>> > >>>> -------- Original Message --------
>>> > >>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
>>> > >>>>
>>> > >>>>> Hello everyone,
>>> > >>>>>
>>> > >>>>> This email is calling a vote on the changes in import paths.
>It's
>>> > been
>>> > >>>>> discussed in
>>> > >>>>>
>>> > >>>>
>>> > >>>
>>> >
>>>
>https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
>>> > >>>>>
>>> > >>>>> The vote will last for at least 1 week (July 30th 6pm CEST),
>and
>>> at
>>> > >>> least
>>> > >>>>> three +1 (binding) votes have been cast.
>>> > >>>>>
>>> > >>>>> The proposal to vote is based on the document from Kamil
>>> > >>>>> <
>>> > >>>>
>>> > >>>
>>> >
>>>
>https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
>>> > >>>>>
>>> > >>>>>
>>> > >>>>> The proposed solution is:
>>> > >>>>>
>>> > >>>>> - *Case 1: B: Contrib vs not: we move all that are "well"
>tested
>>> and
>>> > >>>>> rename contrib to "incubating" or similar.*
>>> > >>>>> - *Case 2: B: Airflow.operators.foo_operator.FooOperator
>could
>>> > >>>>> become airflow.operators.foo.FooOperator*
>>> > >>>>> - *Case 3: C:
>>> > >>>>>
>airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>>> > could
>>> > >>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
>>> > >>>>> - *Case 4: B: no namespace introduction*
>>> > >>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
>class
>>> names*
>>> > >>>>> - *Case 6: We will treat isolated cases on case-by-case (and
>my
>>> team
>>> > >>> can
>>> > >>>>> do the job of GCP-related operators)*
>>> > >>>>>
>>> > >>>>> This is my (binding) +1 vote.
>>> > >>>>>
>>> > >>>>> Best regards,
>>> > >>>>>
>>> > >>>>> J.
>>> > >>>>>
>>> > >>>>> --
>>> > >>>>>
>>> > >>>>> Jarek Potiuk
>>> > >>>>> Polidea <https://www.polidea.com/> | Principal Software
>Engineer
>>> > >>>>>
>>> > >>>>> M: [+48 660 796 129](tel:+48660796129)
>>> > >>> <[+48660796129](tel:+48660796129)>
>>> > >>>>> [image: Polidea] <https://www.polidea.com/>
>>> > >>>
>>> > >>
>>> > >>
>>> > >> --
>>> > >>
>>> > >> Jarek Potiuk
>>> > >> Polidea <https://www.polidea.com/> | Principal Software
>Engineer
>>> > >>
>>> > >> M: +48 660 796 129 <+48660796129>
>>> > >> [image: Polidea] <https://www.polidea.com/>
>>> > >>
>>> > >>
>>> > >
>>> > > --
>>> > >
>>> > > Jarek Potiuk
>>> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>>> > >
>>> > > M: +48 660 796 129 <+48660796129>
>>> > > [image: Polidea] <https://www.polidea.com/>
>>> >
>>> >
>>>
>>
>>
>> --
>> *Kaxil Naik*
>> *Big Data Consultant | DevOps Data Engineer*
>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
>Neo4j
>> Developer
>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>
>
>
>--
>*Kaxil Naik*
>*Big Data Consultant | DevOps Data Engineer*
>*Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
>Neo4j
>Developer
>*LinkedIn*: https://www.linkedin.com/in/kaxil
Re: [VOTE] Changes in import paths
Posted by Kaxil Naik <ka...@gmail.com>.
I agree with Ash and instead of voting on all 7 Cases together, how about
separate vote emails for all the cases? Each email can have the description
copied over from the doc.
It would probably be a bit easier to track a decision in the future as well.
What do you guys think? @Jarek Potiuk <Ja...@polidea.com> @Driesprong,
Fokko <fo...@driesprong.frl>
On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <ka...@gmail.com> wrote:
> *Case #1* airflow.contrib.{resources} : *Solution A *
>
> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
>
> *Case #3* {aws/azure/gcp}_* : *Solution C*
>
> *Case #4* Separate namespace for resources :
> *Solution A*
>
> *Case #5* *Operator : *Solution A*
>
> *Case #7* : *Solution A*
>
> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish <dp...@gmail.com>
> wrote:
>
>> I have tried to add some clarification to Jarek's summary, but I am a
>> little fuzzy on exact proposal for case 3 so hopefully Jarek can further
>> clarify.
>>
>> According to my reading, in this motion cases 4,5,6 are all either
>> proposal
>> *rejections* or otherwise have no effect, so attention can be focused on
>> cases 1,2,3 and 7.
>>
>> *Motion is to adopt the following:*
>>
>> Case 1 - Solution A - Get rid of contrib
>> All objects will be moved out of contrib folder and into respective
>> non-contrib folder.
>>
>> Case 2 - Solution A - Remove object type suffix from modules
>> Example:
>> Airflow.operators.foo_operator.FooOperator becomes
>> airflow.operators.foo.FooOperator
>> Example:
>> Airflow.hooks.foo_hook.FooOperator becomes airflow.hooks.foo.FooHook
>>
>> Case 3 - conventions for cloud provider etc prefixes
>> *I am not sure exactly what the proposal is.*
>> Example case is airflow/contrib/operators/gcp_bigtable_operator.py
>> Since motion adopts case 1 solution A, we can assume it is now named like
>> so: airflow/operators/gcp_bigtable_operator.py
>> So what is proposed for this case?
>>
>> Case 4 - Solution B - *no change*
>> This proposal was about namespaces but the motion is to reject the
>> proposal.
>>
>> Case 5 - Solution B - *no change*
>> The proposal was to remove class type suffix e.g. remove the Operator in
>> FooOperator, but the motion is to reject this proposal -- i.e. motion is
>> no
>> change on this case, i.e. we resolve to keep "Operator" (and "Sensor")
>> suffixes on class names
>>
>> Case 6 - *No change*
>> This case concerns object naming inconsistencies, but motion does not
>> enact
>> any specific proposal.
>>
>> Case 7 - Solution A - use warnings.warn template for deprecation warnings
>> (instead of zope)
>> Example:
>>
>> # in old_package/old_mod.py
>>
>> import warnings
>> from new_package.new_mod import *
>> warnings.warn("old_package.old_mod has moved to new_package.new_mod.
>> Import
>> of "
>> "old_package.old_mod will become unsupported in version 2",
>> DeprecationWarning, 2)
>>
>> Reference implementation here:
>>
>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
>>
>>
>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
>>
>> I am very happy that this motion now gets rid of contrib. Thank you
>> Sergio
>> for turning the tide with your effective argumentation ;)
>>
>>
>>
>>
>>
>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>> > Could someone summarise what the proposed name changes are, here, in
>> this
>> > thread? Pointing at a google doc and only giving partial examples is 1)
>> a
>> > bit difficult to quickly work out what we are voting on, and 2) using a
>> > google doc isn't great for a longer term record.
>> >
>> > Thanks,
>> > Ash
>> >
>> > > On 26 Jul 2019, at 09:14, Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>> > >
>> > > I would like to recast the vote. Let's start from 0 :) . I would like
>> to
>> > > keep the July 30th 6pm CEST date, and at least three +1 (binding)
>> votes
>> > are
>> > > needed to pass. I marked in red the modifications to the previous
>> > proposal.
>> > >
>> > > Here is the proposal (details here
>> > > <
>> >
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
>> > >
>> > > )
>> > >
>> > > - *Case 1: A: Get rid of contrib,*
>> > > - *Case 2: A: Airflow.operators.foo_operator.FooOperator could
>> > > become airflow.operators.foo.FooOperator*
>> > > - *Case 3: C:
>> > > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>> could
>> > > become airflow.gcp.operators.bigtable.BigTableOperator*
>> > > - *Case 4: B: no namespace introduction*
>> > > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class
>> names*
>> > > - *Case 6: We will treat isolated cases on case-by-case (and my team
>> > can
>> > > do the job of GCP-related operators)*
>> > > - *Case 7: Native python solution (with better IDE integration)*
>> > >
>> > > This is my binding (+1) vote.
>> > >
>> > >
>> > >
>> > > On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
>> Jarek.Potiuk@polidea.com>
>> > > wrote:
>> > >
>> > >> Hey Felix,
>> > >>
>> > >> For 7* -> I am in favour of trying the native one as well (I use IDE
>> > quite
>> > >> often ;) )
>> > >>
>> > >> J.
>> > >>
>> > >> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
>> <fokko@driesprong.frl
>> > >
>> > >> wrote:
>> > >>
>> > >>> Thanks Kamil for putting the document together. I wasn't able to
>> > respond
>> > >>> earlier since I was giving Airflow workshops throughout Europe :-)
>> > >>>
>> > >>> *Case 1: *Solution A → Remove everything from contrib into a single
>> > >>> package.
>> > >>>
>> > >>> To make it easier to the end-user, my preference would be to get
>> rid of
>> > >>> the
>> > >>> contrib package altogether. What's in contrib or not is a tricky
>> > question,
>> > >>> I think this is already reflected in the document since "maturity"
>> is
>> > in
>> > >>> quotes. What would be the moment to transfer the operator to the
>> core
>> > or
>> > >>> vice versa? I'm afraid that renaming contrib to incubating will
>> confuse
>> > >>> the
>> > >>> user even more. For people who aren't as known with Airflow it isn't
>> > that
>> > >>> transparent which operator lives where, especially if you don't use
>> an
>> > IDE
>> > >>> such as Ash ;) Besides that I don't think it is worth changing the
>> > public
>> > >>> API just for a different name.
>> > >>>
>> > >>
>> > >> Ok. I am convinced. I will re-cast the vote with this. It will make
>> > easier
>> > >> as we will not have to define other rules as those that we already
>> have.
>> > >>
>> > >>
>> > >>> *Case 2:* Slight preference to Solution B (let's say a +0)
>> > >>>
>> > >>> I like to search on Github with t, and then search for BashOperator.
>> > After
>> > >>> the change, you should search for Operator/Bash.
>> > >>>
>> > >>> Jarek, I'm confused with your vote on this, from your mail you
>> state: -
>> > >>> *Case 2: B: Airflow.operators.foo_operator.FooOperator could become
>> > >>> airflow.operators.foo.FooOperator*. But the B solution is keeping it
>> > the
>> > >>> same, so that would mean - *Case 2: B:
>> > >>> Airflow.operators.foo_operator.FooOperator *would stay
>> > >>> airflow.operators.foo_operator*.FooOperator*
>> > >>>
>> > >>> My mistake. It was of course A. I think there is a little confusion
>> > here.
>> > >> This case is not for changing BashOperator into Bash but for changing
>> > the
>> > >> *module* name not the *class* name. So bash_operator.BashOperator ->
>> > >> bash.BashOperator :) . you will still search for BashOperator in this
>> > case.
>> > >>
>> > >>
>> > >>> *Case 3:* I'm leaning towards B
>> > >>>
>> > >>> Mostly because it isn't as trivial to decide when it gets its own
>> > package
>> > >>> or not. Besides the cloud providers, there are companies like
>> > Databricks
>> > >>> and Qubole which might also end up with their own package because
>> their
>> > >>> integration is getting richer over time. Moving this is a bit
>> painful
>> > >>> because we need to deprecate the old path and move everything which
>> > >>> introduces noise into version control etc.
>> > >>
>> > >>
>> > >> I'd say, we should go for C. As I proposed. It's not as difficult as
>> it
>> > >> seems to group operators in packages and it has numerous advantages.
>> The
>> > >> nice thing about deprecations - we can do them in the __init__.py of
>> the
>> > >> packages which causes the noise fairly small (Git will actually
>> realise
>> > >> that the file has been moved and you can follow that. Maybe not as
>> super
>> > >> easy to merge changes later to 1.10.4 but it's not a rocket science
>> > either.
>> > >>
>> > >>
>> > >>> *Case 7:* Python native solution
>> > >>>
>> > >>
>> > >> Yes.
>> > >>
>> > >>
>> > >>>
>> > >>> ---
>> > >>>
>> > >>> Apart from all, great work!
>> > >>>
>> > >>> Cheers, Fokko
>> > >>>
>> > >>>
>> > >>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
>> > >>> <feluelle@pm.me.invalid
>> > >>>> :
>> > >>>
>> > >>>> I have no strong "No" against any proposed change of these cases.
>> So I
>> > >>> go
>> > >>>> with +1 (non-binding).
>> > >>>>
>> > >>>> P.S. Thanks Jarek for bringing this up again and your intense work
>> > >>> towards
>> > >>>> airflow currently :) and thanks to Kamil for even creating this
>> > >>> document. I
>> > >>>> like how the code is getting more and more consistent and clean :)
>> > >>>>
>> > >>>> Kind regards,
>> > >>>> Felix
>> > >>>>
>> > >>>> Sent from ProtonMail mobile
>> > >>>>
>> > >>>> -------- Original Message --------
>> > >>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
>> > >>>>
>> > >>>>> Hello everyone,
>> > >>>>>
>> > >>>>> This email is calling a vote on the changes in import paths. It's
>> > been
>> > >>>>> discussed in
>> > >>>>>
>> > >>>>
>> > >>>
>> >
>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
>> > >>>>>
>> > >>>>> The vote will last for at least 1 week (July 30th 6pm CEST), and
>> at
>> > >>> least
>> > >>>>> three +1 (binding) votes have been cast.
>> > >>>>>
>> > >>>>> The proposal to vote is based on the document from Kamil
>> > >>>>> <
>> > >>>>
>> > >>>
>> >
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
>> > >>>>>
>> > >>>>>
>> > >>>>> The proposed solution is:
>> > >>>>>
>> > >>>>> - *Case 1: B: Contrib vs not: we move all that are "well" tested
>> and
>> > >>>>> rename contrib to "incubating" or similar.*
>> > >>>>> - *Case 2: B: Airflow.operators.foo_operator.FooOperator could
>> > >>>>> become airflow.operators.foo.FooOperator*
>> > >>>>> - *Case 3: C:
>> > >>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>> > could
>> > >>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
>> > >>>>> - *Case 4: B: no namespace introduction*
>> > >>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class
>> names*
>> > >>>>> - *Case 6: We will treat isolated cases on case-by-case (and my
>> team
>> > >>> can
>> > >>>>> do the job of GCP-related operators)*
>> > >>>>>
>> > >>>>> This is my (binding) +1 vote.
>> > >>>>>
>> > >>>>> Best regards,
>> > >>>>>
>> > >>>>> J.
>> > >>>>>
>> > >>>>> --
>> > >>>>>
>> > >>>>> Jarek Potiuk
>> > >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>> > >>>>>
>> > >>>>> M: [+48 660 796 129](tel:+48660796129)
>> > >>> <[+48660796129](tel:+48660796129)>
>> > >>>>> [image: Polidea] <https://www.polidea.com/>
>> > >>>
>> > >>
>> > >>
>> > >> --
>> > >>
>> > >> Jarek Potiuk
>> > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
>> > >>
>> > >> M: +48 660 796 129 <+48660796129>
>> > >> [image: Polidea] <https://www.polidea.com/>
>> > >>
>> > >>
>> > >
>> > > --
>> > >
>> > > Jarek Potiuk
>> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> > >
>> > > M: +48 660 796 129 <+48660796129>
>> > > [image: Polidea] <https://www.polidea.com/>
>> >
>> >
>>
>
>
> --
> *Kaxil Naik*
> *Big Data Consultant | DevOps Data Engineer*
> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> Developer
> *LinkedIn*: https://www.linkedin.com/in/kaxil
>
--
*Kaxil Naik*
*Big Data Consultant | DevOps Data Engineer*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*LinkedIn*: https://www.linkedin.com/in/kaxil
Re: [VOTE] Changes in import paths
Posted by Kaxil Naik <ka...@gmail.com>.
*Case #1* airflow.contrib.{resources} : *Solution A *
*Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
*Case #3* {aws/azure/gcp}_* : *Solution C*
*Case #4* Separate namespace for resources :
*Solution A*
*Case #5* *Operator : *Solution A*
*Case #7* : *Solution A*
On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish <dp...@gmail.com>
wrote:
> I have tried to add some clarification to Jarek's summary, but I am a
> little fuzzy on exact proposal for case 3 so hopefully Jarek can further
> clarify.
>
> According to my reading, in this motion cases 4,5,6 are all either proposal
> *rejections* or otherwise have no effect, so attention can be focused on
> cases 1,2,3 and 7.
>
> *Motion is to adopt the following:*
>
> Case 1 - Solution A - Get rid of contrib
> All objects will be moved out of contrib folder and into respective
> non-contrib folder.
>
> Case 2 - Solution A - Remove object type suffix from modules
> Example:
> Airflow.operators.foo_operator.FooOperator becomes
> airflow.operators.foo.FooOperator
> Example:
> Airflow.hooks.foo_hook.FooOperator becomes airflow.hooks.foo.FooHook
>
> Case 3 - conventions for cloud provider etc prefixes
> *I am not sure exactly what the proposal is.*
> Example case is airflow/contrib/operators/gcp_bigtable_operator.py
> Since motion adopts case 1 solution A, we can assume it is now named like
> so: airflow/operators/gcp_bigtable_operator.py
> So what is proposed for this case?
>
> Case 4 - Solution B - *no change*
> This proposal was about namespaces but the motion is to reject the
> proposal.
>
> Case 5 - Solution B - *no change*
> The proposal was to remove class type suffix e.g. remove the Operator in
> FooOperator, but the motion is to reject this proposal -- i.e. motion is no
> change on this case, i.e. we resolve to keep "Operator" (and "Sensor")
> suffixes on class names
>
> Case 6 - *No change*
> This case concerns object naming inconsistencies, but motion does not enact
> any specific proposal.
>
> Case 7 - Solution A - use warnings.warn template for deprecation warnings
> (instead of zope)
> Example:
>
> # in old_package/old_mod.py
>
> import warnings
> from new_package.new_mod import *
> warnings.warn("old_package.old_mod has moved to new_package.new_mod. Import
> of "
> "old_package.old_mod will become unsupported in version 2",
> DeprecationWarning, 2)
>
> Reference implementation here:
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
>
>
> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
>
> I am very happy that this motion now gets rid of contrib. Thank you Sergio
> for turning the tide with your effective argumentation ;)
>
>
>
>
>
> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > Could someone summarise what the proposed name changes are, here, in this
> > thread? Pointing at a google doc and only giving partial examples is 1) a
> > bit difficult to quickly work out what we are voting on, and 2) using a
> > google doc isn't great for a longer term record.
> >
> > Thanks,
> > Ash
> >
> > > On 26 Jul 2019, at 09:14, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> > >
> > > I would like to recast the vote. Let's start from 0 :) . I would like
> to
> > > keep the July 30th 6pm CEST date, and at least three +1 (binding) votes
> > are
> > > needed to pass. I marked in red the modifications to the previous
> > proposal.
> > >
> > > Here is the proposal (details here
> > > <
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> > >
> > > )
> > >
> > > - *Case 1: A: Get rid of contrib,*
> > > - *Case 2: A: Airflow.operators.foo_operator.FooOperator could
> > > become airflow.operators.foo.FooOperator*
> > > - *Case 3: C:
> > > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> could
> > > become airflow.gcp.operators.bigtable.BigTableOperator*
> > > - *Case 4: B: no namespace introduction*
> > > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class names*
> > > - *Case 6: We will treat isolated cases on case-by-case (and my team
> > can
> > > do the job of GCP-related operators)*
> > > - *Case 7: Native python solution (with better IDE integration)*
> > >
> > > This is my binding (+1) vote.
> > >
> > >
> > >
> > > On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> > > wrote:
> > >
> > >> Hey Felix,
> > >>
> > >> For 7* -> I am in favour of trying the native one as well (I use IDE
> > quite
> > >> often ;) )
> > >>
> > >> J.
> > >>
> > >> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> <fokko@driesprong.frl
> > >
> > >> wrote:
> > >>
> > >>> Thanks Kamil for putting the document together. I wasn't able to
> > respond
> > >>> earlier since I was giving Airflow workshops throughout Europe :-)
> > >>>
> > >>> *Case 1: *Solution A → Remove everything from contrib into a single
> > >>> package.
> > >>>
> > >>> To make it easier to the end-user, my preference would be to get rid
> of
> > >>> the
> > >>> contrib package altogether. What's in contrib or not is a tricky
> > question,
> > >>> I think this is already reflected in the document since "maturity" is
> > in
> > >>> quotes. What would be the moment to transfer the operator to the core
> > or
> > >>> vice versa? I'm afraid that renaming contrib to incubating will
> confuse
> > >>> the
> > >>> user even more. For people who aren't as known with Airflow it isn't
> > that
> > >>> transparent which operator lives where, especially if you don't use
> an
> > IDE
> > >>> such as Ash ;) Besides that I don't think it is worth changing the
> > public
> > >>> API just for a different name.
> > >>>
> > >>
> > >> Ok. I am convinced. I will re-cast the vote with this. It will make
> > easier
> > >> as we will not have to define other rules as those that we already
> have.
> > >>
> > >>
> > >>> *Case 2:* Slight preference to Solution B (let's say a +0)
> > >>>
> > >>> I like to search on Github with t, and then search for BashOperator.
> > After
> > >>> the change, you should search for Operator/Bash.
> > >>>
> > >>> Jarek, I'm confused with your vote on this, from your mail you
> state: -
> > >>> *Case 2: B: Airflow.operators.foo_operator.FooOperator could become
> > >>> airflow.operators.foo.FooOperator*. But the B solution is keeping it
> > the
> > >>> same, so that would mean - *Case 2: B:
> > >>> Airflow.operators.foo_operator.FooOperator *would stay
> > >>> airflow.operators.foo_operator*.FooOperator*
> > >>>
> > >>> My mistake. It was of course A. I think there is a little confusion
> > here.
> > >> This case is not for changing BashOperator into Bash but for changing
> > the
> > >> *module* name not the *class* name. So bash_operator.BashOperator ->
> > >> bash.BashOperator :) . you will still search for BashOperator in this
> > case.
> > >>
> > >>
> > >>> *Case 3:* I'm leaning towards B
> > >>>
> > >>> Mostly because it isn't as trivial to decide when it gets its own
> > package
> > >>> or not. Besides the cloud providers, there are companies like
> > Databricks
> > >>> and Qubole which might also end up with their own package because
> their
> > >>> integration is getting richer over time. Moving this is a bit painful
> > >>> because we need to deprecate the old path and move everything which
> > >>> introduces noise into version control etc.
> > >>
> > >>
> > >> I'd say, we should go for C. As I proposed. It's not as difficult as
> it
> > >> seems to group operators in packages and it has numerous advantages.
> The
> > >> nice thing about deprecations - we can do them in the __init__.py of
> the
> > >> packages which causes the noise fairly small (Git will actually
> realise
> > >> that the file has been moved and you can follow that. Maybe not as
> super
> > >> easy to merge changes later to 1.10.4 but it's not a rocket science
> > either.
> > >>
> > >>
> > >>> *Case 7:* Python native solution
> > >>>
> > >>
> > >> Yes.
> > >>
> > >>
> > >>>
> > >>> ---
> > >>>
> > >>> Apart from all, great work!
> > >>>
> > >>> Cheers, Fokko
> > >>>
> > >>>
> > >>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> > >>> <feluelle@pm.me.invalid
> > >>>> :
> > >>>
> > >>>> I have no strong "No" against any proposed change of these cases.
> So I
> > >>> go
> > >>>> with +1 (non-binding).
> > >>>>
> > >>>> P.S. Thanks Jarek for bringing this up again and your intense work
> > >>> towards
> > >>>> airflow currently :) and thanks to Kamil for even creating this
> > >>> document. I
> > >>>> like how the code is getting more and more consistent and clean :)
> > >>>>
> > >>>> Kind regards,
> > >>>> Felix
> > >>>>
> > >>>> Sent from ProtonMail mobile
> > >>>>
> > >>>> -------- Original Message --------
> > >>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> > >>>>
> > >>>>> Hello everyone,
> > >>>>>
> > >>>>> This email is calling a vote on the changes in import paths. It's
> > been
> > >>>>> discussed in
> > >>>>>
> > >>>>
> > >>>
> >
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> > >>>>>
> > >>>>> The vote will last for at least 1 week (July 30th 6pm CEST), and at
> > >>> least
> > >>>>> three +1 (binding) votes have been cast.
> > >>>>>
> > >>>>> The proposal to vote is based on the document from Kamil
> > >>>>> <
> > >>>>
> > >>>
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> > >>>>>
> > >>>>>
> > >>>>> The proposed solution is:
> > >>>>>
> > >>>>> - *Case 1: B: Contrib vs not: we move all that are "well" tested
> and
> > >>>>> rename contrib to "incubating" or similar.*
> > >>>>> - *Case 2: B: Airflow.operators.foo_operator.FooOperator could
> > >>>>> become airflow.operators.foo.FooOperator*
> > >>>>> - *Case 3: C:
> > >>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > could
> > >>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> > >>>>> - *Case 4: B: no namespace introduction*
> > >>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class
> names*
> > >>>>> - *Case 6: We will treat isolated cases on case-by-case (and my
> team
> > >>> can
> > >>>>> do the job of GCP-related operators)*
> > >>>>>
> > >>>>> This is my (binding) +1 vote.
> > >>>>>
> > >>>>> Best regards,
> > >>>>>
> > >>>>> J.
> > >>>>>
> > >>>>> --
> > >>>>>
> > >>>>> Jarek Potiuk
> > >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>>>>
> > >>>>> M: [+48 660 796 129](tel:+48660796129)
> > >>> <[+48660796129](tel:+48660796129)>
> > >>>>> [image: Polidea] <https://www.polidea.com/>
> > >>>
> > >>
> > >>
> > >> --
> > >>
> > >> Jarek Potiuk
> > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>
> > >> M: +48 660 796 129 <+48660796129>
> > >> [image: Polidea] <https://www.polidea.com/>
> > >>
> > >>
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> >
> >
>
--
*Kaxil Naik*
*Big Data Consultant | DevOps Data Engineer*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*LinkedIn*: https://www.linkedin.com/in/kaxil
Re: [VOTE] Changes in import paths
Posted by Daniel Standish <dp...@gmail.com>.
I have tried to add some clarification to Jarek's summary, but I am a
little fuzzy on exact proposal for case 3 so hopefully Jarek can further
clarify.
According to my reading, in this motion cases 4,5,6 are all either proposal
*rejections* or otherwise have no effect, so attention can be focused on
cases 1,2,3 and 7.
*Motion is to adopt the following:*
Case 1 - Solution A - Get rid of contrib
All objects will be moved out of contrib folder and into respective
non-contrib folder.
Case 2 - Solution A - Remove object type suffix from modules
Example:
Airflow.operators.foo_operator.FooOperator becomes
airflow.operators.foo.FooOperator
Example:
Airflow.hooks.foo_hook.FooOperator becomes airflow.hooks.foo.FooHook
Case 3 - conventions for cloud provider etc prefixes
*I am not sure exactly what the proposal is.*
Example case is airflow/contrib/operators/gcp_bigtable_operator.py
Since motion adopts case 1 solution A, we can assume it is now named like
so: airflow/operators/gcp_bigtable_operator.py
So what is proposed for this case?
Case 4 - Solution B - *no change*
This proposal was about namespaces but the motion is to reject the proposal.
Case 5 - Solution B - *no change*
The proposal was to remove class type suffix e.g. remove the Operator in
FooOperator, but the motion is to reject this proposal -- i.e. motion is no
change on this case, i.e. we resolve to keep "Operator" (and "Sensor")
suffixes on class names
Case 6 - *No change*
This case concerns object naming inconsistencies, but motion does not enact
any specific proposal.
Case 7 - Solution A - use warnings.warn template for deprecation warnings
(instead of zope)
Example:
# in old_package/old_mod.py
import warnings
from new_package.new_mod import *
warnings.warn("old_package.old_mod has moved to new_package.new_mod. Import
of "
"old_package.old_mod will become unsupported in version 2",
DeprecationWarning, 2)
Reference implementation here:
https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
I am very happy that this motion now gets rid of contrib. Thank you Sergio
for turning the tide with your effective argumentation ;)
On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> Could someone summarise what the proposed name changes are, here, in this
> thread? Pointing at a google doc and only giving partial examples is 1) a
> bit difficult to quickly work out what we are voting on, and 2) using a
> google doc isn't great for a longer term record.
>
> Thanks,
> Ash
>
> > On 26 Jul 2019, at 09:14, Jarek Potiuk <Ja...@polidea.com> wrote:
> >
> > I would like to recast the vote. Let's start from 0 :) . I would like to
> > keep the July 30th 6pm CEST date, and at least three +1 (binding) votes
> are
> > needed to pass. I marked in red the modifications to the previous
> proposal.
> >
> > Here is the proposal (details here
> > <
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> >
> > )
> >
> > - *Case 1: A: Get rid of contrib,*
> > - *Case 2: A: Airflow.operators.foo_operator.FooOperator could
> > become airflow.operators.foo.FooOperator*
> > - *Case 3: C:
> > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator could
> > become airflow.gcp.operators.bigtable.BigTableOperator*
> > - *Case 4: B: no namespace introduction*
> > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class names*
> > - *Case 6: We will treat isolated cases on case-by-case (and my team
> can
> > do the job of GCP-related operators)*
> > - *Case 7: Native python solution (with better IDE integration)*
> >
> > This is my binding (+1) vote.
> >
> >
> >
> > On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> >> Hey Felix,
> >>
> >> For 7* -> I am in favour of trying the native one as well (I use IDE
> quite
> >> often ;) )
> >>
> >> J.
> >>
> >> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko <fokko@driesprong.frl
> >
> >> wrote:
> >>
> >>> Thanks Kamil for putting the document together. I wasn't able to
> respond
> >>> earlier since I was giving Airflow workshops throughout Europe :-)
> >>>
> >>> *Case 1: *Solution A → Remove everything from contrib into a single
> >>> package.
> >>>
> >>> To make it easier to the end-user, my preference would be to get rid of
> >>> the
> >>> contrib package altogether. What's in contrib or not is a tricky
> question,
> >>> I think this is already reflected in the document since "maturity" is
> in
> >>> quotes. What would be the moment to transfer the operator to the core
> or
> >>> vice versa? I'm afraid that renaming contrib to incubating will confuse
> >>> the
> >>> user even more. For people who aren't as known with Airflow it isn't
> that
> >>> transparent which operator lives where, especially if you don't use an
> IDE
> >>> such as Ash ;) Besides that I don't think it is worth changing the
> public
> >>> API just for a different name.
> >>>
> >>
> >> Ok. I am convinced. I will re-cast the vote with this. It will make
> easier
> >> as we will not have to define other rules as those that we already have.
> >>
> >>
> >>> *Case 2:* Slight preference to Solution B (let's say a +0)
> >>>
> >>> I like to search on Github with t, and then search for BashOperator.
> After
> >>> the change, you should search for Operator/Bash.
> >>>
> >>> Jarek, I'm confused with your vote on this, from your mail you state: -
> >>> *Case 2: B: Airflow.operators.foo_operator.FooOperator could become
> >>> airflow.operators.foo.FooOperator*. But the B solution is keeping it
> the
> >>> same, so that would mean - *Case 2: B:
> >>> Airflow.operators.foo_operator.FooOperator *would stay
> >>> airflow.operators.foo_operator*.FooOperator*
> >>>
> >>> My mistake. It was of course A. I think there is a little confusion
> here.
> >> This case is not for changing BashOperator into Bash but for changing
> the
> >> *module* name not the *class* name. So bash_operator.BashOperator ->
> >> bash.BashOperator :) . you will still search for BashOperator in this
> case.
> >>
> >>
> >>> *Case 3:* I'm leaning towards B
> >>>
> >>> Mostly because it isn't as trivial to decide when it gets its own
> package
> >>> or not. Besides the cloud providers, there are companies like
> Databricks
> >>> and Qubole which might also end up with their own package because their
> >>> integration is getting richer over time. Moving this is a bit painful
> >>> because we need to deprecate the old path and move everything which
> >>> introduces noise into version control etc.
> >>
> >>
> >> I'd say, we should go for C. As I proposed. It's not as difficult as it
> >> seems to group operators in packages and it has numerous advantages. The
> >> nice thing about deprecations - we can do them in the __init__.py of the
> >> packages which causes the noise fairly small (Git will actually realise
> >> that the file has been moved and you can follow that. Maybe not as super
> >> easy to merge changes later to 1.10.4 but it's not a rocket science
> either.
> >>
> >>
> >>> *Case 7:* Python native solution
> >>>
> >>
> >> Yes.
> >>
> >>
> >>>
> >>> ---
> >>>
> >>> Apart from all, great work!
> >>>
> >>> Cheers, Fokko
> >>>
> >>>
> >>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> >>> <feluelle@pm.me.invalid
> >>>> :
> >>>
> >>>> I have no strong "No" against any proposed change of these cases. So I
> >>> go
> >>>> with +1 (non-binding).
> >>>>
> >>>> P.S. Thanks Jarek for bringing this up again and your intense work
> >>> towards
> >>>> airflow currently :) and thanks to Kamil for even creating this
> >>> document. I
> >>>> like how the code is getting more and more consistent and clean :)
> >>>>
> >>>> Kind regards,
> >>>> Felix
> >>>>
> >>>> Sent from ProtonMail mobile
> >>>>
> >>>> -------- Original Message --------
> >>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> >>>>
> >>>>> Hello everyone,
> >>>>>
> >>>>> This email is calling a vote on the changes in import paths. It's
> been
> >>>>> discussed in
> >>>>>
> >>>>
> >>>
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> >>>>>
> >>>>> The vote will last for at least 1 week (July 30th 6pm CEST), and at
> >>> least
> >>>>> three +1 (binding) votes have been cast.
> >>>>>
> >>>>> The proposal to vote is based on the document from Kamil
> >>>>> <
> >>>>
> >>>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> >>>>>
> >>>>>
> >>>>> The proposed solution is:
> >>>>>
> >>>>> - *Case 1: B: Contrib vs not: we move all that are "well" tested and
> >>>>> rename contrib to "incubating" or similar.*
> >>>>> - *Case 2: B: Airflow.operators.foo_operator.FooOperator could
> >>>>> become airflow.operators.foo.FooOperator*
> >>>>> - *Case 3: C:
> >>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> could
> >>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> >>>>> - *Case 4: B: no namespace introduction*
> >>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class names*
> >>>>> - *Case 6: We will treat isolated cases on case-by-case (and my team
> >>> can
> >>>>> do the job of GCP-related operators)*
> >>>>>
> >>>>> This is my (binding) +1 vote.
> >>>>>
> >>>>> Best regards,
> >>>>>
> >>>>> J.
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Jarek Potiuk
> >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>>>
> >>>>> M: [+48 660 796 129](tel:+48660796129)
> >>> <[+48660796129](tel:+48660796129)>
> >>>>> [image: Polidea] <https://www.polidea.com/>
> >>>
> >>
> >>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>
> >> M: +48 660 796 129 <+48660796129>
> >> [image: Polidea] <https://www.polidea.com/>
> >>
> >>
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
>
>
Re: [VOTE] Changes in import paths
Posted by Ash Berlin-Taylor <as...@apache.org>.
Could someone summarise what the proposed name changes are, here, in this thread? Pointing at a google doc and only giving partial examples is 1) a bit difficult to quickly work out what we are voting on, and 2) using a google doc isn't great for a longer term record.
Thanks,
Ash
> On 26 Jul 2019, at 09:14, Jarek Potiuk <Ja...@polidea.com> wrote:
>
> I would like to recast the vote. Let's start from 0 :) . I would like to
> keep the July 30th 6pm CEST date, and at least three +1 (binding) votes are
> needed to pass. I marked in red the modifications to the previous proposal.
>
> Here is the proposal (details here
> <https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo>
> )
>
> - *Case 1: A: Get rid of contrib,*
> - *Case 2: A: Airflow.operators.foo_operator.FooOperator could
> become airflow.operators.foo.FooOperator*
> - *Case 3: C:
> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator could
> become airflow.gcp.operators.bigtable.BigTableOperator*
> - *Case 4: B: no namespace introduction*
> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class names*
> - *Case 6: We will treat isolated cases on case-by-case (and my team can
> do the job of GCP-related operators)*
> - *Case 7: Native python solution (with better IDE integration)*
>
> This is my binding (+1) vote.
>
>
>
> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Hey Felix,
>>
>> For 7* -> I am in favour of trying the native one as well (I use IDE quite
>> often ;) )
>>
>> J.
>>
>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko <fo...@driesprong.frl>
>> wrote:
>>
>>> Thanks Kamil for putting the document together. I wasn't able to respond
>>> earlier since I was giving Airflow workshops throughout Europe :-)
>>>
>>> *Case 1: *Solution A → Remove everything from contrib into a single
>>> package.
>>>
>>> To make it easier to the end-user, my preference would be to get rid of
>>> the
>>> contrib package altogether. What's in contrib or not is a tricky question,
>>> I think this is already reflected in the document since "maturity" is in
>>> quotes. What would be the moment to transfer the operator to the core or
>>> vice versa? I'm afraid that renaming contrib to incubating will confuse
>>> the
>>> user even more. For people who aren't as known with Airflow it isn't that
>>> transparent which operator lives where, especially if you don't use an IDE
>>> such as Ash ;) Besides that I don't think it is worth changing the public
>>> API just for a different name.
>>>
>>
>> Ok. I am convinced. I will re-cast the vote with this. It will make easier
>> as we will not have to define other rules as those that we already have.
>>
>>
>>> *Case 2:* Slight preference to Solution B (let's say a +0)
>>>
>>> I like to search on Github with t, and then search for BashOperator. After
>>> the change, you should search for Operator/Bash.
>>>
>>> Jarek, I'm confused with your vote on this, from your mail you state: -
>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator could become
>>> airflow.operators.foo.FooOperator*. But the B solution is keeping it the
>>> same, so that would mean - *Case 2: B:
>>> Airflow.operators.foo_operator.FooOperator *would stay
>>> airflow.operators.foo_operator*.FooOperator*
>>>
>>> My mistake. It was of course A. I think there is a little confusion here.
>> This case is not for changing BashOperator into Bash but for changing the
>> *module* name not the *class* name. So bash_operator.BashOperator ->
>> bash.BashOperator :) . you will still search for BashOperator in this case.
>>
>>
>>> *Case 3:* I'm leaning towards B
>>>
>>> Mostly because it isn't as trivial to decide when it gets its own package
>>> or not. Besides the cloud providers, there are companies like Databricks
>>> and Qubole which might also end up with their own package because their
>>> integration is getting richer over time. Moving this is a bit painful
>>> because we need to deprecate the old path and move everything which
>>> introduces noise into version control etc.
>>
>>
>> I'd say, we should go for C. As I proposed. It's not as difficult as it
>> seems to group operators in packages and it has numerous advantages. The
>> nice thing about deprecations - we can do them in the __init__.py of the
>> packages which causes the noise fairly small (Git will actually realise
>> that the file has been moved and you can follow that. Maybe not as super
>> easy to merge changes later to 1.10.4 but it's not a rocket science either.
>>
>>
>>> *Case 7:* Python native solution
>>>
>>
>> Yes.
>>
>>
>>>
>>> ---
>>>
>>> Apart from all, great work!
>>>
>>> Cheers, Fokko
>>>
>>>
>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
>>> <feluelle@pm.me.invalid
>>>> :
>>>
>>>> I have no strong "No" against any proposed change of these cases. So I
>>> go
>>>> with +1 (non-binding).
>>>>
>>>> P.S. Thanks Jarek for bringing this up again and your intense work
>>> towards
>>>> airflow currently :) and thanks to Kamil for even creating this
>>> document. I
>>>> like how the code is getting more and more consistent and clean :)
>>>>
>>>> Kind regards,
>>>> Felix
>>>>
>>>> Sent from ProtonMail mobile
>>>>
>>>> -------- Original Message --------
>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
>>>>
>>>>> Hello everyone,
>>>>>
>>>>> This email is calling a vote on the changes in import paths. It's been
>>>>> discussed in
>>>>>
>>>>
>>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
>>>>>
>>>>> The vote will last for at least 1 week (July 30th 6pm CEST), and at
>>> least
>>>>> three +1 (binding) votes have been cast.
>>>>>
>>>>> The proposal to vote is based on the document from Kamil
>>>>> <
>>>>
>>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
>>>>>
>>>>>
>>>>> The proposed solution is:
>>>>>
>>>>> - *Case 1: B: Contrib vs not: we move all that are "well" tested and
>>>>> rename contrib to "incubating" or similar.*
>>>>> - *Case 2: B: Airflow.operators.foo_operator.FooOperator could
>>>>> become airflow.operators.foo.FooOperator*
>>>>> - *Case 3: C:
>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator could
>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
>>>>> - *Case 4: B: no namespace introduction*
>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class names*
>>>>> - *Case 6: We will treat isolated cases on case-by-case (and my team
>>> can
>>>>> do the job of GCP-related operators)*
>>>>>
>>>>> This is my (binding) +1 vote.
>>>>>
>>>>> Best regards,
>>>>>
>>>>> J.
>>>>>
>>>>> --
>>>>>
>>>>> Jarek Potiuk
>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>
>>>>> M: [+48 660 796 129](tel:+48660796129)
>>> <[+48660796129](tel:+48660796129)>
>>>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Changes in import paths
Posted by Jarek Potiuk <Ja...@polidea.com>.
I would like to recast the vote. Let's start from 0 :) . I would like to
keep the July 30th 6pm CEST date, and at least three +1 (binding) votes are
needed to pass. I marked in red the modifications to the previous proposal.
Here is the proposal (details here
<https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo>
)
- *Case 1: A: Get rid of contrib,*
- *Case 2: A: Airflow.operators.foo_operator.FooOperator could
become airflow.operators.foo.FooOperator*
- *Case 3: C:
airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator could
become airflow.gcp.operators.bigtable.BigTableOperator*
- *Case 4: B: no namespace introduction*
- *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class names*
- *Case 6: We will treat isolated cases on case-by-case (and my team can
do the job of GCP-related operators)*
- *Case 7: Native python solution (with better IDE integration)*
This is my binding (+1) vote.
On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Hey Felix,
>
> For 7* -> I am in favour of trying the native one as well (I use IDE quite
> often ;) )
>
> J.
>
> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
>> Thanks Kamil for putting the document together. I wasn't able to respond
>> earlier since I was giving Airflow workshops throughout Europe :-)
>>
>> *Case 1: *Solution A → Remove everything from contrib into a single
>> package.
>>
>> To make it easier to the end-user, my preference would be to get rid of
>> the
>> contrib package altogether. What's in contrib or not is a tricky question,
>> I think this is already reflected in the document since "maturity" is in
>> quotes. What would be the moment to transfer the operator to the core or
>> vice versa? I'm afraid that renaming contrib to incubating will confuse
>> the
>> user even more. For people who aren't as known with Airflow it isn't that
>> transparent which operator lives where, especially if you don't use an IDE
>> such as Ash ;) Besides that I don't think it is worth changing the public
>> API just for a different name.
>>
>
> Ok. I am convinced. I will re-cast the vote with this. It will make easier
> as we will not have to define other rules as those that we already have.
>
>
>> *Case 2:* Slight preference to Solution B (let's say a +0)
>>
>> I like to search on Github with t, and then search for BashOperator. After
>> the change, you should search for Operator/Bash.
>>
>> Jarek, I'm confused with your vote on this, from your mail you state: -
>> *Case 2: B: Airflow.operators.foo_operator.FooOperator could become
>> airflow.operators.foo.FooOperator*. But the B solution is keeping it the
>> same, so that would mean - *Case 2: B:
>> Airflow.operators.foo_operator.FooOperator *would stay
>> airflow.operators.foo_operator*.FooOperator*
>>
>> My mistake. It was of course A. I think there is a little confusion here.
> This case is not for changing BashOperator into Bash but for changing the
> *module* name not the *class* name. So bash_operator.BashOperator ->
> bash.BashOperator :) . you will still search for BashOperator in this case.
>
>
>> *Case 3:* I'm leaning towards B
>>
>> Mostly because it isn't as trivial to decide when it gets its own package
>> or not. Besides the cloud providers, there are companies like Databricks
>> and Qubole which might also end up with their own package because their
>> integration is getting richer over time. Moving this is a bit painful
>> because we need to deprecate the old path and move everything which
>> introduces noise into version control etc.
>
>
> I'd say, we should go for C. As I proposed. It's not as difficult as it
> seems to group operators in packages and it has numerous advantages. The
> nice thing about deprecations - we can do them in the __init__.py of the
> packages which causes the noise fairly small (Git will actually realise
> that the file has been moved and you can follow that. Maybe not as super
> easy to merge changes later to 1.10.4 but it's not a rocket science either.
>
>
>> *Case 7:* Python native solution
>>
>
> Yes.
>
>
>>
>> ---
>>
>> Apart from all, great work!
>>
>> Cheers, Fokko
>>
>>
>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
>> <feluelle@pm.me.invalid
>> >:
>>
>> > I have no strong "No" against any proposed change of these cases. So I
>> go
>> > with +1 (non-binding).
>> >
>> > P.S. Thanks Jarek for bringing this up again and your intense work
>> towards
>> > airflow currently :) and thanks to Kamil for even creating this
>> document. I
>> > like how the code is getting more and more consistent and clean :)
>> >
>> > Kind regards,
>> > Felix
>> >
>> > Sent from ProtonMail mobile
>> >
>> > -------- Original Message --------
>> > On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
>> >
>> > > Hello everyone,
>> > >
>> > > This email is calling a vote on the changes in import paths. It's been
>> > > discussed in
>> > >
>> >
>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
>> > >
>> > > The vote will last for at least 1 week (July 30th 6pm CEST), and at
>> least
>> > > three +1 (binding) votes have been cast.
>> > >
>> > > The proposal to vote is based on the document from Kamil
>> > > <
>> >
>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
>> > >
>> > >
>> > > The proposed solution is:
>> > >
>> > > - *Case 1: B: Contrib vs not: we move all that are "well" tested and
>> > > rename contrib to "incubating" or similar.*
>> > > - *Case 2: B: Airflow.operators.foo_operator.FooOperator could
>> > > become airflow.operators.foo.FooOperator*
>> > > - *Case 3: C:
>> > > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator could
>> > > become airflow.gcp.operators.bigtable.BigTableOperator*
>> > > - *Case 4: B: no namespace introduction*
>> > > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class names*
>> > > - *Case 6: We will treat isolated cases on case-by-case (and my team
>> can
>> > > do the job of GCP-related operators)*
>> > >
>> > > This is my (binding) +1 vote.
>> > >
>> > > Best regards,
>> > >
>> > > J.
>> > >
>> > > --
>> > >
>> > > Jarek Potiuk
>> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> > >
>> > > M: [+48 660 796 129](tel:+48660796129)
>> <[+48660796129](tel:+48660796129)>
>> > > [image: Polidea] <https://www.polidea.com/>
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Changes in import paths
Posted by Jarek Potiuk <Ja...@polidea.com>.
Hey Felix,
For 7* -> I am in favour of trying the native one as well (I use IDE quite
often ;) )
J.
On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> Thanks Kamil for putting the document together. I wasn't able to respond
> earlier since I was giving Airflow workshops throughout Europe :-)
>
> *Case 1: *Solution A → Remove everything from contrib into a single
> package.
>
> To make it easier to the end-user, my preference would be to get rid of the
> contrib package altogether. What's in contrib or not is a tricky question,
> I think this is already reflected in the document since "maturity" is in
> quotes. What would be the moment to transfer the operator to the core or
> vice versa? I'm afraid that renaming contrib to incubating will confuse the
> user even more. For people who aren't as known with Airflow it isn't that
> transparent which operator lives where, especially if you don't use an IDE
> such as Ash ;) Besides that I don't think it is worth changing the public
> API just for a different name.
>
Ok. I am convinced. I will re-cast the vote with this. It will make easier
as we will not have to define other rules as those that we already have.
> *Case 2:* Slight preference to Solution B (let's say a +0)
>
> I like to search on Github with t, and then search for BashOperator. After
> the change, you should search for Operator/Bash.
>
> Jarek, I'm confused with your vote on this, from your mail you state: -
> *Case 2: B: Airflow.operators.foo_operator.FooOperator could become
> airflow.operators.foo.FooOperator*. But the B solution is keeping it the
> same, so that would mean - *Case 2: B:
> Airflow.operators.foo_operator.FooOperator *would stay
> airflow.operators.foo_operator*.FooOperator*
>
> My mistake. It was of course A. I think there is a little confusion here.
This case is not for changing BashOperator into Bash but for changing the
*module* name not the *class* name. So bash_operator.BashOperator ->
bash.BashOperator :) . you will still search for BashOperator in this case.
> *Case 3:* I'm leaning towards B
>
> Mostly because it isn't as trivial to decide when it gets its own package
> or not. Besides the cloud providers, there are companies like Databricks
> and Qubole which might also end up with their own package because their
> integration is getting richer over time. Moving this is a bit painful
> because we need to deprecate the old path and move everything which
> introduces noise into version control etc.
I'd say, we should go for C. As I proposed. It's not as difficult as it
seems to group operators in packages and it has numerous advantages. The
nice thing about deprecations - we can do them in the __init__.py of the
packages which causes the noise fairly small (Git will actually realise
that the file has been moved and you can follow that. Maybe not as super
easy to merge changes later to 1.10.4 but it's not a rocket science either.
> *Case 7:* Python native solution
>
Yes.
>
> ---
>
> Apart from all, great work!
>
> Cheers, Fokko
>
>
> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> <feluelle@pm.me.invalid
> >:
>
> > I have no strong "No" against any proposed change of these cases. So I go
> > with +1 (non-binding).
> >
> > P.S. Thanks Jarek for bringing this up again and your intense work
> towards
> > airflow currently :) and thanks to Kamil for even creating this
> document. I
> > like how the code is getting more and more consistent and clean :)
> >
> > Kind regards,
> > Felix
> >
> > Sent from ProtonMail mobile
> >
> > -------- Original Message --------
> > On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> >
> > > Hello everyone,
> > >
> > > This email is calling a vote on the changes in import paths. It's been
> > > discussed in
> > >
> >
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> > >
> > > The vote will last for at least 1 week (July 30th 6pm CEST), and at
> least
> > > three +1 (binding) votes have been cast.
> > >
> > > The proposal to vote is based on the document from Kamil
> > > <
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> > >
> > >
> > > The proposed solution is:
> > >
> > > - *Case 1: B: Contrib vs not: we move all that are "well" tested and
> > > rename contrib to "incubating" or similar.*
> > > - *Case 2: B: Airflow.operators.foo_operator.FooOperator could
> > > become airflow.operators.foo.FooOperator*
> > > - *Case 3: C:
> > > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator could
> > > become airflow.gcp.operators.bigtable.BigTableOperator*
> > > - *Case 4: B: no namespace introduction*
> > > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class names*
> > > - *Case 6: We will treat isolated cases on case-by-case (and my team
> can
> > > do the job of GCP-related operators)*
> > >
> > > This is my (binding) +1 vote.
> > >
> > > Best regards,
> > >
> > > J.
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: [+48 660 796 129](tel:+48660796129)
> <[+48660796129](tel:+48660796129)>
> > > [image: Polidea] <https://www.polidea.com/>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Changes in import paths
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Thanks Kamil for putting the document together. I wasn't able to respond
earlier since I was giving Airflow workshops throughout Europe :-)
*Case 1: *Solution A → Remove everything from contrib into a single
package.
To make it easier to the end-user, my preference would be to get rid of the
contrib package altogether. What's in contrib or not is a tricky question,
I think this is already reflected in the document since "maturity" is in
quotes. What would be the moment to transfer the operator to the core or
vice versa? I'm afraid that renaming contrib to incubating will confuse the
user even more. For people who aren't as known with Airflow it isn't that
transparent which operator lives where, especially if you don't use an IDE
such as Ash ;) Besides that I don't think it is worth changing the public
API just for a different name.
*Case 2:* Slight preference to Solution B (let's say a +0)
I like to search on Github with t, and then search for BashOperator. After
the change, you should search for Operator/Bash.
Jarek, I'm confused with your vote on this, from your mail you state: -
*Case 2: B: Airflow.operators.foo_operator.FooOperator could become
airflow.operators.foo.FooOperator*. But the B solution is keeping it the
same, so that would mean - *Case 2: B:
Airflow.operators.foo_operator.FooOperator *would stay
airflow.operators.foo_operator*.FooOperator*
*Case 3:* I'm leaning towards B
Mostly because it isn't as trivial to decide when it gets its own package
or not. Besides the cloud providers, there are companies like Databricks
and Qubole which might also end up with their own package because their
integration is getting richer over time. Moving this is a bit painful
because we need to deprecate the old path and move everything which
introduces noise into version control etc.
*C**ase 4: *B: No namespace introduction
*Case 5: *B: Keep "Operator" (and "Sensor") suffixes on class names
For the import is might look redundant, but for the UI I like to have it
explicit. Also when you have Python code itself, I prefer DummyOperator( ..
) instead of Dummy( .. ). Let's quote the Python zen here: Explicit is
better than implicit.
*Case 6: *+1 Consistency is king.
*Case 7:* Python native solution
Jarek, what's your vote on this?
---
Apart from all, great work!
Cheers, Fokko
Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall <feluelle@pm.me.invalid
>:
> I have no strong "No" against any proposed change of these cases. So I go
> with +1 (non-binding).
>
> P.S. Thanks Jarek for bringing this up again and your intense work towards
> airflow currently :) and thanks to Kamil for even creating this document. I
> like how the code is getting more and more consistent and clean :)
>
> Kind regards,
> Felix
>
> Sent from ProtonMail mobile
>
> -------- Original Message --------
> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
>
> > Hello everyone,
> >
> > This email is calling a vote on the changes in import paths. It's been
> > discussed in
> >
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> >
> > The vote will last for at least 1 week (July 30th 6pm CEST), and at least
> > three +1 (binding) votes have been cast.
> >
> > The proposal to vote is based on the document from Kamil
> > <
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> >
> >
> > The proposed solution is:
> >
> > - *Case 1: B: Contrib vs not: we move all that are "well" tested and
> > rename contrib to "incubating" or similar.*
> > - *Case 2: B: Airflow.operators.foo_operator.FooOperator could
> > become airflow.operators.foo.FooOperator*
> > - *Case 3: C:
> > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator could
> > become airflow.gcp.operators.bigtable.BigTableOperator*
> > - *Case 4: B: no namespace introduction*
> > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class names*
> > - *Case 6: We will treat isolated cases on case-by-case (and my team can
> > do the job of GCP-related operators)*
> >
> > This is my (binding) +1 vote.
> >
> > Best regards,
> >
> > J.
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: [+48 660 796 129](tel:+48660796129) <[+48660796129](tel:+48660796129)>
> > [image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Changes in import paths
Posted by Felix Uellendall <fe...@pm.me.INVALID>.
I have no strong "No" against any proposed change of these cases. So I go with +1 (non-binding).
P.S. Thanks Jarek for bringing this up again and your intense work towards airflow currently :) and thanks to Kamil for even creating this document. I like how the code is getting more and more consistent and clean :)
Kind regards,
Felix
Sent from ProtonMail mobile
-------- Original Message --------
On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> Hello everyone,
>
> This email is calling a vote on the changes in import paths. It's been
> discussed in
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
>
> The vote will last for at least 1 week (July 30th 6pm CEST), and at least
> three +1 (binding) votes have been cast.
>
> The proposal to vote is based on the document from Kamil
> <https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#>
>
> The proposed solution is:
>
> - *Case 1: B: Contrib vs not: we move all that are "well" tested and
> rename contrib to "incubating" or similar.*
> - *Case 2: B: Airflow.operators.foo_operator.FooOperator could
> become airflow.operators.foo.FooOperator*
> - *Case 3: C:
> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator could
> become airflow.gcp.operators.bigtable.BigTableOperator*
> - *Case 4: B: no namespace introduction*
> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class names*
> - *Case 6: We will treat isolated cases on case-by-case (and my team can
> do the job of GCP-related operators)*
>
> This is my (binding) +1 vote.
>
> Best regards,
>
> J.
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: [+48 660 796 129](tel:+48660796129) <[+48660796129](tel:+48660796129)>
> [image: Polidea] <https://www.polidea.com/>