You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <Ja...@polidea.com> on 2020/05/29 13:00:01 UTC
[DISCUSS] Naming of the transfer operators/Hooks
Hello everyone
We had a discussion in Slack which turned out that we have yet another
opportunity to name the operators/hooks a bit more consistently. Seems that
we did not have any rule on how to name Transfer operators and we have
different conventions already.
Explanation those are the two examples of conventions we have for transfer
operators:
[1] *S3ToHiveOperator*
[2] *S3ToHiveTransferOperator*
[3] *S3ToHiveTransfer*
[4] We do not care about consistency
Some initial comments that I gathered from the discussions::
- Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant
- Why [2] and not [1,3]: Longest, but most descriptive. It's easy to see
that it's an Operator, but you get the Transfer purpose as well. sometimes
when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To" from the
acronyms.
- Why [1, 2] and not [3]: All Operators (but not Sensor) end with "Operator"
- Why [3]: and not [1,2]: To introduce distinction: "Sensor", "Operator",
so maybe "Transfer" should be another "entity" and in the future, we might
implement a more generic Transfer approach
I will let the discussion run till the end of today and cast a formal vote
afterwards
I do not yet cancel Backport RC3, because I am not sure if this is
something we might want to do - maybe after discussion we decide we leave
the status quo.
Discussion in slack here:
https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943
J.
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Naming of the transfer operators/Hooks
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
1 looks nice and clean
Op za 30 mei 2020 om 11:37 schreef Jarek Potiuk <Ja...@polidea.com>
> That's a completely new thing/idea.
>
> We might have separate vote for that. Kamil - can you please lead the vote
> and renaming if it happens?
>
> I am for the change. I think it would make sense to separate them.
>
> I wonder what others think.
>
> J.
>
>
> On Sat, May 30, 2020 at 11:11 AM Kamil Breguła <ka...@polidea.com>
> wrote:
>
> > I would like to move transfer operators to the new package - transfer
> > airflow.providers.*.transfer.*
> > airflow.providers.*.operator.*
> > airflow.providers.*.sensor.*
> >
> > Transfer operators are very often separated in the documentation and it
> > would be helpful if this division was included in the code.
> > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html
> >
> >
> >
> >
> >
> >
> >
> > On Fri, May 29, 2020, 21:27 Xinbin Huang <bi...@gmail.com> wrote:
> >
> > > I also vote [1]: S3ToHiveOperator
> > >
> > > Since there is no such concept on Airflow about transfer operators, we
> > > better not introduce extra complexity with a potential generic
> `Transfer`
> > > entity.
> > >
> > > Here are some personal thoughts:
> > > - there are only `import *.operators.*` and `*.sensors.*` in the
> > codebase
> > > - `Transfer` is just a subclass of `.operators`
> > > - the implementation of a transfer is rather similar to an operator -
> do
> > > something in the `.execute` as opposed to returning True/False in the
> > > `.poke`
> > >
> > > Another thing worth mentioning is that I saw that from different
> places
> > > there was a misconception that there are 3 types of operators:
> Operators,
> > > Sensors, and Transfers.
> > > So whatever the final decision is, we need to communicate this more
> > clearly
> > > in the doc
> > https://airflow.apache.org/docs/stable/concepts.html#operators
> > >
> > > On Fri, May 29, 2020 at 10:43 AM Daniel Standish <dpstandish@gmail.com
> >
> > > wrote:
> > >
> > > > I also vote [1]: S3ToHiveOperator
> > > > Transfer is redundant.
> > > > And why not 3 --- as of now there is no such distinction and I am not
> > > > convinced it is justified. As of now, this is just another operator.
> > > > Probably most operators do some kind of "transferring" of data and
> > trying
> > > > to decide what is transfer and what is not at this time would be
> rather
> > > > squishy.
> > > >
> > > > On Fri, May 29, 2020 at 9:11 AM Samuel Tu <sa...@gmail.com>
> wrote:
> > > >
> > > > > I am in favor of [2] as some operators can handle operations such
> as
> > > > > branching.
> > > > >
> > > > > Thanks
> > > > > Sam
> > > > >
> > > > > > On May 29, 2020, at 9:00 AM, Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>
> > > > > wrote:
> > > > > >
> > > > > > Hello everyone
> > > > > >
> > > > > > We had a discussion in Slack which turned out that we have yet
> > > another
> > > > > > opportunity to name the operators/hooks a bit more consistently.
> > > Seems
> > > > > that
> > > > > > we did not have any rule on how to name Transfer operators and we
> > > have
> > > > > > different conventions already.
> > > > > >
> > > > > > Explanation those are the two examples of conventions we have for
> > > > > transfer
> > > > > > operators:
> > > > > >
> > > > > > [1] *S3ToHiveOperator*
> > > > > > [2] *S3ToHiveTransferOperator*
> > > > > > [3] *S3ToHiveTransfer*
> > > > > > [4] We do not care about consistency
> > > > > >
> > > > > > Some initial comments that I gathered from the discussions::
> > > > > >
> > > > > > - Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant
> > > > > > - Why [2] and not [1,3]: Longest, but most descriptive. It's easy
> > to
> > > > see
> > > > > > that it's an Operator, but you get the Transfer purpose as well.
> > > > > sometimes
> > > > > > when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To"
> > from
> > > > the
> > > > > > acronyms.
> > > > > > - Why [1, 2] and not [3]: All Operators (but not Sensor) end with
> > > > > "Operator"
> > > > > > - Why [3]: and not [1,2]: To introduce distinction: "Sensor",
> > > > "Operator",
> > > > > > so maybe "Transfer" should be another "entity" and in the future,
> > we
> > > > > might
> > > > > > implement a more generic Transfer approach
> > > > > >
> > > > > > I will let the discussion run till the end of today and cast a
> > formal
> > > > > vote
> > > > > > afterwards
> > > > > >
> > > > > > I do not yet cancel Backport RC3, because I am not sure if this
> is
> > > > > > something we might want to do - maybe after discussion we decide
> we
> > > > leave
> > > > > > the status quo.
> > > > > >
> > > > > > Discussion in slack here:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943
> > > > > >
> > > > > >
> > > > > > J.
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Jarek Potiuk
> > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > > >
> > > > > > M: +48 660 796 129 <+48660796129>
> > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > >
> > > >
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
Re: [DISCUSS] Naming of the transfer operators/Hooks
Posted by Jarek Potiuk <Ja...@polidea.com>.
That's a completely new thing/idea.
We might have separate vote for that. Kamil - can you please lead the vote
and renaming if it happens?
I am for the change. I think it would make sense to separate them.
I wonder what others think.
J.
On Sat, May 30, 2020 at 11:11 AM Kamil Breguła <ka...@polidea.com>
wrote:
> I would like to move transfer operators to the new package - transfer
> airflow.providers.*.transfer.*
> airflow.providers.*.operator.*
> airflow.providers.*.sensor.*
>
> Transfer operators are very often separated in the documentation and it
> would be helpful if this division was included in the code.
> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html
>
>
>
>
>
>
>
> On Fri, May 29, 2020, 21:27 Xinbin Huang <bi...@gmail.com> wrote:
>
> > I also vote [1]: S3ToHiveOperator
> >
> > Since there is no such concept on Airflow about transfer operators, we
> > better not introduce extra complexity with a potential generic `Transfer`
> > entity.
> >
> > Here are some personal thoughts:
> > - there are only `import *.operators.*` and `*.sensors.*` in the
> codebase
> > - `Transfer` is just a subclass of `.operators`
> > - the implementation of a transfer is rather similar to an operator - do
> > something in the `.execute` as opposed to returning True/False in the
> > `.poke`
> >
> > Another thing worth mentioning is that I saw that from different places
> > there was a misconception that there are 3 types of operators: Operators,
> > Sensors, and Transfers.
> > So whatever the final decision is, we need to communicate this more
> clearly
> > in the doc
> https://airflow.apache.org/docs/stable/concepts.html#operators
> >
> > On Fri, May 29, 2020 at 10:43 AM Daniel Standish <dp...@gmail.com>
> > wrote:
> >
> > > I also vote [1]: S3ToHiveOperator
> > > Transfer is redundant.
> > > And why not 3 --- as of now there is no such distinction and I am not
> > > convinced it is justified. As of now, this is just another operator.
> > > Probably most operators do some kind of "transferring" of data and
> trying
> > > to decide what is transfer and what is not at this time would be rather
> > > squishy.
> > >
> > > On Fri, May 29, 2020 at 9:11 AM Samuel Tu <sa...@gmail.com> wrote:
> > >
> > > > I am in favor of [2] as some operators can handle operations such as
> > > > branching.
> > > >
> > > > Thanks
> > > > Sam
> > > >
> > > > > On May 29, 2020, at 9:00 AM, Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> > > > wrote:
> > > > >
> > > > > Hello everyone
> > > > >
> > > > > We had a discussion in Slack which turned out that we have yet
> > another
> > > > > opportunity to name the operators/hooks a bit more consistently.
> > Seems
> > > > that
> > > > > we did not have any rule on how to name Transfer operators and we
> > have
> > > > > different conventions already.
> > > > >
> > > > > Explanation those are the two examples of conventions we have for
> > > > transfer
> > > > > operators:
> > > > >
> > > > > [1] *S3ToHiveOperator*
> > > > > [2] *S3ToHiveTransferOperator*
> > > > > [3] *S3ToHiveTransfer*
> > > > > [4] We do not care about consistency
> > > > >
> > > > > Some initial comments that I gathered from the discussions::
> > > > >
> > > > > - Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant
> > > > > - Why [2] and not [1,3]: Longest, but most descriptive. It's easy
> to
> > > see
> > > > > that it's an Operator, but you get the Transfer purpose as well.
> > > > sometimes
> > > > > when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To"
> from
> > > the
> > > > > acronyms.
> > > > > - Why [1, 2] and not [3]: All Operators (but not Sensor) end with
> > > > "Operator"
> > > > > - Why [3]: and not [1,2]: To introduce distinction: "Sensor",
> > > "Operator",
> > > > > so maybe "Transfer" should be another "entity" and in the future,
> we
> > > > might
> > > > > implement a more generic Transfer approach
> > > > >
> > > > > I will let the discussion run till the end of today and cast a
> formal
> > > > vote
> > > > > afterwards
> > > > >
> > > > > I do not yet cancel Backport RC3, because I am not sure if this is
> > > > > something we might want to do - maybe after discussion we decide we
> > > leave
> > > > > the status quo.
> > > > >
> > > > > Discussion in slack here:
> > > > >
> > > >
> > >
> >
> https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943
> > > > >
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129 <+48660796129>
> > > > > [image: Polidea] <https://www.polidea.com/>
> > > >
> > >
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Naming of the transfer operators/Hooks
Posted by Kamil Breguła <ka...@polidea.com>.
I would like to move transfer operators to the new package - transfer
airflow.providers.*.transfer.*
airflow.providers.*.operator.*
airflow.providers.*.sensor.*
Transfer operators are very often separated in the documentation and it
would be helpful if this division was included in the code.
https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html
On Fri, May 29, 2020, 21:27 Xinbin Huang <bi...@gmail.com> wrote:
> I also vote [1]: S3ToHiveOperator
>
> Since there is no such concept on Airflow about transfer operators, we
> better not introduce extra complexity with a potential generic `Transfer`
> entity.
>
> Here are some personal thoughts:
> - there are only `import *.operators.*` and `*.sensors.*` in the codebase
> - `Transfer` is just a subclass of `.operators`
> - the implementation of a transfer is rather similar to an operator - do
> something in the `.execute` as opposed to returning True/False in the
> `.poke`
>
> Another thing worth mentioning is that I saw that from different places
> there was a misconception that there are 3 types of operators: Operators,
> Sensors, and Transfers.
> So whatever the final decision is, we need to communicate this more clearly
> in the doc https://airflow.apache.org/docs/stable/concepts.html#operators
>
> On Fri, May 29, 2020 at 10:43 AM Daniel Standish <dp...@gmail.com>
> wrote:
>
> > I also vote [1]: S3ToHiveOperator
> > Transfer is redundant.
> > And why not 3 --- as of now there is no such distinction and I am not
> > convinced it is justified. As of now, this is just another operator.
> > Probably most operators do some kind of "transferring" of data and trying
> > to decide what is transfer and what is not at this time would be rather
> > squishy.
> >
> > On Fri, May 29, 2020 at 9:11 AM Samuel Tu <sa...@gmail.com> wrote:
> >
> > > I am in favor of [2] as some operators can handle operations such as
> > > branching.
> > >
> > > Thanks
> > > Sam
> > >
> > > > On May 29, 2020, at 9:00 AM, Jarek Potiuk <Ja...@polidea.com>
> > > wrote:
> > > >
> > > > Hello everyone
> > > >
> > > > We had a discussion in Slack which turned out that we have yet
> another
> > > > opportunity to name the operators/hooks a bit more consistently.
> Seems
> > > that
> > > > we did not have any rule on how to name Transfer operators and we
> have
> > > > different conventions already.
> > > >
> > > > Explanation those are the two examples of conventions we have for
> > > transfer
> > > > operators:
> > > >
> > > > [1] *S3ToHiveOperator*
> > > > [2] *S3ToHiveTransferOperator*
> > > > [3] *S3ToHiveTransfer*
> > > > [4] We do not care about consistency
> > > >
> > > > Some initial comments that I gathered from the discussions::
> > > >
> > > > - Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant
> > > > - Why [2] and not [1,3]: Longest, but most descriptive. It's easy to
> > see
> > > > that it's an Operator, but you get the Transfer purpose as well.
> > > sometimes
> > > > when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To" from
> > the
> > > > acronyms.
> > > > - Why [1, 2] and not [3]: All Operators (but not Sensor) end with
> > > "Operator"
> > > > - Why [3]: and not [1,2]: To introduce distinction: "Sensor",
> > "Operator",
> > > > so maybe "Transfer" should be another "entity" and in the future, we
> > > might
> > > > implement a more generic Transfer approach
> > > >
> > > > I will let the discussion run till the end of today and cast a formal
> > > vote
> > > > afterwards
> > > >
> > > > I do not yet cancel Backport RC3, because I am not sure if this is
> > > > something we might want to do - maybe after discussion we decide we
> > leave
> > > > the status quo.
> > > >
> > > > Discussion in slack here:
> > > >
> > >
> >
> https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943
> > > >
> > > >
> > > > J.
> > > >
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > >
> >
>
Re: [DISCUSS] Naming of the transfer operators/Hooks
Posted by Bas Harenslak <ba...@godatadriven.com.INVALID>.
#1 for me, to me the “To” in the name already implies a transfer between two systems.
Bas
> On 29 May 2020, at 21:27, Xinbin Huang <bi...@gmail.com> wrote:
>
> I also vote [1]: S3ToHiveOperator
>
> Since there is no such concept on Airflow about transfer operators, we
> better not introduce extra complexity with a potential generic `Transfer`
> entity.
>
> Here are some personal thoughts:
> - there are only `import *.operators.*` and `*.sensors.*` in the codebase
> - `Transfer` is just a subclass of `.operators`
> - the implementation of a transfer is rather similar to an operator - do
> something in the `.execute` as opposed to returning True/False in the
> `.poke`
>
> Another thing worth mentioning is that I saw that from different places
> there was a misconception that there are 3 types of operators: Operators,
> Sensors, and Transfers.
> So whatever the final decision is, we need to communicate this more clearly
> in the doc https://airflow.apache.org/docs/stable/concepts.html#operators
>
> On Fri, May 29, 2020 at 10:43 AM Daniel Standish <dp...@gmail.com>
> wrote:
>
>> I also vote [1]: S3ToHiveOperator
>> Transfer is redundant.
>> And why not 3 --- as of now there is no such distinction and I am not
>> convinced it is justified. As of now, this is just another operator.
>> Probably most operators do some kind of "transferring" of data and trying
>> to decide what is transfer and what is not at this time would be rather
>> squishy.
>>
>> On Fri, May 29, 2020 at 9:11 AM Samuel Tu <sa...@gmail.com> wrote:
>>
>>> I am in favor of [2] as some operators can handle operations such as
>>> branching.
>>>
>>> Thanks
>>> Sam
>>>
>>>> On May 29, 2020, at 9:00 AM, Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>>
>>>> Hello everyone
>>>>
>>>> We had a discussion in Slack which turned out that we have yet another
>>>> opportunity to name the operators/hooks a bit more consistently. Seems
>>> that
>>>> we did not have any rule on how to name Transfer operators and we have
>>>> different conventions already.
>>>>
>>>> Explanation those are the two examples of conventions we have for
>>> transfer
>>>> operators:
>>>>
>>>> [1] *S3ToHiveOperator*
>>>> [2] *S3ToHiveTransferOperator*
>>>> [3] *S3ToHiveTransfer*
>>>> [4] We do not care about consistency
>>>>
>>>> Some initial comments that I gathered from the discussions::
>>>>
>>>> - Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant
>>>> - Why [2] and not [1,3]: Longest, but most descriptive. It's easy to
>> see
>>>> that it's an Operator, but you get the Transfer purpose as well.
>>> sometimes
>>>> when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To" from
>> the
>>>> acronyms.
>>>> - Why [1, 2] and not [3]: All Operators (but not Sensor) end with
>>> "Operator"
>>>> - Why [3]: and not [1,2]: To introduce distinction: "Sensor",
>> "Operator",
>>>> so maybe "Transfer" should be another "entity" and in the future, we
>>> might
>>>> implement a more generic Transfer approach
>>>>
>>>> I will let the discussion run till the end of today and cast a formal
>>> vote
>>>> afterwards
>>>>
>>>> I do not yet cancel Backport RC3, because I am not sure if this is
>>>> something we might want to do - maybe after discussion we decide we
>> leave
>>>> the status quo.
>>>>
>>>> Discussion in slack here:
>>>>
>>>
>> https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943
>>>>
>>>>
>>>> J.
>>>>
>>>>
>>>> --
>>>>
>>>> Jarek Potiuk
>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>
>>>> M: +48 660 796 129 <+48660796129>
>>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>
Re: [DISCUSS] Naming of the transfer operators/Hooks
Posted by Xinbin Huang <bi...@gmail.com>.
I also vote [1]: S3ToHiveOperator
Since there is no such concept on Airflow about transfer operators, we
better not introduce extra complexity with a potential generic `Transfer`
entity.
Here are some personal thoughts:
- there are only `import *.operators.*` and `*.sensors.*` in the codebase
- `Transfer` is just a subclass of `.operators`
- the implementation of a transfer is rather similar to an operator - do
something in the `.execute` as opposed to returning True/False in the
`.poke`
Another thing worth mentioning is that I saw that from different places
there was a misconception that there are 3 types of operators: Operators,
Sensors, and Transfers.
So whatever the final decision is, we need to communicate this more clearly
in the doc https://airflow.apache.org/docs/stable/concepts.html#operators
On Fri, May 29, 2020 at 10:43 AM Daniel Standish <dp...@gmail.com>
wrote:
> I also vote [1]: S3ToHiveOperator
> Transfer is redundant.
> And why not 3 --- as of now there is no such distinction and I am not
> convinced it is justified. As of now, this is just another operator.
> Probably most operators do some kind of "transferring" of data and trying
> to decide what is transfer and what is not at this time would be rather
> squishy.
>
> On Fri, May 29, 2020 at 9:11 AM Samuel Tu <sa...@gmail.com> wrote:
>
> > I am in favor of [2] as some operators can handle operations such as
> > branching.
> >
> > Thanks
> > Sam
> >
> > > On May 29, 2020, at 9:00 AM, Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> > >
> > > Hello everyone
> > >
> > > We had a discussion in Slack which turned out that we have yet another
> > > opportunity to name the operators/hooks a bit more consistently. Seems
> > that
> > > we did not have any rule on how to name Transfer operators and we have
> > > different conventions already.
> > >
> > > Explanation those are the two examples of conventions we have for
> > transfer
> > > operators:
> > >
> > > [1] *S3ToHiveOperator*
> > > [2] *S3ToHiveTransferOperator*
> > > [3] *S3ToHiveTransfer*
> > > [4] We do not care about consistency
> > >
> > > Some initial comments that I gathered from the discussions::
> > >
> > > - Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant
> > > - Why [2] and not [1,3]: Longest, but most descriptive. It's easy to
> see
> > > that it's an Operator, but you get the Transfer purpose as well.
> > sometimes
> > > when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To" from
> the
> > > acronyms.
> > > - Why [1, 2] and not [3]: All Operators (but not Sensor) end with
> > "Operator"
> > > - Why [3]: and not [1,2]: To introduce distinction: "Sensor",
> "Operator",
> > > so maybe "Transfer" should be another "entity" and in the future, we
> > might
> > > implement a more generic Transfer approach
> > >
> > > I will let the discussion run till the end of today and cast a formal
> > vote
> > > afterwards
> > >
> > > I do not yet cancel Backport RC3, because I am not sure if this is
> > > something we might want to do - maybe after discussion we decide we
> leave
> > > the status quo.
> > >
> > > Discussion in slack here:
> > >
> >
> https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943
> > >
> > >
> > > J.
> > >
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> >
>
Re: [DISCUSS] Naming of the transfer operators/Hooks
Posted by Daniel Standish <dp...@gmail.com>.
I also vote [1]: S3ToHiveOperator
Transfer is redundant.
And why not 3 --- as of now there is no such distinction and I am not
convinced it is justified. As of now, this is just another operator.
Probably most operators do some kind of "transferring" of data and trying
to decide what is transfer and what is not at this time would be rather
squishy.
On Fri, May 29, 2020 at 9:11 AM Samuel Tu <sa...@gmail.com> wrote:
> I am in favor of [2] as some operators can handle operations such as
> branching.
>
> Thanks
> Sam
>
> > On May 29, 2020, at 9:00 AM, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> >
> > Hello everyone
> >
> > We had a discussion in Slack which turned out that we have yet another
> > opportunity to name the operators/hooks a bit more consistently. Seems
> that
> > we did not have any rule on how to name Transfer operators and we have
> > different conventions already.
> >
> > Explanation those are the two examples of conventions we have for
> transfer
> > operators:
> >
> > [1] *S3ToHiveOperator*
> > [2] *S3ToHiveTransferOperator*
> > [3] *S3ToHiveTransfer*
> > [4] We do not care about consistency
> >
> > Some initial comments that I gathered from the discussions::
> >
> > - Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant
> > - Why [2] and not [1,3]: Longest, but most descriptive. It's easy to see
> > that it's an Operator, but you get the Transfer purpose as well.
> sometimes
> > when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To" from the
> > acronyms.
> > - Why [1, 2] and not [3]: All Operators (but not Sensor) end with
> "Operator"
> > - Why [3]: and not [1,2]: To introduce distinction: "Sensor", "Operator",
> > so maybe "Transfer" should be another "entity" and in the future, we
> might
> > implement a more generic Transfer approach
> >
> > I will let the discussion run till the end of today and cast a formal
> vote
> > afterwards
> >
> > I do not yet cancel Backport RC3, because I am not sure if this is
> > something we might want to do - maybe after discussion we decide we leave
> > the status quo.
> >
> > Discussion in slack here:
> >
> https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943
> >
> >
> > J.
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
>
Re: [DISCUSS] Naming of the transfer operators/Hooks
Posted by Samuel Tu <sa...@gmail.com>.
I am in favor of [2] as some operators can handle operations such as branching.
Thanks
Sam
> On May 29, 2020, at 9:00 AM, Jarek Potiuk <Ja...@polidea.com> wrote:
>
> Hello everyone
>
> We had a discussion in Slack which turned out that we have yet another
> opportunity to name the operators/hooks a bit more consistently. Seems that
> we did not have any rule on how to name Transfer operators and we have
> different conventions already.
>
> Explanation those are the two examples of conventions we have for transfer
> operators:
>
> [1] *S3ToHiveOperator*
> [2] *S3ToHiveTransferOperator*
> [3] *S3ToHiveTransfer*
> [4] We do not care about consistency
>
> Some initial comments that I gathered from the discussions::
>
> - Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant
> - Why [2] and not [1,3]: Longest, but most descriptive. It's easy to see
> that it's an Operator, but you get the Transfer purpose as well. sometimes
> when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To" from the
> acronyms.
> - Why [1, 2] and not [3]: All Operators (but not Sensor) end with "Operator"
> - Why [3]: and not [1,2]: To introduce distinction: "Sensor", "Operator",
> so maybe "Transfer" should be another "entity" and in the future, we might
> implement a more generic Transfer approach
>
> I will let the discussion run till the end of today and cast a formal vote
> afterwards
>
> I do not yet cancel Backport RC3, because I am not sure if this is
> something we might want to do - maybe after discussion we decide we leave
> the status quo.
>
> Discussion in slack here:
> https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943
>
>
> J.
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Naming of the transfer operators/Hooks
Posted by Samuel Tu <sa...@gmail.com>.
I am in favor of [2] as sometime an operator can be used to do other operation such as branching.
Thanks
Sam
> On May 29, 2020, at 9:39 AM, Kaxil Naik <ka...@gmail.com> wrote:
>
> I am in favour of [1] : Short and sweet (just personal preference)
>
>> On Fri, May 29, 2020 at 2:00 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> Hello everyone
>>
>> We had a discussion in Slack which turned out that we have yet another
>> opportunity to name the operators/hooks a bit more consistently. Seems that
>> we did not have any rule on how to name Transfer operators and we have
>> different conventions already.
>>
>> Explanation those are the two examples of conventions we have for transfer
>> operators:
>>
>> [1] *S3ToHiveOperator*
>> [2] *S3ToHiveTransferOperator*
>> [3] *S3ToHiveTransfer*
>> [4] We do not care about consistency
>>
>> Some initial comments that I gathered from the discussions::
>>
>> - Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant
>> - Why [2] and not [1,3]: Longest, but most descriptive. It's easy to see
>> that it's an Operator, but you get the Transfer purpose as well. sometimes
>> when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To" from the
>> acronyms.
>> - Why [1, 2] and not [3]: All Operators (but not Sensor) end with
>> "Operator"
>> - Why [3]: and not [1,2]: To introduce distinction: "Sensor", "Operator",
>> so maybe "Transfer" should be another "entity" and in the future, we might
>> implement a more generic Transfer approach
>>
>> I will let the discussion run till the end of today and cast a formal vote
>> afterwards
>>
>> I do not yet cancel Backport RC3, because I am not sure if this is
>> something we might want to do - maybe after discussion we decide we leave
>> the status quo.
>>
>> Discussion in slack here:
>>
>> https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943
>>
>>
>> J.
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
Re: [DISCUSS] Naming of the transfer operators/Hooks
Posted by Kaxil Naik <ka...@gmail.com>.
I am in favour of [1] : Short and sweet (just personal preference)
On Fri, May 29, 2020 at 2:00 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Hello everyone
>
> We had a discussion in Slack which turned out that we have yet another
> opportunity to name the operators/hooks a bit more consistently. Seems that
> we did not have any rule on how to name Transfer operators and we have
> different conventions already.
>
> Explanation those are the two examples of conventions we have for transfer
> operators:
>
> [1] *S3ToHiveOperator*
> [2] *S3ToHiveTransferOperator*
> [3] *S3ToHiveTransfer*
> [4] We do not care about consistency
>
> Some initial comments that I gathered from the discussions::
>
> - Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant
> - Why [2] and not [1,3]: Longest, but most descriptive. It's easy to see
> that it's an Operator, but you get the Transfer purpose as well. sometimes
> when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To" from the
> acronyms.
> - Why [1, 2] and not [3]: All Operators (but not Sensor) end with
> "Operator"
> - Why [3]: and not [1,2]: To introduce distinction: "Sensor", "Operator",
> so maybe "Transfer" should be another "entity" and in the future, we might
> implement a more generic Transfer approach
>
> I will let the discussion run till the end of today and cast a formal vote
> afterwards
>
> I do not yet cancel Backport RC3, because I am not sure if this is
> something we might want to do - maybe after discussion we decide we leave
> the status quo.
>
> Discussion in slack here:
>
> https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943
>
>
> J.
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>