MySQL mysql_postoperator parameter - To be deprecated in 2.0.0 ?

[airflowuser <ai...@protonmail.com.INVALID>]

Question to MySQL users...
I saw that HiveToMySQL have parameter called mysql_postoperator.
This parameter runs 2nd query against the MySQL.
This behavior doesn't exist in any of the other MySQL operators.
While I'm all for having more features I'm not sure it's needed. The user can simply run MySQLOperator with the 2nd query(?)
If it is needed then maybe we should mimic this to all of the MySQL operators Or add deprecation warning for this parameter that it will be removed in 2.0?
https://github.com/apache/airflow/blob/master/airflow/operators/hive_to_mysql.py#L49

Re: MySQL mysql_postoperator parameter - To be deprecated in 2.0.0 ?

[Maxime Beauchemin <ma...@gmail.com>]

That allows for the logic to run atomically with the task. I don't think
it's super important, but it can be nice to allow for this type of thing.

I'm not sure if this applies for mysql specifically, but if you want to
bulk load, you may have to drop indexes and/or constraints and recreate
them after the [bulk] load. This pre/post operators would insure that this
happens atomically with the task. Similarly you could use this to exchange
partitions and whatnot.

There's very little downside in keeping this, and isn't worth breaking
backwards compatibility IMHO

Max

On Thu, May 30, 2019 at 12:31 AM airflowuser
<ai...@protonmail.com.invalid> wrote:

> Question to MySQL users...
> I saw that HiveToMySQL have parameter called mysql_postoperator.
> This parameter runs 2nd query against the MySQL.
> This behavior doesn't exist in any of the other MySQL operators.
> While I'm all for having more features I'm not sure it's needed. The user
> can simply run MySQLOperator with the 2nd query(?)
> If it is needed then maybe we should mimic this to all of the MySQL
> operators Or add deprecation warning for this parameter that it will be
> removed in 2.0?
>
> https://github.com/apache/airflow/blob/master/airflow/operators/hive_to_mysql.py#L49