You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/05 02:29:26 UTC

[GitHub] [airflow] odajun opened a new issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

odajun opened a new issue #15198:
URL: https://github.com/apache/airflow/issues/15198


   **Apache Airflow version**: 1.10.14
   
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.19
   
   **Environment**:
   
   - **Cloud provider or hardware configuration**:  Openstack, 4 cores, 8GB RAM
   - **OS** (e.g. from /etc/os-release): /etc/centos-release CnetOS Linux release 7.9.2009
   - **Kernel** (e.g. `uname -a`): Linux
   - **Install tools**: Docker container (own build), Airflow is installed via 
   ```
   pip install apache-airflow[celery,hive,jdbc,mysql,redis,s3]==1.10.14
   pip install hdfs,requests_kerberos,prometheus-client,apache-airflow-upgrade-check,sqlalchemy=1.3.23,apache-airflow-backport-providers-apache-hive,apache-airflow-backport-providers-http
   ```
   
   **What happened**:
   
   While preparing for the upgrade by [following the steps in the documentation](https://airflow.apache.org/docs/apache-airflow/stable/upgrading-to-2.html), I was faced with an error at the point where I implemented the following changes.
   
   ```
   before : from airflow.operators.http_operator import SimpleHttpOperator
   after  : from airflow.providers.http.operators.http import SimpleHttpOperator
   ```
   
   error message
   ```
   ERROR - cannot import name 'make_kwargs_callable' from 'airflow.utils.operators_helpers'
   ```
   
   
   **Anything else we need to know**:
   
   I suspect the following commit is required for 1.10.x version as well.
   https://github.com/apache/airflow/commit/badd890675d3cb3dfc088bff6a1d73dfdc275f31#diff-d2d9b3696f554c1d2aee440470[…]27dc193705e85b833b1f739271e81b14
   
   There are other people who are encountering the same error.
   https://apache-airflow.slack.com/archives/CCQB40SQJ/p1616504379010200


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-814469792


   Hey @odajun @kurtqq @alexInhert : I just released https://pypi.org/project/apache-airflow-backport-providers-http/2021.4.10rc2/ that should fix the problem. 
   
   Can you please run `pip install apache-airflow-backport-providers-http==2021.4.10rc2` and let me know if the problem is fixed? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813778747


   > Can you at least explore this before dismissing it completely?
   
   There have been drastic differences between the Airflow 2.0 and Airflow 1.10 releases, and cherry-picking changes are very problematic and time-consuming e.g. Airflow 1.10 still supports Python 2.7, so we need to add backward compatibility during cherry-picking. Besides, we haven't made any changes to operators for over a year now, because we started releasing backport providers, to make migrations to Airflow 2.0 a bit easier, but also so that project maintainers don't have to cherry-pick any operators related code. If we wanted to release a fix for this bug, we would therefore have to release it as backport providers. 
   
   The code from the backport providers has already been removed so we would have to start creating a separate branch that would only contain the one fix.  Based on this one branch, we had to prepare a new release of backport package, start voting, and then release this package.  It probably took a few man-hours and took a week or more if we wanted to do it according to all the procedures we have in our community. This is an enormous amount of work that is at odds with what was previously agreed upon by the community. None of the project maintainers will be interested in doing this. 
   
   If you need to fix this error, you can prepare the package yourself and publish it in your organization's private python repository. The branch to start from is [legacy-backport-cutoff-point](https://github.com/apache/airflow/releases/tag/legacy-backport-cutoff-point).  The documentation and tools to build the backports are there.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] odajun commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
odajun commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-814158109


   @potiuk Thank you for your reply.
   
   > I think the biggest overhead there is not doing the changes but testing it. And here is my ask @odajun @kurtqq @alexInhert - if I release it later today or tomorrow as RC - would you volunteer to test it and report it back please ? That could help a lot with our decision of releasing it.
   
   I will probably test in my environment and report.
   
   It would be helpful if you could provide it in a form that allows installation using pip.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-817139620


   And as @kaxil mentioned - if this was a blocker to migrate to 2.0, you are unblocked now! And please migrate ASAP. You won't regret!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813779593


   If you are using Docker, you don't even need to prepare the pip package, but you can overwrite the required files in your image.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk closed issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #15198:
URL: https://github.com/apache/airflow/issues/15198


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813909054


   While we don't officially release the backports and as @mik-laj mentioned it is quite an overhead to release one. 
   
   However we have to take into account that the Http provider is one that is very commonly used, so we might try to think about doing a one time release to fix it as it might be a recurring problem for other users. 
   
   This is an exceptional case - it is not a but to fix but our own backwards incompatibility introduced in November. The reason for the problem was that - unfortunately the `make_kwargs_callable` was implemented  via local imports, thus bypassing the protection we added to catch this kind of problem in #11922 and it slipped under the radar of reviews. I can imagine few ways how it can be "quickly" implemented in an out-of-bands release for just http backport provider - without a lot of overhead. Happy to do it actually.
   
   There are two things to do - one tactical for now for you and more strategic what we do for the provider,
   
   1) For the tactical part - in the meantime @kurtqq and @odajun and @alexInhert there are two mitigations:
   
   a) you can simply stay with the old import when testing migration. It **should** work as long as you do not make use of the `make_kwargs_callable` so far. 
   b) You can install `apache-airflow-backports-provider-http==2020.10.5` - it has no offending code, and it should work fine. There were mostly refactorings/standardizations/documentation changes since then in HTTP provider. So it should **just work**. 
   
   2) I think It will be rather easy to move the "make_kwargs_callable" to inside the http backport provider (this is only problem with http provider it seems) and release an out-of-band backport provider, creating a branch for it branching off the `legacy-backport-cutoff-point`. I think the biggest overhead there is not doing the changes but testing it. And here is my ask @odajun @kurtqq  @alexInhert  - if I release it later today or tomorrow as RC - would you volunteer to test it and report it back please ? That could help a lot with our decision of releasing it.
   
   Another option for strategic solution is simply to yank the releases after 2020.10.5  and leave only that version. Not nicest, but I think it should work and have no side effects.
   
   @mik-laj @kaxil @ashb  WDYT? 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kurtqq commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
kurtqq commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813657471


   to jump in... i don't think it's issue of major or minor changes. It's issue of changes existed.
   Upgrading a major release in production system used by so many different teams is challenge enough.
   Please don't make things harder if you don't have to.
   
   As suggested you can cherry pick the updated `airflow/utils/operator_helpers.py ` https://github.com/apache/airflow/pull/11922 to the next 1.10 release
   
   Can you at least explore this before dismissing it completely? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813909054


   While we don't officially release the backports and as @mik-laj mentioned it is quite an overhead to release one. 
   
   However we have to take into account that the Http provider is one that is very commonly used, so we might try to think about doing a one time release to fix it as it might be a recurring problem for other users. 
   
   This is an exceptional case - it is not a bug to fix but our own backwards incompatibility introduced in November. The reason for the problem was that - unfortunately - the `make_kwargs_callable` was implemented  via local imports, thus bypassing the protection we added to catch this kind of problem (it was indeed added in #11922 and it slipped under the radar of reviews). I can imagine few ways how it can be "quickly" implemented in an out-of-bands release for just http backport provider - without a lot of overhead. Happy to do it actually.
   
   There are two things to do - one tactical for now for you and more strategic what we do for the provider,
   
   1) For the tactical part - in the meantime @kurtqq and @odajun and @alexInhert there are two mitigations:
   
   a) you can simply stay with the old import when testing migration. It **should** work with deprecation warning as long as you do not make use of the `make_kwargs_callable` so far and you can change the imports after the migration even one-by-one.
   
   b) You can install `apache-airflow-backports-provider-http==2020.10.5` - it has no offending code, and it should work fine. There were mostly refactorings/standardizations/documentation changes since then in HTTP provider. So it should **just work**. 
   
   2) I think It will be rather easy to move the "make_kwargs_callable" to inside the http backport provider (this is only problem with http provider it seems) and release an out-of-band backport provider, creating a branch for it branching off the `legacy-backport-cutoff-point`. I think the biggest overhead there is not doing the changes but testing it. And here is my ask @odajun @kurtqq  @alexInhert  - if I release it later today or tomorrow as RC - would you volunteer to test it and report it back please ? That could help a lot with our decision of releasing it.
   
   Another option for strategic solution is simply to yank the releases after 2020.10.5  and leave only that version. Not nicest, but I think it should work and have no side effects.
   
   @mik-laj @kaxil @ashb  WDYT? 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813648759


   Sorry, but we have already given up releasing new updates for backport providers.
   https://lists.apache.org/thread.html/rb34212bd13c44aa61e793d0fef14f4ab9d9408939b1b98dc814cf274%40%3Cdev.airflow.apache.org%3E
   
   This operator didn't have major breaking changes though, so you can keep the import to the airflow.operators package on Airflow 2.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813909054


   While we don't officially release the backports and as @mik-laj mentioned it is quite an overhead to release one. 
   
   However we have to take into account that the Http provider is one that is very commonly used, so we might try to think about doing a one time release to fix it as it might be a recurring problem for other users. 
   
   This is an exceptional case - it is not a bug to fix but our own backwards incompatibility introduced in November. The reason for the problem was that - unfortunately - the `make_kwargs_callable` was implemented  via local imports, thus bypassing the protection we added to catch this kind of problem in #11922 and it slipped under the radar of reviews. I can imagine few ways how it can be "quickly" implemented in an out-of-bands release for just http backport provider - without a lot of overhead. Happy to do it actually.
   
   There are two things to do - one tactical for now for you and more strategic what we do for the provider,
   
   1) For the tactical part - in the meantime @kurtqq and @odajun and @alexInhert there are two mitigations:
   
   a) you can simply stay with the old import when testing migration. It **should** work as long as you do not make use of the `make_kwargs_callable` so far. 
   b) You can install `apache-airflow-backports-provider-http==2020.10.5` - it has no offending code, and it should work fine. There were mostly refactorings/standardizations/documentation changes since then in HTTP provider. So it should **just work**. 
   
   2) I think It will be rather easy to move the "make_kwargs_callable" to inside the http backport provider (this is only problem with http provider it seems) and release an out-of-band backport provider, creating a branch for it branching off the `legacy-backport-cutoff-point`. I think the biggest overhead there is not doing the changes but testing it. And here is my ask @odajun @kurtqq  @alexInhert  - if I release it later today or tomorrow as RC - would you volunteer to test it and report it back please ? That could help a lot with our decision of releasing it.
   
   Another option for strategic solution is simply to yank the releases after 2020.10.5  and leave only that version. Not nicest, but I think it should work and have no side effects.
   
   @mik-laj @kaxil @ashb  WDYT? 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] boring-cyborg[bot] commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813148075


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813909054


   While we don't officially release the backports and as @mik-laj mentioned it is quite an overhead to release one. 
   
   However we have to take into account that the Http provider is one that is very commonly used, so we might try to think about doing a one time release to fix it as it might be a recurring problem for other users. 
   
   This is an exceptional case - it is not a bug to fix but our own backwards incompatibility introduced in November. The reason for the problem was that - unfortunately - the `make_kwargs_callable` was implemented  via local imports, thus bypassing the protection we added to catch this kind of problem (it was indeed added in #11922 and it slipped under the radar of reviews). I can imagine few ways how it can be "quickly" implemented in an out-of-bands release for just http backport provider - without a lot of overhead. Happy to do it actually.
   
   There are two things to do - one tactical for now for you and more strategic what we do for the provider,
   
   1) For the tactical part - in the meantime @kurtqq and @odajun and @alexInhert there are two mitigations:
   
   a) you can simply stay with the old import when testing migration. It **should** work as long as you do not make use of the `make_kwargs_callable` so far. 
   b) You can install `apache-airflow-backports-provider-http==2020.10.5` - it has no offending code, and it should work fine. There were mostly refactorings/standardizations/documentation changes since then in HTTP provider. So it should **just work**. 
   
   2) I think It will be rather easy to move the "make_kwargs_callable" to inside the http backport provider (this is only problem with http provider it seems) and release an out-of-band backport provider, creating a branch for it branching off the `legacy-backport-cutoff-point`. I think the biggest overhead there is not doing the changes but testing it. And here is my ask @odajun @kurtqq  @alexInhert  - if I release it later today or tomorrow as RC - would you volunteer to test it and report it back please ? That could help a lot with our decision of releasing it.
   
   Another option for strategic solution is simply to yank the releases after 2020.10.5  and leave only that version. Not nicest, but I think it should work and have no side effects.
   
   @mik-laj @kaxil @ashb  WDYT? 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] alexInhert commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
alexInhert commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813259788


   This is an issue for me too. It's a blocker for migration to Airflow 2.
   The backport HTTP provider is not working because of this.
   
   `make_kwargs_callable` should be added to a release of Airflow 1.10 or the backport provider should not call this function.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj closed issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #15198:
URL: https://github.com/apache/airflow/issues/15198


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-817139044


   Released :)  https://pypi.org/project/apache-airflow-backport-providers-http/2021.4.10/
   
   Thanks again @odajun for both reporting and testing. We need more users like you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813909054


   While we don't officially release the backports and @mik-laj mentioned it is quite an overhead. However we have to take into account that the Http provider is one that is very commonly used, so we might try to think about doing a one time release to fix it as it might be a recurring problem for other users. 
   
   This is an exceptional case - it is not a but to fix but our own backwards incompatibility introduced in November. The reason for the problem was that - unfortunately the `make_kwargs_callable` was implemented  via local imports, thus bypassing the protection we added to catch this kind of problem in #11922 and it slipped under the radar of reviews. I can imagine few ways how it can be "quickly" implemented in an out-of-bands release for just http backport provider - without a lot of overhead. Happy to do it actually.
   
   There are two things to do - one tactical for now for you and more strategic what we do for the provider,
   
   1) For the tactical part - in the meantime @kurtqq and @odajun and @alexInhert there are two mitigations:
   
   a) you can simply stay with the old import when testing migration. It **should** work as long as you do not make use of the `make_kwargs_callable` so far. 
   b) You can install `apache-airflow-backports-provider-http==2020.10.5` - it has no offending code, and it should work fine. There were mostly refactorings/standardizations/documentation changes since then in HTTP provider. So it should **just work**. 
   
   2) I think It will be rather easy to move the "make_kwargs_callable" to inside the http backport provider (this is only problem with http provider it seems) and release an out-of-band backport provider, creating a branch for it branching off the `legacy-backport-cutoff-point`. I think the biggest overhead there is not doing the changes but testing it. And here is my ask @odajun @kurtqq  @alexInhert  - if I release it later today or tomorrow as RC - would you volunteer to test it and report it back please ? That could help a lot with our decision of releasing it.
   
   Another option for strategic solution is simply to yank the releases after 2020.10.5  and leave only that version. Not nicest, but I think it should work and have no side effects.
   
   @mik-laj @kaxil @ashb  WDYT? 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] odajun commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
odajun commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-815382368


   @potiuk Thank you for your support.
   
   After installing ```apache-airflow-backport-providers-http==2021.4.10rc2``` in my environment, SimpleHttpOperator worked fine and I did not have any problems.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813909054


   While we don't officially release the backports and as @mik-laj mentioned it is quite an overhead to release one. 
   
   However we have to take into account that the Http provider is one that is very commonly used, so we might try to think about doing a one time release to fix it as it might be a recurring problem for other users. 
   
   This is an exceptional case - it is not a bug to fix but our own backwards incompatibility introduced in November. The reason for the problem was that - unfortunately - the `make_kwargs_callable` was implemented  via local imports, thus bypassing the protection we added to catch this kind of problem (it was indeed added in #11922 and it slipped under the radar of reviews). I can imagine few ways how it can be "quickly" implemented in an out-of-bands release for just http backport provider - without a lot of overhead. Happy to do it actually.
   
   There are two things to do - one tactical for now for you and more strategic what we do for the provider,
   
   1) For the tactical part - in the meantime @kurtqq and @odajun and @alexInhert there are two mitigations:
   
   a) you can simply stay with the old import when testing migration. It **should** work with deprecation warning as long as you do not make use of the `make_kwargs_callable` so far and you can change the imports after the migration even one-by-one.
   
   b) You can install `apache-airflow-backports-provider-http==2020.10.5` - it has no offending code, and it should work fine. There were mostly refactorings/standardizations/documentation changes since then in HTTP provider. So it should **just work**. 
   
   2) For the strategi solution - I see two ways:
   
   a) I think It will be rather easy to move the "make_kwargs_callable" to inside the http backport provider (this is only problem with http provider it seems) and release an out-of-band backport provider, creating a branch for it branching off the `legacy-backport-cutoff-point`. I think the biggest overhead there is not doing the changes but testing it. And here is my ask @odajun @kurtqq  @alexInhert  - if I release it later today or tomorrow as RC - would you volunteer to test it and report it back please ? That could help a lot with our decision of releasing it.
   
   B) another option for strategic solution is simply to yank the releases after 2020.10.5  and leave only that version. Not nicest, but I think it should work and have no side effects.
   
   @mik-laj @kaxil @ashb  WDYT? 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-814169016


   Yea we can probably follow (a) move the "make_kwargs_callable" to inside the http backport provider. And release one-off HTTP backport provider since it is very popular. But even 
   
   But like Jarek said this will need help from you'll in testing. And also note that the Backport Providers are only to help users transition to 2.0 and is provided with "best effort" and like Kamil said there is a bit of overhead to it. So please help us test this one and try to migrate to 2.0 ASAP, I know it is difficult for organization to upgrade to a major version but this one comes with huge performance gain. Airflow 1.10.x will reach EOL in June too.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-814479102


   Please also add a non-binding vote if that works for you in https://lists.apache.org/thread.html/r5e870c593ca558c5e05417dc20b9dc55a48bbcb485483b63cd9b416c%40%3Cdev.airflow.apache.org%3E


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813909054


   While we don't officially release the backports and as @mik-laj mentioned it is quite an overhead to release one. 
   
   However we have to take into account that the Http provider is one that is very commonly used, so we might try to think about doing a one time release to fix it as it might be a recurring problem for other users. 
   
   This is an exceptional case - it is not a bug to fix but our own backwards incompatibility introduced in November. The reason for the problem was that - unfortunately the `make_kwargs_callable` was implemented  via local imports, thus bypassing the protection we added to catch this kind of problem in #11922 and it slipped under the radar of reviews. I can imagine few ways how it can be "quickly" implemented in an out-of-bands release for just http backport provider - without a lot of overhead. Happy to do it actually.
   
   There are two things to do - one tactical for now for you and more strategic what we do for the provider,
   
   1) For the tactical part - in the meantime @kurtqq and @odajun and @alexInhert there are two mitigations:
   
   a) you can simply stay with the old import when testing migration. It **should** work as long as you do not make use of the `make_kwargs_callable` so far. 
   b) You can install `apache-airflow-backports-provider-http==2020.10.5` - it has no offending code, and it should work fine. There were mostly refactorings/standardizations/documentation changes since then in HTTP provider. So it should **just work**. 
   
   2) I think It will be rather easy to move the "make_kwargs_callable" to inside the http backport provider (this is only problem with http provider it seems) and release an out-of-band backport provider, creating a branch for it branching off the `legacy-backport-cutoff-point`. I think the biggest overhead there is not doing the changes but testing it. And here is my ask @odajun @kurtqq  @alexInhert  - if I release it later today or tomorrow as RC - would you volunteer to test it and report it back please ? That could help a lot with our decision of releasing it.
   
   Another option for strategic solution is simply to yank the releases after 2020.10.5  and leave only that version. Not nicest, but I think it should work and have no side effects.
   
   @mik-laj @kaxil @ashb  WDYT? 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] odajun commented on issue #15198: "cannot import make_kwargs_callable" occurs when using airflow.providers.http.operators.http in 1.10.14.

Posted by GitBox <gi...@apache.org>.
odajun commented on issue #15198:
URL: https://github.com/apache/airflow/issues/15198#issuecomment-813767714


   @mik-laj Thank you for confirming this.
   
   As for SimpleHttpOperator, I will try to upgrade it to 2.0.x without changing the import source.
   
   ```
    from airflow.operators.http_operator import SimpleHttpOperator
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org