You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/07 01:05:15 UTC
[GitHub] [airflow] potiuk opened a new issue #15241: Testing Providers prepared 2021.04.07
potiuk opened a new issue #15241:
URL: https://github.com/apache/airflow/issues/15241
Since you were providing the fixes,I kindly ask for help :) .
Could you please help testing the providers? Please comment below if you tested `your` part:
New providers:
- [ ] airbyte: @marcosmarxm https://pypi.org/project/apache-airflow-providers-airbyte/1.0.0rc1/
- [ ] trino: @potiuk https://pypi.org/project/apache-airflow-providers-trino/1.0.0rc1/
Providers with fixes:
- [ ] apache.hive: @XD-DENG https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.3rc1/
- [ ] hashicorp: @SangwanP https://pypi.org/project/apache-airflow-providers-hashicorp/1.0.2rc1/
Providers with new features:
- [ ] amazon: @dstandish, @uranusjr @blcksrx @markhopson @Visya @JavierLopezT @mmoyses @ferruzzi https://pypi.org/project/apache-airflow-providers-amazon/1.3.0rc1/
- [ ] apache.livy: @dsynkov https://pypi.org/project/apache-airflow-providers-apache-livy/1.1.0rc1/
- [ ] cncf.kubernetes: @ashb @akrava @dimberman @SamWheating https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.1.0rc1/
- [ ] docker: @mzaforas https://pypi.org/project/apache-airflow-providers-docker/1.1.0rc1/
- [ ] google: @SamWheating @TobKed @fhoda @sschaetz @jmcarp @radim94 https://pypi.org/project/apache-airflow-providers-google/2.2.0rc1/
- [ ] microsoft.azure: @gmcrocetti https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.3.0rc1/
- [ ] microsoft.winrm: @shizidushu https://pypi.org/project/apache-airflow-providers-microsoft-winrm/1.1.0rc1/
- [ ] mysql: @potiuk https://pypi.org/project/apache-airflow-providers-mysql/1.1.0rc1/
- [ ] salesforce: @jyotidhiman0610 https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc2/
- [ ] snowflake: @sunkickr https://pypi.org/project/apache-airflow-providers-snowflake/1.2.0rc1/
Providers with trivial changes (templated fields and using cached_property, type hints) - no need to test:
* facebook https://pypi.org/project/apache-airflow-providers-facebook/1.1.0rc1/
* grpc https://pypi.org/project/apache-airflow-providers-grpc/1.1.0rc1/
* oracle https://pypi.org/project/apache-airflow-providers-oracle/1.1.0rc1/
* opsgenie https://pypi.org/project/apache-airflow-providers-opsgenie/1.0.2rc1/
* singularity https://pypi.org/project/apache-airflow-providers-singularity/1.1.0rc1/
* ssh https://pypi.org/project/apache-airflow-providers-ssh/1.3.0rc1/
Note the links to documentation from PyPI packages are not working until we merge
the changes to airflow site after releasing the packages officially.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] fhoda commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
fhoda commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-816017271
@potiuk I tested mine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sunkickr commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
sunkickr commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817763516
@ptiuk -- I have attempted to "smoke tested" the snowflake connection. In
my local 2.0.0 airflow, values entered into the new Snowflake connection
form fields do not seem to save. They disappear after saving the
connection. I upgraded to 2.0.1(airflow) and everything works fine. I do
not know why the new feature works in 2.0.1 and not in 2.0.0
On Mon, Apr 12, 2021 at 6:40 AM Jarek Potiuk ***@***.***>
wrote:
> Thanks @uranusjr <https://github.com/uranusjr> !
>
> Sorry @potiuk <https://github.com/potiuk> I have been a bit swamped... my
> change was just changing log level from info to debug
>
> Perfectly understand :). No problem
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/15241#issuecomment-817701829>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/APCBCR7K4BJW75ELVDR5CPDTILE2ZANCNFSM42PYW6ZA>
> .
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817265706
Thanks @SangwanP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shizidushu commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
shizidushu commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-816584479
@potiuk Do you mean use the operator to check if there is any problem? In this case, I will do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-818051410
> Hey there, I'm hoping to test my changes to the dataproc operators sometime today or tomorrow. The CNCF changes are harder to test as the issue in question only surfaces occasionally and only when running large workflows on a kubernetes cluster. In [the original PR](https://github.com/apache/airflow/pull/15137) I did a pretty thorough test, I hope this will suffice.
Sure that's perfect. I am going to release it as is.
> While setting up my local environment to test the google provider, the initial installation broke because of the introduction of `plyvel` (added in https://github.com/apache/airflow/pull/14105/files) which depends on a preexisting `leveldb` installation.
Very Good point. Usually we do not consider "dependency" changes as breaking (you generally find out when you try upgrade whether this is a problem or not). For the only "production" supported environment (linux) `plyvel` has binary releases available for manylinux: https://pypi.org/project/plyvel/#files so it should "just work" for those.
This is a major release with new feature/new dependency clearly marked in the documentation, so this should be fine.
Thanks for pointing it out !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817106344
Hey everyone. Thanks to all who've already smoke-tested 9 of those providers :). We are progressing :)
While it is not necessary and is not a blocker, I'd love to get at least some thumbs-up for just a basic testing of the changes for the remaining providers. We can't test them ourselves as we have no external services, but I thin it's much for you just installing it at whatever you have them and rough "looks good" should be ok (it's not a "stamp of approval", it's more "smoke test"). I'd love some comments on those:
* hashicorp: @SangwanP https://pypi.org/project/apache-airflow-providers-hashicorp/1.0.2rc1/
* amazon: @dstandish, @uranusjr @blcksrx @markhopson @Visya @JavierLopezT @mmoyses @ferruzzi https://pypi.org/project/apache-airflow-providers-amazon/1.3.0rc1/
* cncf.kubernetes: @ashb @akrava @dimberman @SamWheating https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.1.0rc1/
* microsoft.azure: @gmcrocetti https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.3.0rc1/
* salesforce: @jyotidhiman0610 https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc2/
* snowflake: @sunkickr https://pypi.org/project/apache-airflow-providers-snowflake/1.2.0rc1/
The google one was smoke tested by @fhoda (thanks!), but there were some other changes so if anyone has a chance I'd love some thumbs up (or down) from others as well:
* google: @SamWheating @TobKed @sschaetz @jmcarp @radim94 https://pypi.org/project/apache-airflow-providers-google/2.2.0rc1/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcosmarxm commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
marcosmarxm commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-816680026
@potiuk I tested mine, it works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-818051410
> Hey there, I'm hoping to test my changes to the dataproc operators sometime today or tomorrow. The CNCF changes are harder to test as the issue in question only surfaces occasionally and only when running large workflows on a kubernetes cluster. In [the original PR](https://github.com/apache/airflow/pull/15137) I did a pretty thorough test, I hope this will suffice.
Sure that's perfect. I am going to release it as is.
> While setting up my local environment to test the google provider, the initial installation broke because of the introduction of `plyvel` (added in https://github.com/apache/airflow/pull/14105/files) which depends on a preexisting `leveldb` installation.
Very Good point. Usually we do not consider "dependency" changes as breaking (you generally find out when you try upgrade whether this is a problem or not). For the only "production" supported environment (linux) `plyvel` has binary releases available for manylinux: https://pypi.org/project/plyvel/#files so it should "just work" for those.
This is a major release with new feature/new dependency clearly marked in the documentation, so this should be fine.
hanks for pointing it out !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817603964
Last chance today to take a look :)
* amazon: @dstandish, @uranusjr @blcksrx @markhopson @Visya @JavierLopezT @mmoyses @ferruzzi https://pypi.org/project/apache-airflow-providers-amazon/1.3.0rc1/
* microsof.azure @gmcrocetti https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.3.0rc1/
* salesforce: @jyotidhiman0610 https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc2/
* snowflake: @sunkickr https://pypi.org/project/apache-airflow-providers-snowflake/1.2.0rc1/
If some of you could just take a quick look and confirm that the last 4 packages are looking good (basically install and passes some basic smoke tests), that would be fantastic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817146826
> @potiuk I tested the GCS timespan transform operator, it works.
Thanks @sschaetz !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mzaforas commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
mzaforas commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-816623625
Hi @potiuk I have tested docker operator, it works as expected :)
Note that I have to applied the workaround described here https://github.com/apache/airflow/issues/13487 to test docker operator because I was getting and issue related to xcom serialization with Docker Operator. This issue is independant of my changes and I saw there is another PR to fix it.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817191952
I have enough votes to release, but if anyone else would still come back with some smoke test results during the weekend - I'd appreciate it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
blcksrx commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-814753789
@potiuk I did mine, but you mean there are some sorts of methods or etc that have no test for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817690580
The Amazon package looks right to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817134575
Thanks @akrava !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-815014937
Hi @potiuk , the most recent fix for `apache.hive` was a docstring fix. I have marked it as test done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-816618525
> @potiuk Do you mean use the operator to check if there is any problem? In this case, I will do it.
yep :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SamWheating edited a comment on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
SamWheating edited a comment on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817842664
Hey there, I'm hoping to test my changes to the dataproc operators sometime today or tomorrow. The CNCF changes are harder to test as the issue in question only surfaces occasionally and only when running large workflows on a kubernetes cluster. In [the original PR](https://github.com/apache/airflow/pull/15137) I did a pretty thorough test, I hope this will suffice.
While setting up my local environment to test the google provider, the initial installation broke because of the introduction of `plyvel` which depends on a preexisting `leveldb` installation.
```
plyvel/_plyvel.cpp:632:10: fatal error: 'leveldb/db.h' file not found
#include "leveldb/db.h"
^~~~~~~~~~~~~~
1 error generated.
error: command 'clang' failed with exit status 1
```
It was easy to fix with a simple `brew install leveldb`, but I just wanted to check that you were aware of a new dependency which could break some people's builds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-818046152
> @ptiuk -- I have attempted to "smoke tested" the snowflake connection. In my local 2.0.0 airflow, values entered into the new Snowflake connection form fields do not seem to save. They disappear after saving the connection. I upgraded to 2.0.1(airflow) and everything works fine. I do not know why the new feature works in 2.0.1 and not in 2.0.0
It was a bug fixed in 2.0.1 : https://github.com/apache/airflow/issues/13597 so all as expected! Thanks @sunkickr !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817701829
Thanks @uranusjr !
> Sorry @potiuk I have been a bit swamped... my change was just changing log level from info to debug
Perfectly understand :). No problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SangwanP commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
SangwanP commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817202788
@potiuk -- I tested the Hashicorp fix, it works as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-818261717
Just adding more comments:
- I looked at microsoft.azure and the changes are fairly trivial - tested that azure_data_factory hook is there
- salesforce provider includes tableu separation - it's been tested before but we have not released salesforce provider back then (copy&paste mistake)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817660215
Sorry @potiuk I have been a bit swamped... my change was just changing log level from info to debug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sschaetz commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
sschaetz commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817143352
@potiuk I tested the GCS timespan transform operator, it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-816512483
Hello everyone here - I'd appreciate more feedback on those provoders :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #15241:
URL: https://github.com/apache/airflow/issues/15241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shizidushu commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
shizidushu commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-816722988
@potiuk Just tested mine, it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] akrava commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
akrava commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817109218
@potiuk I've tested mine, it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817146826
> @potiuk I tested the GCS timespan transform operator, it works.
Thanks @sschaetz !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dsynkov commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
dsynkov commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817010971
@potiuk Tested just now with Spark 3.0.1 on EMR 6.2.0. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-818261951
Thanks everyone. Preparing official release now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-815116114
> @potiuk I did mine, but you mean there are some sorts of methods or etc that have no test for it?
Thanks!
It is mostly about testing the part you added/modfied. We do not (yet) have automated system testing of everything so this is mosty about verification if the change you've done works as expected. There is no way to test it all manually for providers like amazon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SamWheating edited a comment on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
SamWheating edited a comment on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817842664
Hey there, I'm hoping to test my changes to the dataproc operators sometime today or tomorrow. The CNCF changes are harder to test as the issue in question only surfaces occasionally and only when running large workflows on a kubernetes cluster. In [the original PR](https://github.com/apache/airflow/pull/15137) I did a pretty thorough test, I hope this will suffice.
While setting up my local environment to test the google provider, the initial installation broke because of the introduction of `plyvel` (added in https://github.com/apache/airflow/pull/14105/files) which depends on a preexisting `leveldb` installation.
```
plyvel/_plyvel.cpp:632:10: fatal error: 'leveldb/db.h' file not found
#include "leveldb/db.h"
^~~~~~~~~~~~~~
1 error generated.
error: command 'clang' failed with exit status 1
```
It was easy to fix with a simple `brew install leveldb`, but I just wanted to check that you were aware of a new dependency which could break some people's builds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SamWheating commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
SamWheating commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-817842664
Hey there, I'm hoping to test my changes to the dataproc operators sometime today or tomorrow. The CNCF changes are harder to test as the issue in question only surfaces occasionally and only when running large workflows on a kubernetes cluster. In the original PR I did a pretty thorough test, I hope this will suffice.
While setting up my local environment to test the google provider, the initial installation broke because of the introduction of `plyvel` which depends on a preexisting `leveldb` installation.
```
plyvel/_plyvel.cpp:632:10: fatal error: 'leveldb/db.h' file not found
#include "leveldb/db.h"
^~~~~~~~~~~~~~
1 error generated.
error: command 'clang' failed with exit status 1
```
It was easy to fix with a simple `brew install leveldb`, but I just wanted to check that you were aware of a new dependency which could break some people's builds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15241: Testing Providers prepared 2021.04.07
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15241:
URL: https://github.com/apache/airflow/issues/15241#issuecomment-815116114
> @potiuk I did mine, but you mean there are some sorts of methods or etc that have no test for it?
Thanks!
It is mostly about testing the part you added/modfied. We do not (yet) have automated system testing of everything so this is mosty about verification if the change you've done works as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org