You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <Ja...@polidea.com> on 2020/10/02 18:00:43 UTC
[VOTE] Airflow Backport Providers 2020.10.5rc1
Hey all,
I have cut Airflow Backport Providers 2020.10.5rc1. This email is calling a
vote on the release,
which will last for 72 hours - which means that it will end on Mon, 5th
of October 2020, 19:59:56 CEST.
This is the first time we release "cncf.kubernetes" package after big
backporting of kubernetes changes
to 1.10.11 and 1.10.12, so I'd appreciate, if some of our users looking at
the devlist test test this provider
- especially if you are waiting for it.
Consider this my (binding) +1.
Airflow Backport Providers 2020.10.5rc1 are available at:
https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/
*apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is a source
release that comes
with INSTALL instructions.
*apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz* are
the binary
Python "sdist" release.
Public keys are available at:
https://dist.apache.org/repos/dist/release/airflow/KEYS
Please vote accordingly:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Only votes from PMC members are binding, but members of the community are
encouraged to test the release and vote with "(non-binding)".
Please note that the version number excludes the 'rcX' string, so it's now
simply 2020.10.5. This will allow us to rename the artifact without
modifying
the artifact checksums when we actually release.
Each of the packages contains a detailed changelog. Here is the list of
links to
the released packages and changelogs:
https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/
https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/
Cheers,
Jarek
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Airflow Backport Providers 2020.10.5rc1
Posted by Jarek Potiuk <Ja...@polidea.com>.
Technically, the vote has passed - but since the voting was over the
weekend, I will add a day if someone would like to add more testing.
The current consensus about releasing of 2020.10.5 packages is:
1) We release all the packages except "cncf.kubernetes" now
2) Once the 'cncf.kubernetes' gets fixed we release new
version together with 'google' provider (GKEPod\startOperator) that depends
on it.
J.
On Tue, Oct 6, 2020 at 10:36 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> That was exactly what I was proposing :). So we are in perfect sync here.
>
> On Tue, Oct 6, 2020 at 10:34 AM Kamil Breguła <ka...@polidea.com>
> wrote:
>
>> Hi.
>>
>> I discussed this topic with Kaxil and we have a solution that could be
>> good for everyone.
>>
>> What do you think to release everything except Kubernetes now, and next
>> week two packages - Google and Kubernetes? This way, users will be able to
>> use all operators quickly and we will not have packages that have known
>> defects.
>>
>> Best regards,
>> Kamil Breguła
>>
>> On Tue, Oct 6, 2020 at 9:33 AM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> Kamil - I understand your concerns but we discussed it yesterday (at the
>>> 2.0 meeting - notes will be posted soon) that the whole point of the
>>> backport packages and separate providers is that we CAN release a subset of
>>> those separately.
>>>
>>> And the process that seems to be converging for releases is that we can
>>> exclude particular providers from the release without impacting
>>> the releasing of all the other providers.
>>>
>>> The only reason why we are releasing all the packages together is
>>> efficiency in voting (and we only release all of them together because we
>>> black-reformatted all of them and they have modifications that change
>>> source line numbers etc, so it makes sense to release all of them).
>>>
>>> I believe the -1 is not justified here. Can you please reconsider? I'd
>>> love we all agree on the process before I make the decision as release
>>> manager.
>>>
>>> Also just to remind the rules on package releases (
>>> https://www.apache.org/foundation/voting.html) - voting on package
>>> releases cannot be vetoed:
>>>
>>> VOTES ON PACKAGE RELEASES
>>>
>>> Votes on whether a package is ready to be released use majority approval
>>> -- i.e. at least three PMC members must vote affirmatively for release, and
>>> there must be more positive than negative votes. Releases *may not be
>>> vetoed*. Generally the community will cancel the release vote if anyone
>>> identifies serious problems, but in most cases the ultimate decision, lies
>>> with the individual serving as release manager. The specifics of the
>>> process may vary from project to project, but the 'minimum quorum of three
>>> +1 votes' rule is universal.
>>>
>>> J.
>>>
>>>
>>> On Tue, Oct 6, 2020 at 12:14 AM Kamil Breguła <ka...@polidea.com>
>>> wrote:
>>>
>>>> Is there any time pressure or any other pressure for us to release this
>>>> package now? I think Kubernetes is highly anticipated by users and it is
>>>> worth considering the arguments that motivate us to make a given decision.
>>>>
>>>> -1 to release without cncf.kubernetes and an incomplete google package
>>>> (without GKEPodOperator). Sorry.
>>>>
>>>> On Mon, Oct 5, 2020 at 10:29 PM Kaxil Naik <ka...@gmail.com> wrote:
>>>>
>>>>> +1 to release without cncf.kubernetes
>>>>>
>>>>> On Mon, Oct 5, 2020 at 6:27 PM Jarek Potiuk <Ja...@polidea.com>
>>>>> wrote:
>>>>>
>>>>>> (though of course releasing it without Kubernetes might be a little
>>>>>> less exciting Daniel ;)
>>>>>>
>>>>>> On Mon, Oct 5, 2020 at 7:25 PM Jarek Potiuk <Ja...@polidea.com>
>>>>>> wrote:
>>>>>>
>>>>>>> We have 2 +1s now and the comment about the 'cncf.kubernetes" not
>>>>>>> yet working.
>>>>>>>
>>>>>>> My proposal:
>>>>>>>
>>>>>>> * we continue with the voting (hopefully one more +1) and release
>>>>>>> all the sources + backport but do nto publish the cncf.kubernetes one.
>>>>>>> * we fix (who can take a look?) the cncf.kubernetes one and release
>>>>>>> and vote cncf.kubernetes separately.
>>>>>>>
>>>>>>> I think it will also happen in the future where we will be releasing
>>>>>>> a batch of providers and we detect problems with one of them. I think we
>>>>>>> need some agreed process for that and this might be what I explained.
>>>>>>>
>>>>>>> J,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 5, 2020 at 1:56 AM Kamil Breguła <
>>>>>>> kamil.bregula@polidea.com> wrote:
>>>>>>>
>>>>>>>> I see another problem. The kubernetes_engine.GKEStartPodOperator
>>>>>>>> operator is not included in the Google package and the kubernetes_engine
>>>>>>>> module contains references to the
>>>>>>>> airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator
>>>>>>>> operator instead of providers.cncf.kubernetes package.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Oct 5, 2020 at 1:47 AM Kamil Breguła <
>>>>>>>> kamil.bregula@polidea.com> wrote:
>>>>>>>>
>>>>>>>>> I started testing and encountered three problems - Two minor and
>>>>>>>>> one serious. I used official Docker images for
>>>>>>>>> testing: apache/airflow:1.10.12 with gcloud installed for GKE authorization
>>>>>>>>>
>>>>>>>>> 1. The configuration for StackdriverTaskHandlle is not obvious and
>>>>>>>>> is not documented anywhere.
>>>>>>>>> I managed to do this when I created the
>>>>>>>>> /opt/airflow/config/log_config.py file and set an environment variable
>>>>>>>>> - AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG
>>>>>>>>> File log_config.py:
>>>>>>>>> https://gist.github.com/mik-laj/f94e9940bfd3f08cf69287ac62652bc5
>>>>>>>>>
>>>>>>>>> However, I had more problems when I tried to use
>>>>>>>>> KubernetesPodOperator.
>>>>>>>>>
>>>>>>>>> 2. The following files are not included in the backport package
>>>>>>>>> cnd.kubernetes:
>>>>>>>>> -
>>>>>>>>> airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py,
>>>>>>>>>
>>>>>>>>> - airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_spark_pi.yaml
>>>>>>>>>
>>>>>>>>> I downloaded file /example_spark_kubernetes_spark_pi.yaml from our
>>>>>>>>> repository and example DAG works
>>>>>>>>>
>>>>>>>>> 3. I haven't had such success with another example
>>>>>>>>> - example_kubernetes_operator. I performed the following steps which ended
>>>>>>>>> with an error.
>>>>>>>>> $ curl -LO
>>>>>>>>> https://raw.githubusercontent.com/apache/airflow/master/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py
>>>>>>>>> $ cat example_kubernetes.py | sed
>>>>>>>>> 's/operators.bash/operators.bash_operator/g' | sed
>>>>>>>>> 's/in_cluster=True/in_cluster=False/g' > a.py
>>>>>>>>> $ mv a.py example_kubernetes.py
>>>>>>>>> $ airflow test example_kubernetes_operator write-xcom 2010-01-01
>>>>>>>>> .....
>>>>>>>>> [2020-10-04 23:29:06,623] {taskinstance.py:1150} ERROR - type
>>>>>>>>> object 'PodGenerator' has no attribute 'add_xcom_sidecar'
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> File
>>>>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py",
>>>>>>>>> line 984, in _run_raw_task
>>>>>>>>> result = task_copy.execute(context=context)
>>>>>>>>> File
>>>>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>>>>>>>> line 269, in execute
>>>>>>>>> self.pod = self.create_pod_request_obj()
>>>>>>>>> File
>>>>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>>>>>>>> line 405, in create_pod_request_obj
>>>>>>>>> pod = PodGenerator.add_xcom_sidecar(pod)
>>>>>>>>> AttributeError: type object 'PodGenerator' has no attribute
>>>>>>>>> 'add_xcom_sidecar'
>>>>>>>>>
>>>>>>>>> Have I missed any extra step I should do to run this example?
>>>>>>>>>
>>>>>>>>> On Sun, Oct 4, 2020 at 7:11 PM Daniel Imberman <
>>>>>>>>> daniel.imberman@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> +1 excited to finally encourage the Kubernetes providers!
>>>>>>>>>>
>>>>>>>>>> via Newton Mail
>>>>>>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2>
>>>>>>>>>>
>>>>>>>>>> On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk <
>>>>>>>>>> Jarek.Potiuk@polidea.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hey all,
>>>>>>>>>>
>>>>>>>>>> I have cut Airflow Backport Providers 2020.10.5rc1. This email is
>>>>>>>>>> calling a vote on the release,
>>>>>>>>>> which will last for 72 hours - which means that it will end on
>>>>>>>>>> Mon, 5th of October 2020, 19:59:56 CEST.
>>>>>>>>>>
>>>>>>>>>> This is the first time we release "cncf.kubernetes" package after
>>>>>>>>>> big backporting of kubernetes changes
>>>>>>>>>> to 1.10.11 and 1.10.12, so I'd appreciate, if some of our users
>>>>>>>>>> looking at the devlist test test this provider
>>>>>>>>>> - especially if you are waiting for it.
>>>>>>>>>>
>>>>>>>>>> Consider this my (binding) +1.
>>>>>>>>>>
>>>>>>>>>> Airflow Backport Providers 2020.10.5rc1 are available at:
>>>>>>>>>>
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> *apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is
>>>>>>>>>> a source release that comes
>>>>>>>>>> with INSTALL instructions.
>>>>>>>>>>
>>>>>>>>>> *apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz*
>>>>>>>>>> are the binary
>>>>>>>>>> Python "sdist" release.
>>>>>>>>>>
>>>>>>>>>> Public keys are available at:
>>>>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>>>>>>
>>>>>>>>>> Please vote accordingly:
>>>>>>>>>>
>>>>>>>>>> [ ] +1 approve
>>>>>>>>>> [ ] +0 no opinion
>>>>>>>>>> [ ] -1 disapprove with the reason
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Only votes from PMC members are binding, but members of the
>>>>>>>>>> community are
>>>>>>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>>>>>>
>>>>>>>>>> Please note that the version number excludes the 'rcX' string, so
>>>>>>>>>> it's now
>>>>>>>>>> simply 2020.10.5. This will allow us to rename the artifact
>>>>>>>>>> without modifying
>>>>>>>>>> the artifact checksums when we actually release.
>>>>>>>>>>
>>>>>>>>>> Each of the packages contains a detailed changelog. Here is the
>>>>>>>>>> list of links to
>>>>>>>>>> the released packages and changelogs:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Jarek
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Jarek Potiuk
>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>>>>>
>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Jarek Potiuk
>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>>
>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jarek Potiuk
>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>
>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>
>>>>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Airflow Backport Providers 2020.10.5rc1
Posted by Jarek Potiuk <Ja...@polidea.com>.
That was exactly what I was proposing :). So we are in perfect sync here.
On Tue, Oct 6, 2020 at 10:34 AM Kamil Breguła <ka...@polidea.com>
wrote:
> Hi.
>
> I discussed this topic with Kaxil and we have a solution that could be
> good for everyone.
>
> What do you think to release everything except Kubernetes now, and next
> week two packages - Google and Kubernetes? This way, users will be able to
> use all operators quickly and we will not have packages that have known
> defects.
>
> Best regards,
> Kamil Breguła
>
> On Tue, Oct 6, 2020 at 9:33 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Kamil - I understand your concerns but we discussed it yesterday (at the
>> 2.0 meeting - notes will be posted soon) that the whole point of the
>> backport packages and separate providers is that we CAN release a subset of
>> those separately.
>>
>> And the process that seems to be converging for releases is that we can
>> exclude particular providers from the release without impacting
>> the releasing of all the other providers.
>>
>> The only reason why we are releasing all the packages together is
>> efficiency in voting (and we only release all of them together because we
>> black-reformatted all of them and they have modifications that change
>> source line numbers etc, so it makes sense to release all of them).
>>
>> I believe the -1 is not justified here. Can you please reconsider? I'd
>> love we all agree on the process before I make the decision as release
>> manager.
>>
>> Also just to remind the rules on package releases (
>> https://www.apache.org/foundation/voting.html) - voting on package
>> releases cannot be vetoed:
>>
>> VOTES ON PACKAGE RELEASES
>>
>> Votes on whether a package is ready to be released use majority approval
>> -- i.e. at least three PMC members must vote affirmatively for release, and
>> there must be more positive than negative votes. Releases *may not be
>> vetoed*. Generally the community will cancel the release vote if anyone
>> identifies serious problems, but in most cases the ultimate decision, lies
>> with the individual serving as release manager. The specifics of the
>> process may vary from project to project, but the 'minimum quorum of three
>> +1 votes' rule is universal.
>>
>> J.
>>
>>
>> On Tue, Oct 6, 2020 at 12:14 AM Kamil Breguła <ka...@polidea.com>
>> wrote:
>>
>>> Is there any time pressure or any other pressure for us to release this
>>> package now? I think Kubernetes is highly anticipated by users and it is
>>> worth considering the arguments that motivate us to make a given decision.
>>>
>>> -1 to release without cncf.kubernetes and an incomplete google package
>>> (without GKEPodOperator). Sorry.
>>>
>>> On Mon, Oct 5, 2020 at 10:29 PM Kaxil Naik <ka...@gmail.com> wrote:
>>>
>>>> +1 to release without cncf.kubernetes
>>>>
>>>> On Mon, Oct 5, 2020 at 6:27 PM Jarek Potiuk <Ja...@polidea.com>
>>>> wrote:
>>>>
>>>>> (though of course releasing it without Kubernetes might be a little
>>>>> less exciting Daniel ;)
>>>>>
>>>>> On Mon, Oct 5, 2020 at 7:25 PM Jarek Potiuk <Ja...@polidea.com>
>>>>> wrote:
>>>>>
>>>>>> We have 2 +1s now and the comment about the 'cncf.kubernetes" not yet
>>>>>> working.
>>>>>>
>>>>>> My proposal:
>>>>>>
>>>>>> * we continue with the voting (hopefully one more +1) and release all
>>>>>> the sources + backport but do nto publish the cncf.kubernetes one.
>>>>>> * we fix (who can take a look?) the cncf.kubernetes one and release
>>>>>> and vote cncf.kubernetes separately.
>>>>>>
>>>>>> I think it will also happen in the future where we will be releasing
>>>>>> a batch of providers and we detect problems with one of them. I think we
>>>>>> need some agreed process for that and this might be what I explained.
>>>>>>
>>>>>> J,
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 5, 2020 at 1:56 AM Kamil Breguła <
>>>>>> kamil.bregula@polidea.com> wrote:
>>>>>>
>>>>>>> I see another problem. The kubernetes_engine.GKEStartPodOperator
>>>>>>> operator is not included in the Google package and the kubernetes_engine
>>>>>>> module contains references to the
>>>>>>> airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator
>>>>>>> operator instead of providers.cncf.kubernetes package.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 5, 2020 at 1:47 AM Kamil Breguła <
>>>>>>> kamil.bregula@polidea.com> wrote:
>>>>>>>
>>>>>>>> I started testing and encountered three problems - Two minor and
>>>>>>>> one serious. I used official Docker images for
>>>>>>>> testing: apache/airflow:1.10.12 with gcloud installed for GKE authorization
>>>>>>>>
>>>>>>>> 1. The configuration for StackdriverTaskHandlle is not obvious and
>>>>>>>> is not documented anywhere.
>>>>>>>> I managed to do this when I created the
>>>>>>>> /opt/airflow/config/log_config.py file and set an environment variable
>>>>>>>> - AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG
>>>>>>>> File log_config.py:
>>>>>>>> https://gist.github.com/mik-laj/f94e9940bfd3f08cf69287ac62652bc5
>>>>>>>>
>>>>>>>> However, I had more problems when I tried to use
>>>>>>>> KubernetesPodOperator.
>>>>>>>>
>>>>>>>> 2. The following files are not included in the backport package
>>>>>>>> cnd.kubernetes:
>>>>>>>> -
>>>>>>>> airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py,
>>>>>>>>
>>>>>>>> - airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_spark_pi.yaml
>>>>>>>>
>>>>>>>> I downloaded file /example_spark_kubernetes_spark_pi.yaml from our
>>>>>>>> repository and example DAG works
>>>>>>>>
>>>>>>>> 3. I haven't had such success with another example
>>>>>>>> - example_kubernetes_operator. I performed the following steps which ended
>>>>>>>> with an error.
>>>>>>>> $ curl -LO
>>>>>>>> https://raw.githubusercontent.com/apache/airflow/master/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py
>>>>>>>> $ cat example_kubernetes.py | sed
>>>>>>>> 's/operators.bash/operators.bash_operator/g' | sed
>>>>>>>> 's/in_cluster=True/in_cluster=False/g' > a.py
>>>>>>>> $ mv a.py example_kubernetes.py
>>>>>>>> $ airflow test example_kubernetes_operator write-xcom 2010-01-01
>>>>>>>> .....
>>>>>>>> [2020-10-04 23:29:06,623] {taskinstance.py:1150} ERROR - type
>>>>>>>> object 'PodGenerator' has no attribute 'add_xcom_sidecar'
>>>>>>>> Traceback (most recent call last):
>>>>>>>> File
>>>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py",
>>>>>>>> line 984, in _run_raw_task
>>>>>>>> result = task_copy.execute(context=context)
>>>>>>>> File
>>>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>>>>>>> line 269, in execute
>>>>>>>> self.pod = self.create_pod_request_obj()
>>>>>>>> File
>>>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>>>>>>> line 405, in create_pod_request_obj
>>>>>>>> pod = PodGenerator.add_xcom_sidecar(pod)
>>>>>>>> AttributeError: type object 'PodGenerator' has no attribute
>>>>>>>> 'add_xcom_sidecar'
>>>>>>>>
>>>>>>>> Have I missed any extra step I should do to run this example?
>>>>>>>>
>>>>>>>> On Sun, Oct 4, 2020 at 7:11 PM Daniel Imberman <
>>>>>>>> daniel.imberman@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> +1 excited to finally encourage the Kubernetes providers!
>>>>>>>>>
>>>>>>>>> via Newton Mail
>>>>>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2>
>>>>>>>>>
>>>>>>>>> On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk <
>>>>>>>>> Jarek.Potiuk@polidea.com> wrote:
>>>>>>>>>
>>>>>>>>> Hey all,
>>>>>>>>>
>>>>>>>>> I have cut Airflow Backport Providers 2020.10.5rc1. This email is
>>>>>>>>> calling a vote on the release,
>>>>>>>>> which will last for 72 hours - which means that it will end on
>>>>>>>>> Mon, 5th of October 2020, 19:59:56 CEST.
>>>>>>>>>
>>>>>>>>> This is the first time we release "cncf.kubernetes" package after
>>>>>>>>> big backporting of kubernetes changes
>>>>>>>>> to 1.10.11 and 1.10.12, so I'd appreciate, if some of our users
>>>>>>>>> looking at the devlist test test this provider
>>>>>>>>> - especially if you are waiting for it.
>>>>>>>>>
>>>>>>>>> Consider this my (binding) +1.
>>>>>>>>>
>>>>>>>>> Airflow Backport Providers 2020.10.5rc1 are available at:
>>>>>>>>>
>>>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> *apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is
>>>>>>>>> a source release that comes
>>>>>>>>> with INSTALL instructions.
>>>>>>>>>
>>>>>>>>> *apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz*
>>>>>>>>> are the binary
>>>>>>>>> Python "sdist" release.
>>>>>>>>>
>>>>>>>>> Public keys are available at:
>>>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>>>>>
>>>>>>>>> Please vote accordingly:
>>>>>>>>>
>>>>>>>>> [ ] +1 approve
>>>>>>>>> [ ] +0 no opinion
>>>>>>>>> [ ] -1 disapprove with the reason
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Only votes from PMC members are binding, but members of the
>>>>>>>>> community are
>>>>>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>>>>>
>>>>>>>>> Please note that the version number excludes the 'rcX' string, so
>>>>>>>>> it's now
>>>>>>>>> simply 2020.10.5. This will allow us to rename the artifact
>>>>>>>>> without modifying
>>>>>>>>> the artifact checksums when we actually release.
>>>>>>>>>
>>>>>>>>> Each of the packages contains a detailed changelog. Here is the
>>>>>>>>> list of links to
>>>>>>>>> the released packages and changelogs:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Jarek
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Jarek Potiuk
>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>>>>
>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jarek Potiuk
>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>
>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Jarek Potiuk
>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>
>>>>> M: +48 660 796 129 <+48660796129>
>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>
>>>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Airflow Backport Providers 2020.10.5rc1
Posted by Kamil Breguła <ka...@polidea.com>.
Hi.
I discussed this topic with Kaxil and we have a solution that could be good
for everyone.
What do you think to release everything except Kubernetes now, and next
week two packages - Google and Kubernetes? This way, users will be able to
use all operators quickly and we will not have packages that have known
defects.
Best regards,
Kamil Breguła
On Tue, Oct 6, 2020 at 9:33 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Kamil - I understand your concerns but we discussed it yesterday (at the
> 2.0 meeting - notes will be posted soon) that the whole point of the
> backport packages and separate providers is that we CAN release a subset of
> those separately.
>
> And the process that seems to be converging for releases is that we can
> exclude particular providers from the release without impacting
> the releasing of all the other providers.
>
> The only reason why we are releasing all the packages together is
> efficiency in voting (and we only release all of them together because we
> black-reformatted all of them and they have modifications that change
> source line numbers etc, so it makes sense to release all of them).
>
> I believe the -1 is not justified here. Can you please reconsider? I'd
> love we all agree on the process before I make the decision as release
> manager.
>
> Also just to remind the rules on package releases (
> https://www.apache.org/foundation/voting.html) - voting on package
> releases cannot be vetoed:
>
> VOTES ON PACKAGE RELEASES
>
> Votes on whether a package is ready to be released use majority approval
> -- i.e. at least three PMC members must vote affirmatively for release, and
> there must be more positive than negative votes. Releases *may not be
> vetoed*. Generally the community will cancel the release vote if anyone
> identifies serious problems, but in most cases the ultimate decision, lies
> with the individual serving as release manager. The specifics of the
> process may vary from project to project, but the 'minimum quorum of three
> +1 votes' rule is universal.
>
> J.
>
>
> On Tue, Oct 6, 2020 at 12:14 AM Kamil Breguła <ka...@polidea.com>
> wrote:
>
>> Is there any time pressure or any other pressure for us to release this
>> package now? I think Kubernetes is highly anticipated by users and it is
>> worth considering the arguments that motivate us to make a given decision.
>>
>> -1 to release without cncf.kubernetes and an incomplete google package
>> (without GKEPodOperator). Sorry.
>>
>> On Mon, Oct 5, 2020 at 10:29 PM Kaxil Naik <ka...@gmail.com> wrote:
>>
>>> +1 to release without cncf.kubernetes
>>>
>>> On Mon, Oct 5, 2020 at 6:27 PM Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>>> (though of course releasing it without Kubernetes might be a little
>>>> less exciting Daniel ;)
>>>>
>>>> On Mon, Oct 5, 2020 at 7:25 PM Jarek Potiuk <Ja...@polidea.com>
>>>> wrote:
>>>>
>>>>> We have 2 +1s now and the comment about the 'cncf.kubernetes" not yet
>>>>> working.
>>>>>
>>>>> My proposal:
>>>>>
>>>>> * we continue with the voting (hopefully one more +1) and release all
>>>>> the sources + backport but do nto publish the cncf.kubernetes one.
>>>>> * we fix (who can take a look?) the cncf.kubernetes one and release
>>>>> and vote cncf.kubernetes separately.
>>>>>
>>>>> I think it will also happen in the future where we will be releasing a
>>>>> batch of providers and we detect problems with one of them. I think we need
>>>>> some agreed process for that and this might be what I explained.
>>>>>
>>>>> J,
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 5, 2020 at 1:56 AM Kamil Breguła <
>>>>> kamil.bregula@polidea.com> wrote:
>>>>>
>>>>>> I see another problem. The kubernetes_engine.GKEStartPodOperator
>>>>>> operator is not included in the Google package and the kubernetes_engine
>>>>>> module contains references to the
>>>>>> airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator
>>>>>> operator instead of providers.cncf.kubernetes package.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 5, 2020 at 1:47 AM Kamil Breguła <
>>>>>> kamil.bregula@polidea.com> wrote:
>>>>>>
>>>>>>> I started testing and encountered three problems - Two minor and one
>>>>>>> serious. I used official Docker images for
>>>>>>> testing: apache/airflow:1.10.12 with gcloud installed for GKE authorization
>>>>>>>
>>>>>>> 1. The configuration for StackdriverTaskHandlle is not obvious and
>>>>>>> is not documented anywhere.
>>>>>>> I managed to do this when I created the
>>>>>>> /opt/airflow/config/log_config.py file and set an environment variable
>>>>>>> - AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG
>>>>>>> File log_config.py:
>>>>>>> https://gist.github.com/mik-laj/f94e9940bfd3f08cf69287ac62652bc5
>>>>>>>
>>>>>>> However, I had more problems when I tried to use
>>>>>>> KubernetesPodOperator.
>>>>>>>
>>>>>>> 2. The following files are not included in the backport package
>>>>>>> cnd.kubernetes:
>>>>>>> -
>>>>>>> airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py,
>>>>>>>
>>>>>>> - airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_spark_pi.yaml
>>>>>>>
>>>>>>> I downloaded file /example_spark_kubernetes_spark_pi.yaml from our
>>>>>>> repository and example DAG works
>>>>>>>
>>>>>>> 3. I haven't had such success with another example
>>>>>>> - example_kubernetes_operator. I performed the following steps which ended
>>>>>>> with an error.
>>>>>>> $ curl -LO
>>>>>>> https://raw.githubusercontent.com/apache/airflow/master/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py
>>>>>>> $ cat example_kubernetes.py | sed
>>>>>>> 's/operators.bash/operators.bash_operator/g' | sed
>>>>>>> 's/in_cluster=True/in_cluster=False/g' > a.py
>>>>>>> $ mv a.py example_kubernetes.py
>>>>>>> $ airflow test example_kubernetes_operator write-xcom 2010-01-01
>>>>>>> .....
>>>>>>> [2020-10-04 23:29:06,623] {taskinstance.py:1150} ERROR - type object
>>>>>>> 'PodGenerator' has no attribute 'add_xcom_sidecar'
>>>>>>> Traceback (most recent call last):
>>>>>>> File
>>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py",
>>>>>>> line 984, in _run_raw_task
>>>>>>> result = task_copy.execute(context=context)
>>>>>>> File
>>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>>>>>> line 269, in execute
>>>>>>> self.pod = self.create_pod_request_obj()
>>>>>>> File
>>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>>>>>> line 405, in create_pod_request_obj
>>>>>>> pod = PodGenerator.add_xcom_sidecar(pod)
>>>>>>> AttributeError: type object 'PodGenerator' has no attribute
>>>>>>> 'add_xcom_sidecar'
>>>>>>>
>>>>>>> Have I missed any extra step I should do to run this example?
>>>>>>>
>>>>>>> On Sun, Oct 4, 2020 at 7:11 PM Daniel Imberman <
>>>>>>> daniel.imberman@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1 excited to finally encourage the Kubernetes providers!
>>>>>>>>
>>>>>>>> via Newton Mail
>>>>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2>
>>>>>>>>
>>>>>>>> On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk <
>>>>>>>> Jarek.Potiuk@polidea.com> wrote:
>>>>>>>>
>>>>>>>> Hey all,
>>>>>>>>
>>>>>>>> I have cut Airflow Backport Providers 2020.10.5rc1. This email is
>>>>>>>> calling a vote on the release,
>>>>>>>> which will last for 72 hours - which means that it will end on Mon,
>>>>>>>> 5th of October 2020, 19:59:56 CEST.
>>>>>>>>
>>>>>>>> This is the first time we release "cncf.kubernetes" package after
>>>>>>>> big backporting of kubernetes changes
>>>>>>>> to 1.10.11 and 1.10.12, so I'd appreciate, if some of our users
>>>>>>>> looking at the devlist test test this provider
>>>>>>>> - especially if you are waiting for it.
>>>>>>>>
>>>>>>>> Consider this my (binding) +1.
>>>>>>>>
>>>>>>>> Airflow Backport Providers 2020.10.5rc1 are available at:
>>>>>>>>
>>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/
>>>>>>>>
>>>>>>>> *apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is a
>>>>>>>> source release that comes
>>>>>>>> with INSTALL instructions.
>>>>>>>>
>>>>>>>> *apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz*
>>>>>>>> are the binary
>>>>>>>> Python "sdist" release.
>>>>>>>>
>>>>>>>> Public keys are available at:
>>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>>>>
>>>>>>>> Please vote accordingly:
>>>>>>>>
>>>>>>>> [ ] +1 approve
>>>>>>>> [ ] +0 no opinion
>>>>>>>> [ ] -1 disapprove with the reason
>>>>>>>>
>>>>>>>>
>>>>>>>> Only votes from PMC members are binding, but members of the
>>>>>>>> community are
>>>>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>>>>
>>>>>>>> Please note that the version number excludes the 'rcX' string, so
>>>>>>>> it's now
>>>>>>>> simply 2020.10.5. This will allow us to rename the artifact without
>>>>>>>> modifying
>>>>>>>> the artifact checksums when we actually release.
>>>>>>>>
>>>>>>>> Each of the packages contains a detailed changelog. Here is the
>>>>>>>> list of links to
>>>>>>>> the released packages and changelogs:
>>>>>>>>
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Jarek
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Jarek Potiuk
>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>>>
>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>
>>>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Jarek Potiuk
>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>
>>>>> M: +48 660 796 129 <+48660796129>
>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Jarek Potiuk
>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>
>>>> M: +48 660 796 129 <+48660796129>
>>>> [image: Polidea] <https://www.polidea.com/>
>>>>
>>>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
Re: [VOTE] Airflow Backport Providers 2020.10.5rc1
Posted by Jarek Potiuk <Ja...@polidea.com>.
Kamil - I understand your concerns but we discussed it yesterday (at the
2.0 meeting - notes will be posted soon) that the whole point of the
backport packages and separate providers is that we CAN release a subset of
those separately.
And the process that seems to be converging for releases is that we can
exclude particular providers from the release without impacting
the releasing of all the other providers.
The only reason why we are releasing all the packages together is
efficiency in voting (and we only release all of them together because we
black-reformatted all of them and they have modifications that change
source line numbers etc, so it makes sense to release all of them).
I believe the -1 is not justified here. Can you please reconsider? I'd love
we all agree on the process before I make the decision as release manager.
Also just to remind the rules on package releases (
https://www.apache.org/foundation/voting.html) - voting on package releases
cannot be vetoed:
VOTES ON PACKAGE RELEASES
Votes on whether a package is ready to be released use majority approval --
i.e. at least three PMC members must vote affirmatively for release, and
there must be more positive than negative votes. Releases *may not be
vetoed*. Generally the community will cancel the release vote if anyone
identifies serious problems, but in most cases the ultimate decision, lies
with the individual serving as release manager. The specifics of the
process may vary from project to project, but the 'minimum quorum of three
+1 votes' rule is universal.
J.
On Tue, Oct 6, 2020 at 12:14 AM Kamil Breguła <ka...@polidea.com>
wrote:
> Is there any time pressure or any other pressure for us to release this
> package now? I think Kubernetes is highly anticipated by users and it is
> worth considering the arguments that motivate us to make a given decision.
>
> -1 to release without cncf.kubernetes and an incomplete google package
> (without GKEPodOperator). Sorry.
>
> On Mon, Oct 5, 2020 at 10:29 PM Kaxil Naik <ka...@gmail.com> wrote:
>
>> +1 to release without cncf.kubernetes
>>
>> On Mon, Oct 5, 2020 at 6:27 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> (though of course releasing it without Kubernetes might be a little less
>>> exciting Daniel ;)
>>>
>>> On Mon, Oct 5, 2020 at 7:25 PM Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>>> We have 2 +1s now and the comment about the 'cncf.kubernetes" not yet
>>>> working.
>>>>
>>>> My proposal:
>>>>
>>>> * we continue with the voting (hopefully one more +1) and release all
>>>> the sources + backport but do nto publish the cncf.kubernetes one.
>>>> * we fix (who can take a look?) the cncf.kubernetes one and release and
>>>> vote cncf.kubernetes separately.
>>>>
>>>> I think it will also happen in the future where we will be releasing a
>>>> batch of providers and we detect problems with one of them. I think we need
>>>> some agreed process for that and this might be what I explained.
>>>>
>>>> J,
>>>>
>>>>
>>>>
>>>> On Mon, Oct 5, 2020 at 1:56 AM Kamil Breguła <ka...@polidea.com>
>>>> wrote:
>>>>
>>>>> I see another problem. The kubernetes_engine.GKEStartPodOperator
>>>>> operator is not included in the Google package and the kubernetes_engine
>>>>> module contains references to the
>>>>> airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator
>>>>> operator instead of providers.cncf.kubernetes package.
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 5, 2020 at 1:47 AM Kamil Breguła <
>>>>> kamil.bregula@polidea.com> wrote:
>>>>>
>>>>>> I started testing and encountered three problems - Two minor and one
>>>>>> serious. I used official Docker images for
>>>>>> testing: apache/airflow:1.10.12 with gcloud installed for GKE authorization
>>>>>>
>>>>>> 1. The configuration for StackdriverTaskHandlle is not obvious and is
>>>>>> not documented anywhere.
>>>>>> I managed to do this when I created the
>>>>>> /opt/airflow/config/log_config.py file and set an environment variable
>>>>>> - AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG
>>>>>> File log_config.py:
>>>>>> https://gist.github.com/mik-laj/f94e9940bfd3f08cf69287ac62652bc5
>>>>>>
>>>>>> However, I had more problems when I tried to use
>>>>>> KubernetesPodOperator.
>>>>>>
>>>>>> 2. The following files are not included in the backport package
>>>>>> cnd.kubernetes:
>>>>>> -
>>>>>> airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py,
>>>>>>
>>>>>> - airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_spark_pi.yaml
>>>>>>
>>>>>> I downloaded file /example_spark_kubernetes_spark_pi.yaml from our
>>>>>> repository and example DAG works
>>>>>>
>>>>>> 3. I haven't had such success with another example
>>>>>> - example_kubernetes_operator. I performed the following steps which ended
>>>>>> with an error.
>>>>>> $ curl -LO
>>>>>> https://raw.githubusercontent.com/apache/airflow/master/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py
>>>>>> $ cat example_kubernetes.py | sed
>>>>>> 's/operators.bash/operators.bash_operator/g' | sed
>>>>>> 's/in_cluster=True/in_cluster=False/g' > a.py
>>>>>> $ mv a.py example_kubernetes.py
>>>>>> $ airflow test example_kubernetes_operator write-xcom 2010-01-01
>>>>>> .....
>>>>>> [2020-10-04 23:29:06,623] {taskinstance.py:1150} ERROR - type object
>>>>>> 'PodGenerator' has no attribute 'add_xcom_sidecar'
>>>>>> Traceback (most recent call last):
>>>>>> File
>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py",
>>>>>> line 984, in _run_raw_task
>>>>>> result = task_copy.execute(context=context)
>>>>>> File
>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>>>>> line 269, in execute
>>>>>> self.pod = self.create_pod_request_obj()
>>>>>> File
>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>>>>> line 405, in create_pod_request_obj
>>>>>> pod = PodGenerator.add_xcom_sidecar(pod)
>>>>>> AttributeError: type object 'PodGenerator' has no attribute
>>>>>> 'add_xcom_sidecar'
>>>>>>
>>>>>> Have I missed any extra step I should do to run this example?
>>>>>>
>>>>>> On Sun, Oct 4, 2020 at 7:11 PM Daniel Imberman <
>>>>>> daniel.imberman@gmail.com> wrote:
>>>>>>
>>>>>>> +1 excited to finally encourage the Kubernetes providers!
>>>>>>>
>>>>>>> via Newton Mail
>>>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2>
>>>>>>>
>>>>>>> On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk <
>>>>>>> Jarek.Potiuk@polidea.com> wrote:
>>>>>>>
>>>>>>> Hey all,
>>>>>>>
>>>>>>> I have cut Airflow Backport Providers 2020.10.5rc1. This email is
>>>>>>> calling a vote on the release,
>>>>>>> which will last for 72 hours - which means that it will end on Mon,
>>>>>>> 5th of October 2020, 19:59:56 CEST.
>>>>>>>
>>>>>>> This is the first time we release "cncf.kubernetes" package after
>>>>>>> big backporting of kubernetes changes
>>>>>>> to 1.10.11 and 1.10.12, so I'd appreciate, if some of our users
>>>>>>> looking at the devlist test test this provider
>>>>>>> - especially if you are waiting for it.
>>>>>>>
>>>>>>> Consider this my (binding) +1.
>>>>>>>
>>>>>>> Airflow Backport Providers 2020.10.5rc1 are available at:
>>>>>>>
>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/
>>>>>>>
>>>>>>> *apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is a
>>>>>>> source release that comes
>>>>>>> with INSTALL instructions.
>>>>>>>
>>>>>>> *apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz*
>>>>>>> are the binary
>>>>>>> Python "sdist" release.
>>>>>>>
>>>>>>> Public keys are available at:
>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>>>
>>>>>>> Please vote accordingly:
>>>>>>>
>>>>>>> [ ] +1 approve
>>>>>>> [ ] +0 no opinion
>>>>>>> [ ] -1 disapprove with the reason
>>>>>>>
>>>>>>>
>>>>>>> Only votes from PMC members are binding, but members of the
>>>>>>> community are
>>>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>>>
>>>>>>> Please note that the version number excludes the 'rcX' string, so
>>>>>>> it's now
>>>>>>> simply 2020.10.5. This will allow us to rename the artifact without
>>>>>>> modifying
>>>>>>> the artifact checksums when we actually release.
>>>>>>>
>>>>>>> Each of the packages contains a detailed changelog. Here is the list
>>>>>>> of links to
>>>>>>> the released packages and changelogs:
>>>>>>>
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/
>>>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Jarek
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Jarek Potiuk
>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>>
>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>>
>>>> Jarek Potiuk
>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>
>>>> M: +48 660 796 129 <+48660796129>
>>>> [image: Polidea] <https://www.polidea.com/>
>>>>
>>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Airflow Backport Providers 2020.10.5rc1
Posted by Kamil Breguła <ka...@polidea.com>.
Is there any time pressure or any other pressure for us to release this
package now? I think Kubernetes is highly anticipated by users and it is
worth considering the arguments that motivate us to make a given decision.
-1 to release without cncf.kubernetes and an incomplete google package
(without GKEPodOperator). Sorry.
On Mon, Oct 5, 2020 at 10:29 PM Kaxil Naik <ka...@gmail.com> wrote:
> +1 to release without cncf.kubernetes
>
> On Mon, Oct 5, 2020 at 6:27 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> (though of course releasing it without Kubernetes might be a little less
>> exciting Daniel ;)
>>
>> On Mon, Oct 5, 2020 at 7:25 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> We have 2 +1s now and the comment about the 'cncf.kubernetes" not yet
>>> working.
>>>
>>> My proposal:
>>>
>>> * we continue with the voting (hopefully one more +1) and release all
>>> the sources + backport but do nto publish the cncf.kubernetes one.
>>> * we fix (who can take a look?) the cncf.kubernetes one and release and
>>> vote cncf.kubernetes separately.
>>>
>>> I think it will also happen in the future where we will be releasing a
>>> batch of providers and we detect problems with one of them. I think we need
>>> some agreed process for that and this might be what I explained.
>>>
>>> J,
>>>
>>>
>>>
>>> On Mon, Oct 5, 2020 at 1:56 AM Kamil Breguła <ka...@polidea.com>
>>> wrote:
>>>
>>>> I see another problem. The kubernetes_engine.GKEStartPodOperator
>>>> operator is not included in the Google package and the kubernetes_engine
>>>> module contains references to the
>>>> airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator
>>>> operator instead of providers.cncf.kubernetes package.
>>>>
>>>>
>>>>
>>>> On Mon, Oct 5, 2020 at 1:47 AM Kamil Breguła <ka...@polidea.com>
>>>> wrote:
>>>>
>>>>> I started testing and encountered three problems - Two minor and one
>>>>> serious. I used official Docker images for
>>>>> testing: apache/airflow:1.10.12 with gcloud installed for GKE authorization
>>>>>
>>>>> 1. The configuration for StackdriverTaskHandlle is not obvious and is
>>>>> not documented anywhere.
>>>>> I managed to do this when I created the
>>>>> /opt/airflow/config/log_config.py file and set an environment variable
>>>>> - AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG
>>>>> File log_config.py:
>>>>> https://gist.github.com/mik-laj/f94e9940bfd3f08cf69287ac62652bc5
>>>>>
>>>>> However, I had more problems when I tried to use KubernetesPodOperator.
>>>>>
>>>>> 2. The following files are not included in the backport package
>>>>> cnd.kubernetes:
>>>>> -
>>>>> airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py,
>>>>>
>>>>> - airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_spark_pi.yaml
>>>>>
>>>>> I downloaded file /example_spark_kubernetes_spark_pi.yaml from our
>>>>> repository and example DAG works
>>>>>
>>>>> 3. I haven't had such success with another example
>>>>> - example_kubernetes_operator. I performed the following steps which ended
>>>>> with an error.
>>>>> $ curl -LO
>>>>> https://raw.githubusercontent.com/apache/airflow/master/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py
>>>>> $ cat example_kubernetes.py | sed
>>>>> 's/operators.bash/operators.bash_operator/g' | sed
>>>>> 's/in_cluster=True/in_cluster=False/g' > a.py
>>>>> $ mv a.py example_kubernetes.py
>>>>> $ airflow test example_kubernetes_operator write-xcom 2010-01-01
>>>>> .....
>>>>> [2020-10-04 23:29:06,623] {taskinstance.py:1150} ERROR - type object
>>>>> 'PodGenerator' has no attribute 'add_xcom_sidecar'
>>>>> Traceback (most recent call last):
>>>>> File
>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py",
>>>>> line 984, in _run_raw_task
>>>>> result = task_copy.execute(context=context)
>>>>> File
>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>>>> line 269, in execute
>>>>> self.pod = self.create_pod_request_obj()
>>>>> File
>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>>>> line 405, in create_pod_request_obj
>>>>> pod = PodGenerator.add_xcom_sidecar(pod)
>>>>> AttributeError: type object 'PodGenerator' has no attribute
>>>>> 'add_xcom_sidecar'
>>>>>
>>>>> Have I missed any extra step I should do to run this example?
>>>>>
>>>>> On Sun, Oct 4, 2020 at 7:11 PM Daniel Imberman <
>>>>> daniel.imberman@gmail.com> wrote:
>>>>>
>>>>>> +1 excited to finally encourage the Kubernetes providers!
>>>>>>
>>>>>> via Newton Mail
>>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2>
>>>>>>
>>>>>> On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk <
>>>>>> Jarek.Potiuk@polidea.com> wrote:
>>>>>>
>>>>>> Hey all,
>>>>>>
>>>>>> I have cut Airflow Backport Providers 2020.10.5rc1. This email is
>>>>>> calling a vote on the release,
>>>>>> which will last for 72 hours - which means that it will end on Mon,
>>>>>> 5th of October 2020, 19:59:56 CEST.
>>>>>>
>>>>>> This is the first time we release "cncf.kubernetes" package after big
>>>>>> backporting of kubernetes changes
>>>>>> to 1.10.11 and 1.10.12, so I'd appreciate, if some of our users
>>>>>> looking at the devlist test test this provider
>>>>>> - especially if you are waiting for it.
>>>>>>
>>>>>> Consider this my (binding) +1.
>>>>>>
>>>>>> Airflow Backport Providers 2020.10.5rc1 are available at:
>>>>>>
>>>>>> https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/
>>>>>>
>>>>>> *apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is a
>>>>>> source release that comes
>>>>>> with INSTALL instructions.
>>>>>>
>>>>>> *apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz*
>>>>>> are the binary
>>>>>> Python "sdist" release.
>>>>>>
>>>>>> Public keys are available at:
>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>>
>>>>>> Please vote accordingly:
>>>>>>
>>>>>> [ ] +1 approve
>>>>>> [ ] +0 no opinion
>>>>>> [ ] -1 disapprove with the reason
>>>>>>
>>>>>>
>>>>>> Only votes from PMC members are binding, but members of the community
>>>>>> are
>>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>>
>>>>>> Please note that the version number excludes the 'rcX' string, so
>>>>>> it's now
>>>>>> simply 2020.10.5. This will allow us to rename the artifact without
>>>>>> modifying
>>>>>> the artifact checksums when we actually release.
>>>>>>
>>>>>> Each of the packages contains a detailed changelog. Here is the list
>>>>>> of links to
>>>>>> the released packages and changelogs:
>>>>>>
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>> Jarek
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jarek Potiuk
>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>
>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>
>>>>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
Re: [VOTE] Airflow Backport Providers 2020.10.5rc1
Posted by Kaxil Naik <ka...@gmail.com>.
+1 to release without cncf.kubernetes
On Mon, Oct 5, 2020 at 6:27 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> (though of course releasing it without Kubernetes might be a little less
> exciting Daniel ;)
>
> On Mon, Oct 5, 2020 at 7:25 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> We have 2 +1s now and the comment about the 'cncf.kubernetes" not yet
>> working.
>>
>> My proposal:
>>
>> * we continue with the voting (hopefully one more +1) and release all the
>> sources + backport but do nto publish the cncf.kubernetes one.
>> * we fix (who can take a look?) the cncf.kubernetes one and release and
>> vote cncf.kubernetes separately.
>>
>> I think it will also happen in the future where we will be releasing a
>> batch of providers and we detect problems with one of them. I think we need
>> some agreed process for that and this might be what I explained.
>>
>> J,
>>
>>
>>
>> On Mon, Oct 5, 2020 at 1:56 AM Kamil Breguła <ka...@polidea.com>
>> wrote:
>>
>>> I see another problem. The kubernetes_engine.GKEStartPodOperator
>>> operator is not included in the Google package and the kubernetes_engine
>>> module contains references to the
>>> airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator
>>> operator instead of providers.cncf.kubernetes package.
>>>
>>>
>>>
>>> On Mon, Oct 5, 2020 at 1:47 AM Kamil Breguła <ka...@polidea.com>
>>> wrote:
>>>
>>>> I started testing and encountered three problems - Two minor and one
>>>> serious. I used official Docker images for
>>>> testing: apache/airflow:1.10.12 with gcloud installed for GKE authorization
>>>>
>>>> 1. The configuration for StackdriverTaskHandlle is not obvious and is
>>>> not documented anywhere.
>>>> I managed to do this when I created the
>>>> /opt/airflow/config/log_config.py file and set an environment variable
>>>> - AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG
>>>> File log_config.py:
>>>> https://gist.github.com/mik-laj/f94e9940bfd3f08cf69287ac62652bc5
>>>>
>>>> However, I had more problems when I tried to use KubernetesPodOperator.
>>>>
>>>> 2. The following files are not included in the backport package
>>>> cnd.kubernetes:
>>>> -
>>>> airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py,
>>>>
>>>> - airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_spark_pi.yaml
>>>>
>>>> I downloaded file /example_spark_kubernetes_spark_pi.yaml from our
>>>> repository and example DAG works
>>>>
>>>> 3. I haven't had such success with another example
>>>> - example_kubernetes_operator. I performed the following steps which ended
>>>> with an error.
>>>> $ curl -LO
>>>> https://raw.githubusercontent.com/apache/airflow/master/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py
>>>> $ cat example_kubernetes.py | sed
>>>> 's/operators.bash/operators.bash_operator/g' | sed
>>>> 's/in_cluster=True/in_cluster=False/g' > a.py
>>>> $ mv a.py example_kubernetes.py
>>>> $ airflow test example_kubernetes_operator write-xcom 2010-01-01
>>>> .....
>>>> [2020-10-04 23:29:06,623] {taskinstance.py:1150} ERROR - type object
>>>> 'PodGenerator' has no attribute 'add_xcom_sidecar'
>>>> Traceback (most recent call last):
>>>> File
>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py",
>>>> line 984, in _run_raw_task
>>>> result = task_copy.execute(context=context)
>>>> File
>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>>> line 269, in execute
>>>> self.pod = self.create_pod_request_obj()
>>>> File
>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>>> line 405, in create_pod_request_obj
>>>> pod = PodGenerator.add_xcom_sidecar(pod)
>>>> AttributeError: type object 'PodGenerator' has no attribute
>>>> 'add_xcom_sidecar'
>>>>
>>>> Have I missed any extra step I should do to run this example?
>>>>
>>>> On Sun, Oct 4, 2020 at 7:11 PM Daniel Imberman <
>>>> daniel.imberman@gmail.com> wrote:
>>>>
>>>>> +1 excited to finally encourage the Kubernetes providers!
>>>>>
>>>>> via Newton Mail
>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2>
>>>>>
>>>>> On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk <
>>>>> Jarek.Potiuk@polidea.com> wrote:
>>>>>
>>>>> Hey all,
>>>>>
>>>>> I have cut Airflow Backport Providers 2020.10.5rc1. This email is
>>>>> calling a vote on the release,
>>>>> which will last for 72 hours - which means that it will end on Mon,
>>>>> 5th of October 2020, 19:59:56 CEST.
>>>>>
>>>>> This is the first time we release "cncf.kubernetes" package after big
>>>>> backporting of kubernetes changes
>>>>> to 1.10.11 and 1.10.12, so I'd appreciate, if some of our users
>>>>> looking at the devlist test test this provider
>>>>> - especially if you are waiting for it.
>>>>>
>>>>> Consider this my (binding) +1.
>>>>>
>>>>> Airflow Backport Providers 2020.10.5rc1 are available at:
>>>>>
>>>>> https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/
>>>>>
>>>>> *apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is a
>>>>> source release that comes
>>>>> with INSTALL instructions.
>>>>>
>>>>> *apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz*
>>>>> are the binary
>>>>> Python "sdist" release.
>>>>>
>>>>> Public keys are available at:
>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>
>>>>> Please vote accordingly:
>>>>>
>>>>> [ ] +1 approve
>>>>> [ ] +0 no opinion
>>>>> [ ] -1 disapprove with the reason
>>>>>
>>>>>
>>>>> Only votes from PMC members are binding, but members of the community
>>>>> are
>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>
>>>>> Please note that the version number excludes the 'rcX' string, so it's
>>>>> now
>>>>> simply 2020.10.5. This will allow us to rename the artifact without
>>>>> modifying
>>>>> the artifact checksums when we actually release.
>>>>>
>>>>> Each of the packages contains a detailed changelog. Here is the list
>>>>> of links to
>>>>> the released packages and changelogs:
>>>>>
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/
>>>>>
>>>>>
>>>>> Cheers,
>>>>> Jarek
>>>>>
>>>>> --
>>>>>
>>>>> Jarek Potiuk
>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>
>>>>> M: +48 660 796 129 <+48660796129>
>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>
>>>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
Re: [VOTE] Airflow Backport Providers 2020.10.5rc1
Posted by Jarek Potiuk <Ja...@polidea.com>.
(though of course releasing it without Kubernetes might be a little less
exciting Daniel ;)
On Mon, Oct 5, 2020 at 7:25 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> We have 2 +1s now and the comment about the 'cncf.kubernetes" not yet
> working.
>
> My proposal:
>
> * we continue with the voting (hopefully one more +1) and release all the
> sources + backport but do nto publish the cncf.kubernetes one.
> * we fix (who can take a look?) the cncf.kubernetes one and release and
> vote cncf.kubernetes separately.
>
> I think it will also happen in the future where we will be releasing a
> batch of providers and we detect problems with one of them. I think we need
> some agreed process for that and this might be what I explained.
>
> J,
>
>
>
> On Mon, Oct 5, 2020 at 1:56 AM Kamil Breguła <ka...@polidea.com>
> wrote:
>
>> I see another problem. The kubernetes_engine.GKEStartPodOperator operator
>> is not included in the Google package and the kubernetes_engine module
>> contains references to the
>> airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator
>> operator instead of providers.cncf.kubernetes package.
>>
>>
>>
>> On Mon, Oct 5, 2020 at 1:47 AM Kamil Breguła <ka...@polidea.com>
>> wrote:
>>
>>> I started testing and encountered three problems - Two minor and one
>>> serious. I used official Docker images for
>>> testing: apache/airflow:1.10.12 with gcloud installed for GKE authorization
>>>
>>> 1. The configuration for StackdriverTaskHandlle is not obvious and is
>>> not documented anywhere.
>>> I managed to do this when I created the
>>> /opt/airflow/config/log_config.py file and set an environment variable
>>> - AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG
>>> File log_config.py:
>>> https://gist.github.com/mik-laj/f94e9940bfd3f08cf69287ac62652bc5
>>>
>>> However, I had more problems when I tried to use KubernetesPodOperator.
>>>
>>> 2. The following files are not included in the backport package
>>> cnd.kubernetes:
>>> -
>>> airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py,
>>>
>>> - airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_spark_pi.yaml
>>>
>>> I downloaded file /example_spark_kubernetes_spark_pi.yaml from our
>>> repository and example DAG works
>>>
>>> 3. I haven't had such success with another example
>>> - example_kubernetes_operator. I performed the following steps which ended
>>> with an error.
>>> $ curl -LO
>>> https://raw.githubusercontent.com/apache/airflow/master/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py
>>> $ cat example_kubernetes.py | sed
>>> 's/operators.bash/operators.bash_operator/g' | sed
>>> 's/in_cluster=True/in_cluster=False/g' > a.py
>>> $ mv a.py example_kubernetes.py
>>> $ airflow test example_kubernetes_operator write-xcom 2010-01-01
>>> .....
>>> [2020-10-04 23:29:06,623] {taskinstance.py:1150} ERROR - type object
>>> 'PodGenerator' has no attribute 'add_xcom_sidecar'
>>> Traceback (most recent call last):
>>> File
>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py",
>>> line 984, in _run_raw_task
>>> result = task_copy.execute(context=context)
>>> File
>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>> line 269, in execute
>>> self.pod = self.create_pod_request_obj()
>>> File
>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>>> line 405, in create_pod_request_obj
>>> pod = PodGenerator.add_xcom_sidecar(pod)
>>> AttributeError: type object 'PodGenerator' has no attribute
>>> 'add_xcom_sidecar'
>>>
>>> Have I missed any extra step I should do to run this example?
>>>
>>> On Sun, Oct 4, 2020 at 7:11 PM Daniel Imberman <
>>> daniel.imberman@gmail.com> wrote:
>>>
>>>> +1 excited to finally encourage the Kubernetes providers!
>>>>
>>>> via Newton Mail
>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2>
>>>>
>>>> On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk <Ja...@polidea.com>
>>>> wrote:
>>>>
>>>> Hey all,
>>>>
>>>> I have cut Airflow Backport Providers 2020.10.5rc1. This email is
>>>> calling a vote on the release,
>>>> which will last for 72 hours - which means that it will end on Mon, 5th
>>>> of October 2020, 19:59:56 CEST.
>>>>
>>>> This is the first time we release "cncf.kubernetes" package after big
>>>> backporting of kubernetes changes
>>>> to 1.10.11 and 1.10.12, so I'd appreciate, if some of our users looking
>>>> at the devlist test test this provider
>>>> - especially if you are waiting for it.
>>>>
>>>> Consider this my (binding) +1.
>>>>
>>>> Airflow Backport Providers 2020.10.5rc1 are available at:
>>>>
>>>> https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/
>>>>
>>>> *apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is a
>>>> source release that comes
>>>> with INSTALL instructions.
>>>>
>>>> *apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz*
>>>> are the binary
>>>> Python "sdist" release.
>>>>
>>>> Public keys are available at:
>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>
>>>> Please vote accordingly:
>>>>
>>>> [ ] +1 approve
>>>> [ ] +0 no opinion
>>>> [ ] -1 disapprove with the reason
>>>>
>>>>
>>>> Only votes from PMC members are binding, but members of the community
>>>> are
>>>> encouraged to test the release and vote with "(non-binding)".
>>>>
>>>> Please note that the version number excludes the 'rcX' string, so it's
>>>> now
>>>> simply 2020.10.5. This will allow us to rename the artifact without
>>>> modifying
>>>> the artifact checksums when we actually release.
>>>>
>>>> Each of the packages contains a detailed changelog. Here is the list of
>>>> links to
>>>> the released packages and changelogs:
>>>>
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/
>>>>
>>>>
>>>> Cheers,
>>>> Jarek
>>>>
>>>> --
>>>>
>>>> Jarek Potiuk
>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>
>>>> M: +48 660 796 129 <+48660796129>
>>>> [image: Polidea] <https://www.polidea.com/>
>>>>
>>>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Airflow Backport Providers 2020.10.5rc1
Posted by Jarek Potiuk <Ja...@polidea.com>.
We have 2 +1s now and the comment about the 'cncf.kubernetes" not yet
working.
My proposal:
* we continue with the voting (hopefully one more +1) and release all the
sources + backport but do nto publish the cncf.kubernetes one.
* we fix (who can take a look?) the cncf.kubernetes one and release and
vote cncf.kubernetes separately.
I think it will also happen in the future where we will be releasing a
batch of providers and we detect problems with one of them. I think we need
some agreed process for that and this might be what I explained.
J,
On Mon, Oct 5, 2020 at 1:56 AM Kamil Breguła <ka...@polidea.com>
wrote:
> I see another problem. The kubernetes_engine.GKEStartPodOperator operator
> is not included in the Google package and the kubernetes_engine module
> contains references to the
> airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator
> operator instead of providers.cncf.kubernetes package.
>
>
>
> On Mon, Oct 5, 2020 at 1:47 AM Kamil Breguła <ka...@polidea.com>
> wrote:
>
>> I started testing and encountered three problems - Two minor and one
>> serious. I used official Docker images for
>> testing: apache/airflow:1.10.12 with gcloud installed for GKE authorization
>>
>> 1. The configuration for StackdriverTaskHandlle is not obvious and is not
>> documented anywhere.
>> I managed to do this when I created the /opt/airflow/config/log_config.py
>> file and set an environment variable
>> - AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG
>> File log_config.py:
>> https://gist.github.com/mik-laj/f94e9940bfd3f08cf69287ac62652bc5
>>
>> However, I had more problems when I tried to use KubernetesPodOperator.
>>
>> 2. The following files are not included in the backport package
>> cnd.kubernetes:
>> - airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py,
>>
>> - airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_spark_pi.yaml
>>
>> I downloaded file /example_spark_kubernetes_spark_pi.yaml from our
>> repository and example DAG works
>>
>> 3. I haven't had such success with another example
>> - example_kubernetes_operator. I performed the following steps which ended
>> with an error.
>> $ curl -LO
>> https://raw.githubusercontent.com/apache/airflow/master/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py
>> $ cat example_kubernetes.py | sed
>> 's/operators.bash/operators.bash_operator/g' | sed
>> 's/in_cluster=True/in_cluster=False/g' > a.py
>> $ mv a.py example_kubernetes.py
>> $ airflow test example_kubernetes_operator write-xcom 2010-01-01
>> .....
>> [2020-10-04 23:29:06,623] {taskinstance.py:1150} ERROR - type object
>> 'PodGenerator' has no attribute 'add_xcom_sidecar'
>> Traceback (most recent call last):
>> File
>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py",
>> line 984, in _run_raw_task
>> result = task_copy.execute(context=context)
>> File
>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>> line 269, in execute
>> self.pod = self.create_pod_request_obj()
>> File
>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
>> line 405, in create_pod_request_obj
>> pod = PodGenerator.add_xcom_sidecar(pod)
>> AttributeError: type object 'PodGenerator' has no attribute
>> 'add_xcom_sidecar'
>>
>> Have I missed any extra step I should do to run this example?
>>
>> On Sun, Oct 4, 2020 at 7:11 PM Daniel Imberman <da...@gmail.com>
>> wrote:
>>
>>> +1 excited to finally encourage the Kubernetes providers!
>>>
>>> via Newton Mail
>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2>
>>>
>>> On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>> Hey all,
>>>
>>> I have cut Airflow Backport Providers 2020.10.5rc1. This email is
>>> calling a vote on the release,
>>> which will last for 72 hours - which means that it will end on Mon, 5th
>>> of October 2020, 19:59:56 CEST.
>>>
>>> This is the first time we release "cncf.kubernetes" package after big
>>> backporting of kubernetes changes
>>> to 1.10.11 and 1.10.12, so I'd appreciate, if some of our users looking
>>> at the devlist test test this provider
>>> - especially if you are waiting for it.
>>>
>>> Consider this my (binding) +1.
>>>
>>> Airflow Backport Providers 2020.10.5rc1 are available at:
>>>
>>> https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/
>>>
>>> *apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is a
>>> source release that comes
>>> with INSTALL instructions.
>>>
>>> *apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz*
>>> are the binary
>>> Python "sdist" release.
>>>
>>> Public keys are available at:
>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>
>>> Please vote accordingly:
>>>
>>> [ ] +1 approve
>>> [ ] +0 no opinion
>>> [ ] -1 disapprove with the reason
>>>
>>>
>>> Only votes from PMC members are binding, but members of the community are
>>> encouraged to test the release and vote with "(non-binding)".
>>>
>>> Please note that the version number excludes the 'rcX' string, so it's
>>> now
>>> simply 2020.10.5. This will allow us to rename the artifact without
>>> modifying
>>> the artifact checksums when we actually release.
>>>
>>> Each of the packages contains a detailed changelog. Here is the list of
>>> links to
>>> the released packages and changelogs:
>>>
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/
>>>
>>> https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/
>>>
>>>
>>> Cheers,
>>> Jarek
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Airflow Backport Providers 2020.10.5rc1
Posted by Kamil Breguła <ka...@polidea.com>.
I see another problem. The kubernetes_engine.GKEStartPodOperator operator
is not included in the Google package and the kubernetes_engine module
contains references to the
airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator
operator instead of providers.cncf.kubernetes package.
On Mon, Oct 5, 2020 at 1:47 AM Kamil Breguła <ka...@polidea.com>
wrote:
> I started testing and encountered three problems - Two minor and one
> serious. I used official Docker images for
> testing: apache/airflow:1.10.12 with gcloud installed for GKE authorization
>
> 1. The configuration for StackdriverTaskHandlle is not obvious and is not
> documented anywhere.
> I managed to do this when I created the /opt/airflow/config/log_config.py
> file and set an environment variable
> - AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG
> File log_config.py:
> https://gist.github.com/mik-laj/f94e9940bfd3f08cf69287ac62652bc5
>
> However, I had more problems when I tried to use KubernetesPodOperator.
>
> 2. The following files are not included in the backport package
> cnd.kubernetes:
> - airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py,
>
> - airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_spark_pi.yaml
>
> I downloaded file /example_spark_kubernetes_spark_pi.yaml from our
> repository and example DAG works
>
> 3. I haven't had such success with another example
> - example_kubernetes_operator. I performed the following steps which ended
> with an error.
> $ curl -LO
> https://raw.githubusercontent.com/apache/airflow/master/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py
> $ cat example_kubernetes.py | sed
> 's/operators.bash/operators.bash_operator/g' | sed
> 's/in_cluster=True/in_cluster=False/g' > a.py
> $ mv a.py example_kubernetes.py
> $ airflow test example_kubernetes_operator write-xcom 2010-01-01
> .....
> [2020-10-04 23:29:06,623] {taskinstance.py:1150} ERROR - type object
> 'PodGenerator' has no attribute 'add_xcom_sidecar'
> Traceback (most recent call last):
> File
> "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py",
> line 984, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
> line 269, in execute
> self.pod = self.create_pod_request_obj()
> File
> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
> line 405, in create_pod_request_obj
> pod = PodGenerator.add_xcom_sidecar(pod)
> AttributeError: type object 'PodGenerator' has no attribute
> 'add_xcom_sidecar'
>
> Have I missed any extra step I should do to run this example?
>
> On Sun, Oct 4, 2020 at 7:11 PM Daniel Imberman <da...@gmail.com>
> wrote:
>
>> +1 excited to finally encourage the Kubernetes providers!
>>
>> via Newton Mail
>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2>
>>
>> On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> Hey all,
>>
>> I have cut Airflow Backport Providers 2020.10.5rc1. This email is calling
>> a vote on the release,
>> which will last for 72 hours - which means that it will end on Mon, 5th
>> of October 2020, 19:59:56 CEST.
>>
>> This is the first time we release "cncf.kubernetes" package after big
>> backporting of kubernetes changes
>> to 1.10.11 and 1.10.12, so I'd appreciate, if some of our users looking
>> at the devlist test test this provider
>> - especially if you are waiting for it.
>>
>> Consider this my (binding) +1.
>>
>> Airflow Backport Providers 2020.10.5rc1 are available at:
>>
>> https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/
>>
>> *apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is a
>> source release that comes
>> with INSTALL instructions.
>>
>> *apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz*
>> are the binary
>> Python "sdist" release.
>>
>> Public keys are available at:
>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>
>> Please vote accordingly:
>>
>> [ ] +1 approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>>
>>
>> Only votes from PMC members are binding, but members of the community are
>> encouraged to test the release and vote with "(non-binding)".
>>
>> Please note that the version number excludes the 'rcX' string, so it's now
>> simply 2020.10.5. This will allow us to rename the artifact without
>> modifying
>> the artifact checksums when we actually release.
>>
>> Each of the packages contains a detailed changelog. Here is the list of
>> links to
>> the released packages and changelogs:
>>
>>
>> https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/
>>
>> https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/
>>
>>
>> Cheers,
>> Jarek
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
Re: [VOTE] Airflow Backport Providers 2020.10.5rc1
Posted by Kamil Breguła <ka...@polidea.com>.
I started testing and encountered three problems - Two minor and one
serious. I used official Docker images for
testing: apache/airflow:1.10.12 with gcloud installed for GKE authorization
1. The configuration for StackdriverTaskHandlle is not obvious and is not
documented anywhere.
I managed to do this when I created the /opt/airflow/config/log_config.py
file and set an environment variable
- AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG
File log_config.py:
https://gist.github.com/mik-laj/f94e9940bfd3f08cf69287ac62652bc5
However, I had more problems when I tried to use KubernetesPodOperator.
2. The following files are not included in the backport package
cnd.kubernetes:
- airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py,
- airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_spark_pi.yaml
I downloaded file /example_spark_kubernetes_spark_pi.yaml from our
repository and example DAG works
3. I haven't had such success with another example
- example_kubernetes_operator. I performed the following steps which ended
with an error.
$ curl -LO
https://raw.githubusercontent.com/apache/airflow/master/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py
$ cat example_kubernetes.py | sed
's/operators.bash/operators.bash_operator/g' | sed
's/in_cluster=True/in_cluster=False/g' > a.py
$ mv a.py example_kubernetes.py
$ airflow test example_kubernetes_operator write-xcom 2010-01-01
.....
[2020-10-04 23:29:06,623] {taskinstance.py:1150} ERROR - type object
'PodGenerator' has no attribute 'add_xcom_sidecar'
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py",
line 984, in _run_raw_task
result = task_copy.execute(context=context)
File
"/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 269, in execute
self.pod = self.create_pod_request_obj()
File
"/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 405, in create_pod_request_obj
pod = PodGenerator.add_xcom_sidecar(pod)
AttributeError: type object 'PodGenerator' has no attribute
'add_xcom_sidecar'
Have I missed any extra step I should do to run this example?
On Sun, Oct 4, 2020 at 7:11 PM Daniel Imberman <da...@gmail.com>
wrote:
> +1 excited to finally encourage the Kubernetes providers!
>
> via Newton Mail
> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2>
>
> On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> Hey all,
>
> I have cut Airflow Backport Providers 2020.10.5rc1. This email is calling
> a vote on the release,
> which will last for 72 hours - which means that it will end on Mon, 5th of
> October 2020, 19:59:56 CEST.
>
> This is the first time we release "cncf.kubernetes" package after big
> backporting of kubernetes changes
> to 1.10.11 and 1.10.12, so I'd appreciate, if some of our users looking at
> the devlist test test this provider
> - especially if you are waiting for it.
>
> Consider this my (binding) +1.
>
> Airflow Backport Providers 2020.10.5rc1 are available at:
>
> https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/
>
> *apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is a source
> release that comes
> with INSTALL instructions.
>
> *apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz* are
> the binary
> Python "sdist" release.
>
> Public keys are available at:
> https://dist.apache.org/repos/dist/release/airflow/KEYS
>
> Please vote accordingly:
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
>
> Only votes from PMC members are binding, but members of the community are
> encouraged to test the release and vote with "(non-binding)".
>
> Please note that the version number excludes the 'rcX' string, so it's now
> simply 2020.10.5. This will allow us to rename the artifact without
> modifying
> the artifact checksums when we actually release.
>
> Each of the packages contains a detailed changelog. Here is the list of
> links to
> the released packages and changelogs:
>
>
> https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/
>
> https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/
>
>
> Cheers,
> Jarek
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
Re: [VOTE] Airflow Backport Providers 2020.10.5rc1
Posted by Daniel Imberman <da...@gmail.com>.
+1 excited to finally encourage the Kubernetes providers!
via Newton Mail
[https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2]
On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk <Ja...@polidea.com>
wrote:
Hey all,
I have cut Airflow Backport Providers 2020.10.5rc1. This email is calling a
vote on the release,
which will last for 72 hours - which means that it will end on Mon, 5th of
October 2020, 19:59:56 CEST.
This is the first time we release "cncf.kubernetes" package after big
backporting of kubernetes changes to 1.10.11 and 1.10.12, so I'd
appreciate, if some of our users looking at the devlist test test this
provider - especially if you are waiting for it.
Consider this my (binding) +1.
Airflow Backport Providers 2020.10.5rc1 are available at:
https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/
[https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/]
*apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is a source
release that comes
with INSTALL instructions.
*apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz* are
the binary
Python "sdist" release.
Public keys are available at:
https://dist.apache.org/repos/dist/release/airflow/KEYS
[https://dist.apache.org/repos/dist/release/airflow/KEYS]
Please vote accordingly:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Only votes from PMC members are binding, but members of the community are
encouraged to test the release and vote with "(non-binding)".
Please note that the version number excludes the 'rcX' string, so it's now
simply 2020.10.5. This will allow us to rename the artifact without
modifying
the artifact checksums when we actually release.
Each of the packages contains a detailed changelog. Here is the list of
links to
the released packages and changelogs:
https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/]
https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/
[https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/]
Cheers,
Jarek
--
Jarek Potiuk
Polidea [https://www.polidea.com/] | Principal Software Engineer
M: +48 660 796 129 [tel:+48660796129]
[https://www.polidea.com/]