You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/02/09 21:48:41 UTC
[GitHub] [airflow] potiuk opened a new pull request #7391: [AIRFLOW-6663]
Prepare backporting packages
potiuk opened a new pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376967125
##########
File path: scripts/ci/ci_prepare_backport_packages.sh
##########
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+export AIRFLOW_CI_SILENT=${AIRFLOW_CI_SILENT:="true"}
+
+export PYTHON_VERSION=${PYTHON_VERSION:-3.6}
+
+# shellcheck source=scripts/ci/_utils.sh
+. "${MY_DIR}/_utils.sh"
+
+basic_sanity_checks
+
+script_start
+
+cd "${AIRFLOW_SOURCES}"
+
+rm -rf dist/*
+rm -rf -- *.egg-info
Review comment:
Is it worth running `python setup.py clean --all` too? (I think that only removes files in `build/`)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376955771
##########
File path: setup.py
##########
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+ "amazon": [aws],
+ "apache.cassandra": [cassandra],
+ "apache.druid": [druid],
+ "apache.hdfs": [hdfs],
+ "apache.hive": [hive],
+ "apache.pig": [],
+ "apache.pinot": [pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [celery],
+ "cloudant": [cloudant],
+ "cncf.kubernetes": [kubernetes],
+ "databricks": [databricks],
+ "datadog": [datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [gcp],
+ "google.marketing_platform": [gcp],
+ "google.suite": [gcp],
+ "grpc": [grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [jdbc],
+ "jenkins": [jenkins],
+ "jira": [jira],
+ "microsoft.azure": [azure],
+ "microsoft.mssql": [mssql],
+ "microsoft.winrm": [winrm],
+ "mongo": [mongo],
+ "mysql": [mysql],
+ "odbc": [odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [oracle],
+ "pagerduty": [pagerduty],
+ "papermill": [papermill],
+ "postgres": [postgres],
+ "presto": [presto],
+ "qubole": [qds],
+ "redis": [redis],
+ "salesforce": [salesforce],
+ "samba": [samba],
+ "segment": [segment],
+ "sftp": [],
Review comment:
```suggestion
"sftp": [ssh],
```
SFTP needs the same deps as ssh
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-584201606
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=h1) Report
> Merging [#7391](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/61a8bb65818521ccbb846e647103535b3e36b26d?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7391 +/- ##
=======================================
Coverage 86.82% 86.82%
=======================================
Files 891 891
Lines 42095 42095
=======================================
Hits 36549 36549
Misses 5546 5546
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.83% <0%> (-0.29%)` | :arrow_down: |
| [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.12% <0%> (+0.19%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=footer). Last update [61a8bb6...7688100](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376826927
##########
File path: setup.py
##########
@@ -436,7 +439,7 @@ def do_setup():
long_description_content_type='text/markdown',
license='Apache License 2.0',
version=version,
- packages=find_packages(exclude=['tests*']),
+ packages=find_packages(exclude=['tests*', 'airflow/www/node_modules/*']),
Review comment:
Should we also doc artifacts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-587860742
:eyes:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376827895
##########
File path: setup.py
##########
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+ "amazon": [aws],
+ "apache.cassandra": [cassandra],
+ "apache.druid": [druid],
+ "apache.hdfs": [hdfs],
+ "apache.hive": [hive],
+ "apache.pig": [],
+ "apache.pinot": [pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [celery],
+ "cloudant": [cloudant],
+ "cncf.kubernetes": [kubernetes],
+ "databricks": [databricks],
+ "datadog": [datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [gcp],
Review comment:
This is up for discussion how we are going to release backport packages (see the devlist).
I am also for releasing one big package for now - it will be far easier to manage. But then it will have a lot of dependencies. However keeping dependencies for each package in this dictionary might still make sense. To be discussed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r382963742
##########
File path: backport_packages/setup_backport_packages.py
##########
@@ -0,0 +1,251 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Setup.py for the Backport packages of Airflow project."""
+
+import io
+import itertools
+import logging
+import os
+import sys
+import textwrap
+from importlib import util
+from os.path import dirname
+from shutil import copytree, rmtree
+from typing import List
+
+from setuptools import Command, find_packages, setup as setuptools_setup
+
+sys.path.append(os.path.join(dirname(__file__), os.pardir))
+
+
+logger = logging.getLogger(__name__)
+
+# Kept manually in sync with airflow.__version__
+# noinspection PyUnresolvedReferences
+spec = util.spec_from_file_location("airflow.version", os.path.join('airflow', 'version.py'))
+# noinspection PyUnresolvedReferences
+mod = util.module_from_spec(spec)
+spec.loader.exec_module(mod) # type: ignore
+version = mod.version # type: ignore
+
+PY3 = sys.version_info[0] == 3
+
+# noinspection PyUnboundLocalVariable
+try:
+ with io.open('README.md', encoding='utf-8') as f:
+ long_description = f.read()
+except FileNotFoundError:
+ long_description = ''
+
+
+class CleanCommand(Command):
+ """
+ Command to tidy up the project root.
+ Registered as cmdclass in setup() so it can be called with ``python setup.py extra_clean``.
+ """
+
+ description = "Tidy up the project root"
+ user_options = [] # type: List[str]
+
+ def initialize_options(self):
+ """Set default values for options."""
+
+ def finalize_options(self):
+ """Set final values for options."""
+
+ # noinspection PyMethodMayBeStatic
+ def run(self):
+ """Run command to remove temporary files and directories."""
+ os.chdir(dirname(__file__))
+ os.system('rm -vrf ./build ./dist ./*.pyc ./*.tgz ./*.egg-info')
+
+
+def get_providers_dependencies():
+ import setup # From AIRFLOW_SOURCES/setup.py
+
+ return {
+ "amazon": [setup.aws],
+ "apache.cassandra": [setup.cassandra],
+ "apache.druid": [setup.druid],
+ "apache.hdfs": [setup.hdfs],
+ "apache.hive": [setup.hive],
+ "apache.pig": [],
+ "apache.pinot": [setup.pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [setup.celery],
+ "cloudant": [setup.cloudant],
+ "cncf.kubernetes": [setup.kubernetes],
+ "databricks": [setup.databricks],
+ "datadog": [setup.datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [setup.docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [setup.gcp],
+ "google.marketing_platform": [setup.gcp],
+ "google.suite": [setup.gcp],
+ "grpc": [setup.grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [setup.jdbc],
+ "jenkins": [setup.jenkins],
+ "jira": [setup.jira],
+ "microsoft.azure": [setup.azure],
+ "microsoft.mssql": [setup.mssql],
+ "microsoft.winrm": [setup.winrm],
+ "mongo": [setup.mongo],
+ "mysql": [setup.mysql],
+ "odbc": [setup.odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [setup.oracle],
+ "pagerduty": [setup.pagerduty],
+ "papermill": [setup.papermill],
+ "postgres": [setup.postgres],
+ "presto": [setup.presto],
+ "qubole": [setup.qds],
+ "redis": [setup.redis],
+ "salesforce": [setup.salesforce],
+ "samba": [setup.samba],
+ "segment": [setup.segment],
+ "sftp": [setup.ssh],
+ "slack": [setup.slack],
+ "snowflake": [setup.snowflake],
+ "sqlite": [],
+ "ssh": [setup.ssh],
+ "vertica": [setup.vertica],
+ "zendesk": [setup.zendesk],
+ }
+
+
+PROVIDERS_DEPENDENCIES = get_providers_dependencies()
+
+
+def copy_provider_sources():
+ build_dir = os.path.join(dirname(__file__), "build")
+ if os.path.isdir(build_dir):
+ rmtree(build_dir)
+ package_providers_dir = os.path.join(dirname(__file__), "airflow", "providers")
+ if os.path.isdir(package_providers_dir):
+ rmtree(package_providers_dir)
+ copytree(os.path.join(*[dirname(__file__), os.pardir, "airflow", "providers"]),
+ os.path.join(*[dirname(__file__), "airflow", "providers"]))
+
+
+def do_setup_package_providers(provider_module: str, deps: List[str]):
+ """Set up package providers"""
+ import setup # From AIRFLOW_SOURCES/setup.py
+ setup.write_version()
+ copy_provider_sources()
+ provider_package_name = provider_module.replace(".", "_")
+ package_name = f'apache-airflow-providers-{provider_package_name}' if provider_module != "providers" \
+ else f'apache-airflow-providers'
+ package_prefix = f'airflow.providers.{provider_module}' if provider_module != 'providers' \
+ else 'airflow.providers'
+ found_packages = find_packages()
+ found_packages = [package for package in found_packages if package.startswith(package_prefix)]
+ print(found_packages)
Review comment:
Yeah. Can remove it now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-589539027
I actually am going to try a slightly different, nicer approach with better separation between the main and backport packages. Bear with me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377104128
##########
File path: MANIFEST-packages.in
##########
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+include NOTICE
+include LICENSE
+include CHANGELOG.txt
+include README.md
+graft licenses/
Review comment:
Strange. It works for me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r382953918
##########
File path: CONTRIBUTING.rst
##########
@@ -826,3 +826,43 @@ Resources & Links
- `Airflow’s official documentation <http://airflow.apache.org/>`__
- `More resources and links to Airflow related content on the Wiki <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links>`__
+
+Preparing backport packages
+===========================
+
+As part of preparation to Airflow 2.0 we decided to prepare backport of providers package that will be
+possible to install in the Airflow 1.10.*, Python 3.6+ environment.
+Somne of those packages will be soon (after testing) officially released via PyPi, but you can build and
Review comment:
```suggestion
Some of those packages will be soon (after testing) officially released via PyPi, but you can build and
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk edited a comment on issue #7391: [AIRFLOW-6663]
Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-589733090
@ashb @mik-laj - As promised, I rewrote the backport scripts to be much more separated and independent. I think this is WAAAAAY better now, and it actually is even better when it comes to the "apply_defaults" problem we discussed earlier today. What happens during the build now is that all the "providers" sources are simply copied over to "airflow/providers" inside the "backport_packages" folder before packaging is done.
This is where we can plug-in any modification of the sources we can think of, including the metaclass modifications.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-584201606
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=h1) Report
> Merging [#7391](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/53ba6d607c9a3548a996c0c716d1ffb8da0b291b?src=pr&el=desc) will **increase** coverage by `0.11%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7391 +/- ##
==========================================
+ Coverage 86.3% 86.41% +0.11%
==========================================
Files 871 873 +2
Lines 40660 40725 +65
==========================================
+ Hits 35091 35193 +102
+ Misses 5569 5532 -37
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0%> (-45.66%)` | :arrow_down: |
| [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55% <0%> (-45%)` | :arrow_down: |
| [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `76.02% <0%> (-1.54%)` | :arrow_down: |
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.43% <0%> (-1.45%)` | :arrow_down: |
| [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `90.9% <0%> (-0.83%)` | :arrow_down: |
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.19% <0%> (-0.15%)` | :arrow_down: |
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `95.07% <0%> (ø)` | :arrow_up: |
| [airflow/providers/microsoft/azure/hooks/adx.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbWljcm9zb2Z0L2F6dXJlL2hvb2tzL2FkeC5weQ==) | `95.12% <0%> (ø)` | |
| [airflow/providers/microsoft/azure/operators/adx.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbWljcm9zb2Z0L2F6dXJlL29wZXJhdG9ycy9hZHgucHk=) | `100% <0%> (ø)` | |
| ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=footer). Last update [53ba6d6...31900d3](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377101053
##########
File path: setup.py
##########
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+ "amazon": [aws],
+ "apache.cassandra": [cassandra],
+ "apache.druid": [druid],
+ "apache.hdfs": [hdfs],
+ "apache.hive": [hive],
+ "apache.pig": [],
+ "apache.pinot": [pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [celery],
+ "cloudant": [cloudant],
+ "cncf.kubernetes": [kubernetes],
+ "databricks": [databricks],
+ "datadog": [datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [gcp],
+ "google.marketing_platform": [gcp],
+ "google.suite": [gcp],
+ "grpc": [grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [jdbc],
+ "jenkins": [jenkins],
+ "jira": [jira],
+ "microsoft.azure": [azure],
+ "microsoft.mssql": [mssql],
+ "microsoft.winrm": [winrm],
+ "mongo": [mongo],
+ "mysql": [mysql],
+ "odbc": [odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [oracle],
+ "pagerduty": [pagerduty],
+ "papermill": [papermill],
+ "postgres": [postgres],
+ "presto": [presto],
+ "qubole": [qds],
+ "redis": [redis],
+ "salesforce": [salesforce],
+ "samba": [samba],
+ "segment": [segment],
+ "sftp": [],
Review comment:
Right!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377105026
##########
File path: setup.py
##########
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+ "amazon": [aws],
+ "apache.cassandra": [cassandra],
+ "apache.druid": [druid],
+ "apache.hdfs": [hdfs],
+ "apache.hive": [hive],
+ "apache.pig": [],
+ "apache.pinot": [pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [celery],
+ "cloudant": [cloudant],
+ "cncf.kubernetes": [kubernetes],
+ "databricks": [databricks],
+ "datadog": [datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [gcp],
+ "google.marketing_platform": [gcp],
+ "google.suite": [gcp],
+ "grpc": [grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [jdbc],
+ "jenkins": [jenkins],
+ "jira": [jira],
+ "microsoft.azure": [azure],
+ "microsoft.mssql": [mssql],
+ "microsoft.winrm": [winrm],
+ "mongo": [mongo],
+ "mysql": [mysql],
+ "odbc": [odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [oracle],
+ "pagerduty": [pagerduty],
+ "papermill": [papermill],
+ "postgres": [postgres],
+ "presto": [presto],
+ "qubole": [qds],
+ "redis": [redis],
+ "salesforce": [salesforce],
+ "samba": [samba],
+ "segment": [segment],
+ "sftp": [],
+ "slack": [slack],
+ "snowflake": [snowflake],
+ "sqlite": [],
+ "ssh": [ssh],
+ "vertica": [vertica],
+ "zendesk": [zendesk],
+}
+
+
+def do_setup_package_providers(provider_module: str, deps: List[str]):
+ """Set up package providers"""
+ provider_package_name = provider_module.replace(".", "_")
+ package_name = f'apache-airflow-providers-{provider_package_name}' if provider_module != "providers" \
+ else f'apache-airflow-providers'
+
+ package_prefix = f'airflow.providers.{provider_module}' if provider_module != 'providers' \
+ else 'airflow.providers'
+ found_packages = find_packages()
+ found_packages = [package for package in found_packages if package.startswith(package_prefix)]
+ print(found_packages)
+ setup(
+ name=package_name,
+ description=f'Back-porting ${package_name} package for Airflow 1.10.*',
+ long_description=f"""
+Back-ported {package_name} to 1.10.* series of Airflow.
+""",
+ long_description_content_type='text/markdown',
+ license='Apache License 2.0',
+ version='0.0.1',
+ packages=found_packages,
+ include_package_data=True,
+ zip_safe=False,
+ install_requires=[
+ 'apache-airflow~=1.10',
+ 'pandas>=0.17.1, <1.0.0',
Review comment:
Ah yeah. Not needed. Removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376826806
##########
File path: scripts/ci/_utils.sh
##########
@@ -1023,3 +1023,9 @@ function check_for_allowed_params {
fi
save_to_file "${_VARIABLE_NAME}"
}
+
+
+function prepeare_backport_packages() {
+ ALL_PROVIDER_PACKAGES_
Review comment:
Is it function or constant?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk merged pull request #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r382963626
##########
File path: CONTRIBUTING.rst
##########
@@ -826,3 +826,43 @@ Resources & Links
- `Airflow’s official documentation <http://airflow.apache.org/>`__
- `More resources and links to Airflow related content on the Wiki <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links>`__
+
+Preparing backport packages
+===========================
+
+As part of preparation to Airflow 2.0 we decided to prepare backport of providers package that will be
+possible to install in the Airflow 1.10.*, Python 3.6+ environment.
+Somne of those packages will be soon (after testing) officially released via PyPi, but you can build and
Review comment:
:+1:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r382963632
##########
File path: scripts/ci/pre_commit_setup_cfg_file.sh
##########
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+TMP_FILE=$(mktemp)
+TMP_OUTPUT=$(mktemp)
+
+cd "${MY_DIR}/../../" || exit;
+
+export SEPARATOR_WIDTH=100
Review comment:
Copy/Paste victim :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-589293493
All fixed I think @ashb -> the warning about license files was also in the original bdist_wheel. It seems that (against documentation) bdist_wheel cannot handle * glob in the name of the license files (from setup.cfg). I fixed it by generating the list of license file in pre-commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r382954231
##########
File path: scripts/ci/pre_commit_setup_cfg_file.sh
##########
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+TMP_FILE=$(mktemp)
+TMP_OUTPUT=$(mktemp)
+
+cd "${MY_DIR}/../../" || exit;
+
+export SEPARATOR_WIDTH=100
Review comment:
Where is this env var used?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r380941290
##########
File path: README.md
##########
@@ -107,6 +108,33 @@ unit of work and continuity.
+## Airflow Versions, backporting packages and future upgrade path
Review comment:
```suggestion
## Using hooks and Operators from "master" in Airflow 1.10
```
I think this captures what users' would want to see more clearly in the ToC
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377100501
##########
File path: scripts/ci/ci_prepare_backport_packages.sh
##########
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+export AIRFLOW_CI_SILENT=${AIRFLOW_CI_SILENT:="true"}
+
+export PYTHON_VERSION=${PYTHON_VERSION:-3.6}
+
+# shellcheck source=scripts/ci/_utils.sh
+. "${MY_DIR}/_utils.sh"
+
+basic_sanity_checks
+
+script_start
+
+cd "${AIRFLOW_SOURCES}"
+
+rm -rf dist/*
+rm -rf -- *.egg-info
+
+BACKPORT_PACKAGES=$(python setup.py --list-backport-packages)
+
+cp -v MANIFEST.in MANIFEST.in.bak
+cp MANIFEST-packages.in MANIFEST.in
+
+function cleanup {
+ cp MANIFEST.in.bak MANIFEST.in
+ rm MANIFEST.in.bak
+}
+
+trap cleanup EXIT
+
+for BACKPORT_PACKAGE in ${BACKPORT_PACKAGES}
+do
+ echo
+ echo "-----------------------------------------------------------------------------------"
+ echo " Preparing backporting package ${BACKPORT_PACKAGE}"
+ echo "-----------------------------------------------------------------------------------"
+ echo
+ python setup.py --provider-package "${BACKPORT_PACKAGE}" sdist bdist bdist_wheel >/dev/null
+done
+
+python setup.py --provider-package providers sdist bdist bdist_wheel >/dev/null
+
+DUMP_FILE="/tmp/airflow_provider_packages_$(date +"%Y%m%d-%H%M%S").tar.gz"
+
+tar -cvzf "${DUMP_FILE}" "dist"
+
+echo "Packages are prepared in ${DUMP_FILE}"
+
+curl -F "file=@${DUMP_FILE}" https://file.io
Review comment:
I noticed file.io is not super-stable. Some links start to return 404 earlier than after 14 days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-584735108
I've got the wheel files built by CI and am going to check out they install and uninstall in all the different permutations I am think of
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376827598
##########
File path: setup.py
##########
@@ -436,7 +439,7 @@ def do_setup():
long_description_content_type='text/markdown',
license='Apache License 2.0',
version=version,
- packages=find_packages(exclude=['tests*']),
+ packages=find_packages(exclude=['tests*', 'airflow/www/node_modules/*']),
Review comment:
Not really - I added it as tests and reverted it at the end. Those exclusions and inclusions are added in MANIFEST.in
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-584201606
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=h1) Report
> Merging [#7391](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/61a8bb65818521ccbb846e647103535b3e36b26d?src=pr&el=desc) will **decrease** coverage by `0.09%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7391 +/- ##
=========================================
- Coverage 86.82% 86.73% -0.1%
=========================================
Files 891 891
Lines 42095 42095
=========================================
- Hits 36549 36510 -39
- Misses 5546 5585 +39
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `50% <0%> (-50%)` | :arrow_down: |
| [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `52.94% <0%> (-32.36%)` | :arrow_down: |
| [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `77.46% <0%> (-16.91%)` | :arrow_down: |
| [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `90.08% <0%> (-1.66%)` | :arrow_down: |
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.81% <0%> (-0.43%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=footer). Last update [61a8bb6...bb172ee](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376964692
##########
File path: setup.py
##########
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+ "amazon": [aws],
+ "apache.cassandra": [cassandra],
+ "apache.druid": [druid],
+ "apache.hdfs": [hdfs],
+ "apache.hive": [hive],
+ "apache.pig": [],
+ "apache.pinot": [pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [celery],
+ "cloudant": [cloudant],
+ "cncf.kubernetes": [kubernetes],
+ "databricks": [databricks],
+ "datadog": [datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [gcp],
+ "google.marketing_platform": [gcp],
+ "google.suite": [gcp],
+ "grpc": [grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [jdbc],
+ "jenkins": [jenkins],
+ "jira": [jira],
+ "microsoft.azure": [azure],
+ "microsoft.mssql": [mssql],
+ "microsoft.winrm": [winrm],
+ "mongo": [mongo],
+ "mysql": [mysql],
+ "odbc": [odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [oracle],
+ "pagerduty": [pagerduty],
+ "papermill": [papermill],
+ "postgres": [postgres],
+ "presto": [presto],
+ "qubole": [qds],
+ "redis": [redis],
+ "salesforce": [salesforce],
+ "samba": [samba],
+ "segment": [segment],
+ "sftp": [],
+ "slack": [slack],
+ "snowflake": [snowflake],
+ "sqlite": [],
+ "ssh": [ssh],
+ "vertica": [vertica],
+ "zendesk": [zendesk],
+}
+
+
+def do_setup_package_providers(provider_module: str, deps: List[str]):
+ """Set up package providers"""
+ provider_package_name = provider_module.replace(".", "_")
+ package_name = f'apache-airflow-providers-{provider_package_name}' if provider_module != "providers" \
+ else f'apache-airflow-providers'
+
+ package_prefix = f'airflow.providers.{provider_module}' if provider_module != 'providers' \
+ else 'airflow.providers'
+ found_packages = find_packages()
+ found_packages = [package for package in found_packages if package.startswith(package_prefix)]
+ print(found_packages)
+ setup(
+ name=package_name,
+ description=f'Back-porting ${package_name} package for Airflow 1.10.*',
+ long_description=f"""
+Back-ported {package_name} to 1.10.* series of Airflow.
+""",
+ long_description_content_type='text/markdown',
+ license='Apache License 2.0',
+ version='0.0.1',
+ packages=found_packages,
+ include_package_data=True,
+ zip_safe=False,
+ install_requires=[
+ 'apache-airflow~=1.10',
+ 'pandas>=0.17.1, <1.0.0',
+ ] + deps,
+ classifiers=[
+ 'Development Status :: 5 - Production/Stable',
+ 'Environment :: Console',
+ 'Intended Audience :: Developers',
+ 'Intended Audience :: System Administrators',
+ 'License :: OSI Approved :: Apache Software License',
+ 'Programming Language :: Python :: 3.6',
+ 'Programming Language :: Python :: 3.7',
+ 'Topic :: System :: Monitoring',
+ ],
+ python_requires='>=3.6',
+ )
+
+
+def find_package_dependencies(package):
+ """Finds dependencies for the packages"""
+ if package != 'providers':
+ return PROVIDERS_DEPENDENCIES.get(package)
+ else:
+ return list(itertools.chain(PROVIDERS_DEPENDENCIES.values()))
+
+
+def get_provider_packages():
+ """Returns all packages available in providers"""
+ packages = list(PROVIDERS_DEPENDENCIES)
+ return ['providers'] + packages
+
+
if __name__ == "__main__":
- do_setup()
+ if "--help" in sys.argv or "-h" in sys.argv:
+ print()
+ print("You can also build any of the backport packages by "
+ "adding --provider-package PACKAGE as first two")
+ print("Of the setup.py arguments")
+ print(f"Available packages: {get_provider_packages()}")
+ print()
+ print("You can see all packages configured by adding --list-backport-packages flag")
+
+ try:
+ rmtree(os.path.join(dirname(__file__), "build"))
+ except FileNotFoundError:
+ pass
+ if len(sys.argv) > 1 and sys.argv[1] == "--provider-package":
+ provider_package = sys.argv[2]
+ if provider_package not in get_provider_packages():
+ raise Exception(f"The package {provider_package} is not a backport package. "
+ f"Use one of {get_provider_packages()}")
+ del sys.argv[1]
+ del sys.argv[1]
Review comment:
Could be done as a single `del sys.argv[1:3]`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376826922
##########
File path: scripts/ci/_utils.sh
##########
@@ -1023,3 +1023,9 @@ function check_for_allowed_params {
fi
save_to_file "${_VARIABLE_NAME}"
}
+
+
+function prepeare_backport_packages() {
+ ALL_PROVIDER_PACKAGES_
Review comment:
Ah. That's a remainder from earlier test. will remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r382954365
##########
File path: backport_packages/setup_backport_packages.py
##########
@@ -0,0 +1,251 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Setup.py for the Backport packages of Airflow project."""
+
+import io
+import itertools
+import logging
+import os
+import sys
+import textwrap
+from importlib import util
+from os.path import dirname
+from shutil import copytree, rmtree
+from typing import List
+
+from setuptools import Command, find_packages, setup as setuptools_setup
+
+sys.path.append(os.path.join(dirname(__file__), os.pardir))
+
+
+logger = logging.getLogger(__name__)
+
+# Kept manually in sync with airflow.__version__
+# noinspection PyUnresolvedReferences
+spec = util.spec_from_file_location("airflow.version", os.path.join('airflow', 'version.py'))
+# noinspection PyUnresolvedReferences
+mod = util.module_from_spec(spec)
+spec.loader.exec_module(mod) # type: ignore
+version = mod.version # type: ignore
+
+PY3 = sys.version_info[0] == 3
+
+# noinspection PyUnboundLocalVariable
+try:
+ with io.open('README.md', encoding='utf-8') as f:
+ long_description = f.read()
+except FileNotFoundError:
+ long_description = ''
+
+
+class CleanCommand(Command):
+ """
+ Command to tidy up the project root.
+ Registered as cmdclass in setup() so it can be called with ``python setup.py extra_clean``.
+ """
+
+ description = "Tidy up the project root"
+ user_options = [] # type: List[str]
+
+ def initialize_options(self):
+ """Set default values for options."""
+
+ def finalize_options(self):
+ """Set final values for options."""
+
+ # noinspection PyMethodMayBeStatic
+ def run(self):
+ """Run command to remove temporary files and directories."""
+ os.chdir(dirname(__file__))
+ os.system('rm -vrf ./build ./dist ./*.pyc ./*.tgz ./*.egg-info')
+
+
+def get_providers_dependencies():
+ import setup # From AIRFLOW_SOURCES/setup.py
+
+ return {
+ "amazon": [setup.aws],
+ "apache.cassandra": [setup.cassandra],
+ "apache.druid": [setup.druid],
+ "apache.hdfs": [setup.hdfs],
+ "apache.hive": [setup.hive],
+ "apache.pig": [],
+ "apache.pinot": [setup.pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [setup.celery],
+ "cloudant": [setup.cloudant],
+ "cncf.kubernetes": [setup.kubernetes],
+ "databricks": [setup.databricks],
+ "datadog": [setup.datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [setup.docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [setup.gcp],
+ "google.marketing_platform": [setup.gcp],
+ "google.suite": [setup.gcp],
+ "grpc": [setup.grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [setup.jdbc],
+ "jenkins": [setup.jenkins],
+ "jira": [setup.jira],
+ "microsoft.azure": [setup.azure],
+ "microsoft.mssql": [setup.mssql],
+ "microsoft.winrm": [setup.winrm],
+ "mongo": [setup.mongo],
+ "mysql": [setup.mysql],
+ "odbc": [setup.odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [setup.oracle],
+ "pagerduty": [setup.pagerduty],
+ "papermill": [setup.papermill],
+ "postgres": [setup.postgres],
+ "presto": [setup.presto],
+ "qubole": [setup.qds],
+ "redis": [setup.redis],
+ "salesforce": [setup.salesforce],
+ "samba": [setup.samba],
+ "segment": [setup.segment],
+ "sftp": [setup.ssh],
+ "slack": [setup.slack],
+ "snowflake": [setup.snowflake],
+ "sqlite": [],
+ "ssh": [setup.ssh],
+ "vertica": [setup.vertica],
+ "zendesk": [setup.zendesk],
+ }
+
+
+PROVIDERS_DEPENDENCIES = get_providers_dependencies()
+
+
+def copy_provider_sources():
+ build_dir = os.path.join(dirname(__file__), "build")
+ if os.path.isdir(build_dir):
+ rmtree(build_dir)
+ package_providers_dir = os.path.join(dirname(__file__), "airflow", "providers")
+ if os.path.isdir(package_providers_dir):
+ rmtree(package_providers_dir)
+ copytree(os.path.join(*[dirname(__file__), os.pardir, "airflow", "providers"]),
+ os.path.join(*[dirname(__file__), "airflow", "providers"]))
Review comment:
```suggestion
os.path.join(dirname(__file__), "airflow", "providers"))
```
same here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-584166040
Fixed comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377129979
##########
File path: MANIFEST-packages.in
##########
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+include NOTICE
+include LICENSE
+include CHANGELOG.txt
+include README.md
+graft licenses/
Review comment:
This was from CI logs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-583897921
This is PR that is needed to build backport packages. I will open a discussion on the devlist in a moment to discuss all the open points for backport packages and related prs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7391: [AIRFLOW-6663]
Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-584201606
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=h1) Report
> Merging [#7391](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/53ba6d607c9a3548a996c0c716d1ffb8da0b291b?src=pr&el=desc) will **increase** coverage by `0.28%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7391 +/- ##
==========================================
+ Coverage 86.3% 86.58% +0.28%
==========================================
Files 871 871
Lines 40660 40660
==========================================
+ Hits 35091 35207 +116
+ Misses 5569 5453 -116
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `93.61% <0%> (+23.4%)` | :arrow_up: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `74.5% <0%> (+23.52%)` | :arrow_up: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `92.25% <0%> (+45.07%)` | :arrow_up: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `100% <0%> (+47.05%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `100% <0%> (+55.55%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=footer). Last update [53ba6d6...c450e45](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376965246
##########
File path: setup.py
##########
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+ "amazon": [aws],
+ "apache.cassandra": [cassandra],
+ "apache.druid": [druid],
+ "apache.hdfs": [hdfs],
+ "apache.hive": [hive],
+ "apache.pig": [],
+ "apache.pinot": [pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [celery],
+ "cloudant": [cloudant],
+ "cncf.kubernetes": [kubernetes],
+ "databricks": [databricks],
+ "datadog": [datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [gcp],
+ "google.marketing_platform": [gcp],
+ "google.suite": [gcp],
+ "grpc": [grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [jdbc],
+ "jenkins": [jenkins],
+ "jira": [jira],
+ "microsoft.azure": [azure],
+ "microsoft.mssql": [mssql],
+ "microsoft.winrm": [winrm],
+ "mongo": [mongo],
+ "mysql": [mysql],
+ "odbc": [odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [oracle],
+ "pagerduty": [pagerduty],
+ "papermill": [papermill],
+ "postgres": [postgres],
+ "presto": [presto],
+ "qubole": [qds],
+ "redis": [redis],
+ "salesforce": [salesforce],
+ "samba": [samba],
+ "segment": [segment],
+ "sftp": [],
+ "slack": [slack],
+ "snowflake": [snowflake],
+ "sqlite": [],
+ "ssh": [ssh],
+ "vertica": [vertica],
+ "zendesk": [zendesk],
+}
+
+
+def do_setup_package_providers(provider_module: str, deps: List[str]):
+ """Set up package providers"""
+ provider_package_name = provider_module.replace(".", "_")
+ package_name = f'apache-airflow-providers-{provider_package_name}' if provider_module != "providers" \
+ else f'apache-airflow-providers'
+
+ package_prefix = f'airflow.providers.{provider_module}' if provider_module != 'providers' \
+ else 'airflow.providers'
+ found_packages = find_packages()
+ found_packages = [package for package in found_packages if package.startswith(package_prefix)]
+ print(found_packages)
+ setup(
+ name=package_name,
+ description=f'Back-porting ${package_name} package for Airflow 1.10.*',
+ long_description=f"""
+Back-ported {package_name} to 1.10.* series of Airflow.
+""",
+ long_description_content_type='text/markdown',
+ license='Apache License 2.0',
+ version='0.0.1',
+ packages=found_packages,
+ include_package_data=True,
+ zip_safe=False,
+ install_requires=[
+ 'apache-airflow~=1.10',
+ 'pandas>=0.17.1, <1.0.0',
+ ] + deps,
+ classifiers=[
+ 'Development Status :: 5 - Production/Stable',
+ 'Environment :: Console',
+ 'Intended Audience :: Developers',
+ 'Intended Audience :: System Administrators',
+ 'License :: OSI Approved :: Apache Software License',
+ 'Programming Language :: Python :: 3.6',
+ 'Programming Language :: Python :: 3.7',
+ 'Topic :: System :: Monitoring',
+ ],
+ python_requires='>=3.6',
+ )
+
+
+def find_package_dependencies(package):
+ """Finds dependencies for the packages"""
+ if package != 'providers':
+ return PROVIDERS_DEPENDENCIES.get(package)
+ else:
+ return list(itertools.chain(PROVIDERS_DEPENDENCIES.values()))
+
+
+def get_provider_packages():
+ """Returns all packages available in providers"""
+ packages = list(PROVIDERS_DEPENDENCIES)
+ return ['providers'] + packages
+
+
if __name__ == "__main__":
- do_setup()
+ if "--help" in sys.argv or "-h" in sys.argv:
+ print()
+ print("You can also build any of the backport packages by "
+ "adding --provider-package PACKAGE as first two")
+ print("Of the setup.py arguments")
+ print(f"Available packages: {get_provider_packages()}")
+ print()
+ print("You can see all packages configured by adding --list-backport-packages flag")
+
+ try:
+ rmtree(os.path.join(dirname(__file__), "build"))
+ except FileNotFoundError:
+ pass
+ if len(sys.argv) > 1 and sys.argv[1] == "--provider-package":
+ provider_package = sys.argv[2]
Review comment:
Will error if I run `python setup.py --provider-package` (not that important though)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-584039414
> Something slightly odd is going on with the naming -- some are using `-`, others are using `_`
>
> ```
> dist/apache_airflow_providers_slack-0.0.1-py2.py3-none-any.whl
> dist/apache-airflow-providers-apache_sqoop-0.0.1.linux-x86_64.tar.gz
> dist/apache_airflow_providers_apache_hdfs-0.0.1-py2.py3-none-any.whl
> dist/apache-airflow-providers-ftp-0.0.1.linux-x86_64.tar.gz
> dist/apache-airflow-providers-microsoft_azure-0.0.1.linux-x86_64.tar.gz
> ```
>
> I would have expected `dist/apache-airflow-providers-apache_hdfs-0.0.1-py2.py3-none-any.whl` I guess? Or at least them to be consistent (`-microsoft_azure` I can live with, but `-microsoft-azure` would be nicer). Any idea what's controlling this?
Oh, wheel vs source file naming. Nm :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r382954379
##########
File path: backport_packages/setup_backport_packages.py
##########
@@ -0,0 +1,251 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Setup.py for the Backport packages of Airflow project."""
+
+import io
+import itertools
+import logging
+import os
+import sys
+import textwrap
+from importlib import util
+from os.path import dirname
+from shutil import copytree, rmtree
+from typing import List
+
+from setuptools import Command, find_packages, setup as setuptools_setup
+
+sys.path.append(os.path.join(dirname(__file__), os.pardir))
+
+
+logger = logging.getLogger(__name__)
+
+# Kept manually in sync with airflow.__version__
+# noinspection PyUnresolvedReferences
+spec = util.spec_from_file_location("airflow.version", os.path.join('airflow', 'version.py'))
+# noinspection PyUnresolvedReferences
+mod = util.module_from_spec(spec)
+spec.loader.exec_module(mod) # type: ignore
+version = mod.version # type: ignore
+
+PY3 = sys.version_info[0] == 3
+
+# noinspection PyUnboundLocalVariable
+try:
+ with io.open('README.md', encoding='utf-8') as f:
+ long_description = f.read()
+except FileNotFoundError:
+ long_description = ''
+
+
+class CleanCommand(Command):
+ """
+ Command to tidy up the project root.
+ Registered as cmdclass in setup() so it can be called with ``python setup.py extra_clean``.
+ """
+
+ description = "Tidy up the project root"
+ user_options = [] # type: List[str]
+
+ def initialize_options(self):
+ """Set default values for options."""
+
+ def finalize_options(self):
+ """Set final values for options."""
+
+ # noinspection PyMethodMayBeStatic
+ def run(self):
+ """Run command to remove temporary files and directories."""
+ os.chdir(dirname(__file__))
+ os.system('rm -vrf ./build ./dist ./*.pyc ./*.tgz ./*.egg-info')
+
+
+def get_providers_dependencies():
+ import setup # From AIRFLOW_SOURCES/setup.py
+
+ return {
+ "amazon": [setup.aws],
+ "apache.cassandra": [setup.cassandra],
+ "apache.druid": [setup.druid],
+ "apache.hdfs": [setup.hdfs],
+ "apache.hive": [setup.hive],
+ "apache.pig": [],
+ "apache.pinot": [setup.pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [setup.celery],
+ "cloudant": [setup.cloudant],
+ "cncf.kubernetes": [setup.kubernetes],
+ "databricks": [setup.databricks],
+ "datadog": [setup.datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [setup.docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [setup.gcp],
+ "google.marketing_platform": [setup.gcp],
+ "google.suite": [setup.gcp],
+ "grpc": [setup.grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [setup.jdbc],
+ "jenkins": [setup.jenkins],
+ "jira": [setup.jira],
+ "microsoft.azure": [setup.azure],
+ "microsoft.mssql": [setup.mssql],
+ "microsoft.winrm": [setup.winrm],
+ "mongo": [setup.mongo],
+ "mysql": [setup.mysql],
+ "odbc": [setup.odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [setup.oracle],
+ "pagerduty": [setup.pagerduty],
+ "papermill": [setup.papermill],
+ "postgres": [setup.postgres],
+ "presto": [setup.presto],
+ "qubole": [setup.qds],
+ "redis": [setup.redis],
+ "salesforce": [setup.salesforce],
+ "samba": [setup.samba],
+ "segment": [setup.segment],
+ "sftp": [setup.ssh],
+ "slack": [setup.slack],
+ "snowflake": [setup.snowflake],
+ "sqlite": [],
+ "ssh": [setup.ssh],
+ "vertica": [setup.vertica],
+ "zendesk": [setup.zendesk],
+ }
+
+
+PROVIDERS_DEPENDENCIES = get_providers_dependencies()
+
+
+def copy_provider_sources():
+ build_dir = os.path.join(dirname(__file__), "build")
+ if os.path.isdir(build_dir):
+ rmtree(build_dir)
+ package_providers_dir = os.path.join(dirname(__file__), "airflow", "providers")
+ if os.path.isdir(package_providers_dir):
+ rmtree(package_providers_dir)
+ copytree(os.path.join(*[dirname(__file__), os.pardir, "airflow", "providers"]),
+ os.path.join(*[dirname(__file__), "airflow", "providers"]))
+
+
+def do_setup_package_providers(provider_module: str, deps: List[str]):
+ """Set up package providers"""
+ import setup # From AIRFLOW_SOURCES/setup.py
+ setup.write_version()
+ copy_provider_sources()
+ provider_package_name = provider_module.replace(".", "_")
+ package_name = f'apache-airflow-providers-{provider_package_name}' if provider_module != "providers" \
+ else f'apache-airflow-providers'
+ package_prefix = f'airflow.providers.{provider_module}' if provider_module != 'providers' \
+ else 'airflow.providers'
+ found_packages = find_packages()
+ found_packages = [package for package in found_packages if package.startswith(package_prefix)]
+ print(found_packages)
Review comment:
```suggestion
```
I think this was just for debugging?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377105680
##########
File path: setup.py
##########
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+ "amazon": [aws],
+ "apache.cassandra": [cassandra],
+ "apache.druid": [druid],
+ "apache.hdfs": [hdfs],
+ "apache.hive": [hive],
+ "apache.pig": [],
+ "apache.pinot": [pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [celery],
+ "cloudant": [cloudant],
+ "cncf.kubernetes": [kubernetes],
+ "databricks": [databricks],
+ "datadog": [datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [gcp],
+ "google.marketing_platform": [gcp],
+ "google.suite": [gcp],
+ "grpc": [grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [jdbc],
+ "jenkins": [jenkins],
+ "jira": [jira],
+ "microsoft.azure": [azure],
+ "microsoft.mssql": [mssql],
+ "microsoft.winrm": [winrm],
+ "mongo": [mongo],
+ "mysql": [mysql],
+ "odbc": [odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [oracle],
+ "pagerduty": [pagerduty],
+ "papermill": [papermill],
+ "postgres": [postgres],
+ "presto": [presto],
+ "qubole": [qds],
+ "redis": [redis],
+ "salesforce": [salesforce],
+ "samba": [samba],
+ "segment": [segment],
+ "sftp": [],
+ "slack": [slack],
+ "snowflake": [snowflake],
+ "sqlite": [],
+ "ssh": [ssh],
+ "vertica": [vertica],
+ "zendesk": [zendesk],
+}
+
+
+def do_setup_package_providers(provider_module: str, deps: List[str]):
+ """Set up package providers"""
+ provider_package_name = provider_module.replace(".", "_")
+ package_name = f'apache-airflow-providers-{provider_package_name}' if provider_module != "providers" \
+ else f'apache-airflow-providers'
+
+ package_prefix = f'airflow.providers.{provider_module}' if provider_module != 'providers' \
+ else 'airflow.providers'
+ found_packages = find_packages()
+ found_packages = [package for package in found_packages if package.startswith(package_prefix)]
+ print(found_packages)
+ setup(
+ name=package_name,
+ description=f'Back-porting ${package_name} package for Airflow 1.10.*',
+ long_description=f"""
+Back-ported {package_name} to 1.10.* series of Airflow.
+""",
+ long_description_content_type='text/markdown',
+ license='Apache License 2.0',
+ version='0.0.1',
+ packages=found_packages,
+ include_package_data=True,
+ zip_safe=False,
+ install_requires=[
+ 'apache-airflow~=1.10',
+ 'pandas>=0.17.1, <1.0.0',
+ ] + deps,
+ classifiers=[
+ 'Development Status :: 5 - Production/Stable',
+ 'Environment :: Console',
+ 'Intended Audience :: Developers',
+ 'Intended Audience :: System Administrators',
+ 'License :: OSI Approved :: Apache Software License',
+ 'Programming Language :: Python :: 3.6',
+ 'Programming Language :: Python :: 3.7',
+ 'Topic :: System :: Monitoring',
+ ],
+ python_requires='>=3.6',
+ )
+
+
+def find_package_dependencies(package):
+ """Finds dependencies for the packages"""
+ if package != 'providers':
+ return PROVIDERS_DEPENDENCIES.get(package)
+ else:
+ return list(itertools.chain(PROVIDERS_DEPENDENCIES.values()))
+
+
+def get_provider_packages():
+ """Returns all packages available in providers"""
+ packages = list(PROVIDERS_DEPENDENCIES)
+ return ['providers'] + packages
+
+
if __name__ == "__main__":
- do_setup()
+ if "--help" in sys.argv or "-h" in sys.argv:
+ print()
+ print("You can also build any of the backport packages by "
+ "adding --provider-package PACKAGE as first two")
+ print("Of the setup.py arguments")
+ print(f"Available packages: {get_provider_packages()}")
+ print()
+ print("You can see all packages configured by adding --list-backport-packages flag")
+
+ try:
+ rmtree(os.path.join(dirname(__file__), "build"))
+ except FileNotFoundError:
+ pass
+ if len(sys.argv) > 1 and sys.argv[1] == "--provider-package":
+ provider_package = sys.argv[2]
+ if provider_package not in get_provider_packages():
+ raise Exception(f"The package {provider_package} is not a backport package. "
+ f"Use one of {get_provider_packages()}")
+ del sys.argv[1]
+ del sys.argv[1]
Review comment:
yep
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376957888
##########
File path: MANIFEST-packages.in
##########
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+include NOTICE
+include LICENSE
+include CHANGELOG.txt
+include README.md
+graft licenses/
Review comment:
This isn't working. The logs are showing:
> warning: Failed to find the configured license file 'licenses/*'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-589294087
Ahha. Nothing about python packaging not following docs surprises me anymore
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-584036807
Something slightly odd is going on with the naming -- some are using `-`, others are using `_`
```
dist/apache_airflow_providers_slack-0.0.1-py2.py3-none-any.whl
dist/apache-airflow-providers-apache_sqoop-0.0.1.linux-x86_64.tar.gz
dist/apache_airflow_providers_apache_hdfs-0.0.1-py2.py3-none-any.whl
dist/apache-airflow-providers-ftp-0.0.1.linux-x86_64.tar.gz
dist/apache-airflow-providers-microsoft_azure-0.0.1.linux-x86_64.tar.gz
```
I would have expected `dist/apache-airflow-providers-apache_hdfs-0.0.1-py2.py3-none-any.whl` I guess? Or at least them to be consistent (`-microsoft_azure` I can live with, but `-microsoft-azure` would be nicer). Any idea what's controlling this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376955115
##########
File path: scripts/ci/ci_prepare_backport_packages.sh
##########
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+export AIRFLOW_CI_SILENT=${AIRFLOW_CI_SILENT:="true"}
+
+export PYTHON_VERSION=${PYTHON_VERSION:-3.6}
+
+# shellcheck source=scripts/ci/_utils.sh
+. "${MY_DIR}/_utils.sh"
+
+basic_sanity_checks
+
+script_start
+
+cd "${AIRFLOW_SOURCES}"
+
+rm -rf dist/*
+rm -rf -- *.egg-info
+
+BACKPORT_PACKAGES=$(python setup.py --list-backport-packages)
+
+cp -v MANIFEST.in MANIFEST.in.bak
+cp MANIFEST-packages.in MANIFEST.in
+
+function cleanup {
+ cp MANIFEST.in.bak MANIFEST.in
+ rm MANIFEST.in.bak
+}
+
+trap cleanup EXIT
+
+for BACKPORT_PACKAGE in ${BACKPORT_PACKAGES}
+do
+ echo
+ echo "-----------------------------------------------------------------------------------"
+ echo " Preparing backporting package ${BACKPORT_PACKAGE}"
+ echo "-----------------------------------------------------------------------------------"
+ echo
+ python setup.py --provider-package "${BACKPORT_PACKAGE}" sdist bdist bdist_wheel >/dev/null
+done
+
+python setup.py --provider-package providers sdist bdist bdist_wheel >/dev/null
+
+DUMP_FILE="/tmp/airflow_provider_packages_$(date +"%Y%m%d-%H%M%S").tar.gz"
+
+tar -cvzf "${DUMP_FILE}" "dist"
+
+echo "Packages are prepared in ${DUMP_FILE}"
+
+curl -F "file=@${DUMP_FILE}" https://file.io
Review comment:
How do I download this to test it out? I tried visiting the URL form the logs of Travis, but it gives me a 404?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377112953
##########
File path: scripts/ci/ci_prepare_backport_packages.sh
##########
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+export AIRFLOW_CI_SILENT=${AIRFLOW_CI_SILENT:="true"}
+
+export PYTHON_VERSION=${PYTHON_VERSION:-3.6}
+
+# shellcheck source=scripts/ci/_utils.sh
+. "${MY_DIR}/_utils.sh"
+
+basic_sanity_checks
+
+script_start
+
+cd "${AIRFLOW_SOURCES}"
+
+rm -rf dist/*
+rm -rf -- *.egg-info
Review comment:
Added. No harm in that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-584201606
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=h1) Report
> Merging [#7391](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/61a8bb65818521ccbb846e647103535b3e36b26d?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7391 +/- ##
=======================================
Coverage 86.82% 86.82%
=======================================
Files 891 891
Lines 42095 42095
=======================================
Hits 36549 36549
Misses 5546 5546
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `100% <0%> (ø)` | :arrow_up: |
| [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `85.29% <0%> (ø)` | :arrow_up: |
| [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `94.36% <0%> (ø)` | :arrow_up: |
| [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (ø)` | :arrow_up: |
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.24% <0%> (ø)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=footer). Last update [61a8bb6...bb172ee](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r380940650
##########
File path: README.md
##########
@@ -107,6 +108,33 @@ unit of work and continuity.
+## Airflow Versions, backporting packages and future upgrade path
+
+
+Currently stable versions of Apache Airflow are released in 1.10.* series. We are working on the
+future, incompatible version of Airflow from the 2.0.* series. It is going to be released in
Review comment:
Let's not say "incompatible" like this please -- that's a stronger word than we need.
```
Currently stable versions of Apache Airflow are released in 1.10.* series. We are working on the
future, Airflow 2.x series which has some backwards-incompatible changes
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376827479
##########
File path: setup.py
##########
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+ "amazon": [aws],
+ "apache.cassandra": [cassandra],
+ "apache.druid": [druid],
+ "apache.hdfs": [hdfs],
+ "apache.hive": [hive],
+ "apache.pig": [],
+ "apache.pinot": [pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [celery],
+ "cloudant": [cloudant],
+ "cncf.kubernetes": [kubernetes],
+ "databricks": [databricks],
+ "datadog": [datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [gcp],
Review comment:
For me it seems reasonable to release it as one package because all integration requires a common hook - CloudBaseHook. The division into various departments e.g. cloud, ads, marketing_platform, analytics can also change over time. Currently, this is not a big problem, because we focused mainly on GCP, where the division into departments is clear, but other services do not have a clear division. For example: Google Ads is not related to Google Marketing Platform, Google Analytics is also not related to GMP. Why? I don't know that, but for new people, understanding these structures will be problematic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376961704
##########
File path: setup.py
##########
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+ "amazon": [aws],
+ "apache.cassandra": [cassandra],
+ "apache.druid": [druid],
+ "apache.hdfs": [hdfs],
+ "apache.hive": [hive],
+ "apache.pig": [],
+ "apache.pinot": [pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [celery],
+ "cloudant": [cloudant],
+ "cncf.kubernetes": [kubernetes],
+ "databricks": [databricks],
+ "datadog": [datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [gcp],
+ "google.marketing_platform": [gcp],
+ "google.suite": [gcp],
+ "grpc": [grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [jdbc],
+ "jenkins": [jenkins],
+ "jira": [jira],
+ "microsoft.azure": [azure],
+ "microsoft.mssql": [mssql],
+ "microsoft.winrm": [winrm],
+ "mongo": [mongo],
+ "mysql": [mysql],
+ "odbc": [odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [oracle],
+ "pagerduty": [pagerduty],
+ "papermill": [papermill],
+ "postgres": [postgres],
+ "presto": [presto],
+ "qubole": [qds],
+ "redis": [redis],
+ "salesforce": [salesforce],
+ "samba": [samba],
+ "segment": [segment],
+ "sftp": [],
+ "slack": [slack],
+ "snowflake": [snowflake],
+ "sqlite": [],
+ "ssh": [ssh],
+ "vertica": [vertica],
+ "zendesk": [zendesk],
+}
+
+
+def do_setup_package_providers(provider_module: str, deps: List[str]):
+ """Set up package providers"""
+ provider_package_name = provider_module.replace(".", "_")
+ package_name = f'apache-airflow-providers-{provider_package_name}' if provider_module != "providers" \
+ else f'apache-airflow-providers'
+
+ package_prefix = f'airflow.providers.{provider_module}' if provider_module != 'providers' \
+ else 'airflow.providers'
+ found_packages = find_packages()
+ found_packages = [package for package in found_packages if package.startswith(package_prefix)]
+ print(found_packages)
+ setup(
+ name=package_name,
+ description=f'Back-porting ${package_name} package for Airflow 1.10.*',
+ long_description=f"""
+Back-ported {package_name} to 1.10.* series of Airflow.
+""",
+ long_description_content_type='text/markdown',
+ license='Apache License 2.0',
+ version='0.0.1',
+ packages=found_packages,
+ include_package_data=True,
+ zip_safe=False,
+ install_requires=[
+ 'apache-airflow~=1.10',
+ 'pandas>=0.17.1, <1.0.0',
Review comment:
What's this for?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377115621
##########
File path: setup.py
##########
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+ "amazon": [aws],
+ "apache.cassandra": [cassandra],
+ "apache.druid": [druid],
+ "apache.hdfs": [hdfs],
+ "apache.hive": [hive],
+ "apache.pig": [],
+ "apache.pinot": [pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [celery],
+ "cloudant": [cloudant],
+ "cncf.kubernetes": [kubernetes],
+ "databricks": [databricks],
+ "datadog": [datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [gcp],
+ "google.marketing_platform": [gcp],
+ "google.suite": [gcp],
+ "grpc": [grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [jdbc],
+ "jenkins": [jenkins],
+ "jira": [jira],
+ "microsoft.azure": [azure],
+ "microsoft.mssql": [mssql],
+ "microsoft.winrm": [winrm],
+ "mongo": [mongo],
+ "mysql": [mysql],
+ "odbc": [odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [oracle],
+ "pagerduty": [pagerduty],
+ "papermill": [papermill],
+ "postgres": [postgres],
+ "presto": [presto],
+ "qubole": [qds],
+ "redis": [redis],
+ "salesforce": [salesforce],
+ "samba": [samba],
+ "segment": [segment],
+ "sftp": [],
+ "slack": [slack],
+ "snowflake": [snowflake],
+ "sqlite": [],
+ "ssh": [ssh],
+ "vertica": [vertica],
+ "zendesk": [zendesk],
+}
+
+
+def do_setup_package_providers(provider_module: str, deps: List[str]):
+ """Set up package providers"""
+ provider_package_name = provider_module.replace(".", "_")
+ package_name = f'apache-airflow-providers-{provider_package_name}' if provider_module != "providers" \
+ else f'apache-airflow-providers'
+
+ package_prefix = f'airflow.providers.{provider_module}' if provider_module != 'providers' \
+ else 'airflow.providers'
+ found_packages = find_packages()
+ found_packages = [package for package in found_packages if package.startswith(package_prefix)]
+ print(found_packages)
+ setup(
+ name=package_name,
+ description=f'Back-porting ${package_name} package for Airflow 1.10.*',
+ long_description=f"""
+Back-ported {package_name} to 1.10.* series of Airflow.
+""",
+ long_description_content_type='text/markdown',
+ license='Apache License 2.0',
+ version='0.0.1',
+ packages=found_packages,
+ include_package_data=True,
+ zip_safe=False,
+ install_requires=[
+ 'apache-airflow~=1.10',
+ 'pandas>=0.17.1, <1.0.0',
+ ] + deps,
+ classifiers=[
+ 'Development Status :: 5 - Production/Stable',
+ 'Environment :: Console',
+ 'Intended Audience :: Developers',
+ 'Intended Audience :: System Administrators',
+ 'License :: OSI Approved :: Apache Software License',
+ 'Programming Language :: Python :: 3.6',
+ 'Programming Language :: Python :: 3.7',
+ 'Topic :: System :: Monitoring',
+ ],
+ python_requires='>=3.6',
+ )
+
+
+def find_package_dependencies(package):
+ """Finds dependencies for the packages"""
+ if package != 'providers':
+ return PROVIDERS_DEPENDENCIES.get(package)
+ else:
+ return list(itertools.chain(PROVIDERS_DEPENDENCIES.values()))
+
+
+def get_provider_packages():
+ """Returns all packages available in providers"""
+ packages = list(PROVIDERS_DEPENDENCIES)
+ return ['providers'] + packages
+
+
if __name__ == "__main__":
- do_setup()
+ if "--help" in sys.argv or "-h" in sys.argv:
+ print()
+ print("You can also build any of the backport packages by "
+ "adding --provider-package PACKAGE as first two")
+ print("Of the setup.py arguments")
+ print(f"Available packages: {get_provider_packages()}")
+ print()
+ print("You can see all packages configured by adding --list-backport-packages flag")
+
+ try:
+ rmtree(os.path.join(dirname(__file__), "build"))
Review comment:
I will do both but I move it to "providers-package if". It happened far too often that I forgot about it and build much bigger package than I thought. We are anyhow abusing setup.py slightly in this case so cleaning build in case we run providers-package is a small performance penalty.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376967815
##########
File path: setup.py
##########
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+ "amazon": [aws],
+ "apache.cassandra": [cassandra],
+ "apache.druid": [druid],
+ "apache.hdfs": [hdfs],
+ "apache.hive": [hive],
+ "apache.pig": [],
+ "apache.pinot": [pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [celery],
+ "cloudant": [cloudant],
+ "cncf.kubernetes": [kubernetes],
+ "databricks": [databricks],
+ "datadog": [datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [gcp],
+ "google.marketing_platform": [gcp],
+ "google.suite": [gcp],
+ "grpc": [grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [jdbc],
+ "jenkins": [jenkins],
+ "jira": [jira],
+ "microsoft.azure": [azure],
+ "microsoft.mssql": [mssql],
+ "microsoft.winrm": [winrm],
+ "mongo": [mongo],
+ "mysql": [mysql],
+ "odbc": [odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [oracle],
+ "pagerduty": [pagerduty],
+ "papermill": [papermill],
+ "postgres": [postgres],
+ "presto": [presto],
+ "qubole": [qds],
+ "redis": [redis],
+ "salesforce": [salesforce],
+ "samba": [samba],
+ "segment": [segment],
+ "sftp": [],
+ "slack": [slack],
+ "snowflake": [snowflake],
+ "sqlite": [],
+ "ssh": [ssh],
+ "vertica": [vertica],
+ "zendesk": [zendesk],
+}
+
+
+def do_setup_package_providers(provider_module: str, deps: List[str]):
+ """Set up package providers"""
+ provider_package_name = provider_module.replace(".", "_")
+ package_name = f'apache-airflow-providers-{provider_package_name}' if provider_module != "providers" \
+ else f'apache-airflow-providers'
+
+ package_prefix = f'airflow.providers.{provider_module}' if provider_module != 'providers' \
+ else 'airflow.providers'
+ found_packages = find_packages()
+ found_packages = [package for package in found_packages if package.startswith(package_prefix)]
+ print(found_packages)
+ setup(
+ name=package_name,
+ description=f'Back-porting ${package_name} package for Airflow 1.10.*',
+ long_description=f"""
+Back-ported {package_name} to 1.10.* series of Airflow.
+""",
+ long_description_content_type='text/markdown',
+ license='Apache License 2.0',
+ version='0.0.1',
+ packages=found_packages,
+ include_package_data=True,
+ zip_safe=False,
+ install_requires=[
+ 'apache-airflow~=1.10',
+ 'pandas>=0.17.1, <1.0.0',
+ ] + deps,
+ classifiers=[
+ 'Development Status :: 5 - Production/Stable',
+ 'Environment :: Console',
+ 'Intended Audience :: Developers',
+ 'Intended Audience :: System Administrators',
+ 'License :: OSI Approved :: Apache Software License',
+ 'Programming Language :: Python :: 3.6',
+ 'Programming Language :: Python :: 3.7',
+ 'Topic :: System :: Monitoring',
+ ],
+ python_requires='>=3.6',
+ )
+
+
+def find_package_dependencies(package):
+ """Finds dependencies for the packages"""
+ if package != 'providers':
+ return PROVIDERS_DEPENDENCIES.get(package)
+ else:
+ return list(itertools.chain(PROVIDERS_DEPENDENCIES.values()))
+
+
+def get_provider_packages():
+ """Returns all packages available in providers"""
+ packages = list(PROVIDERS_DEPENDENCIES)
+ return ['providers'] + packages
+
+
if __name__ == "__main__":
- do_setup()
+ if "--help" in sys.argv or "-h" in sys.argv:
+ print()
+ print("You can also build any of the backport packages by "
+ "adding --provider-package PACKAGE as first two")
+ print("Of the setup.py arguments")
+ print(f"Available packages: {get_provider_packages()}")
+ print()
+ print("You can see all packages configured by adding --list-backport-packages flag")
+
+ try:
+ rmtree(os.path.join(dirname(__file__), "build"))
Review comment:
`clean` sub-command does this, we should probably add that in to the loop in ci_prepare_backport_packages.sh instead of doing that here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r380939740
##########
File path: MANIFEST-packages.in
##########
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+include NOTICE
+include LICENSE
+include CHANGELOG.txt
+include README.md
+graft licenses/
Review comment:
This is warning is still showing up on CI @potiuk
```
-----------------------------------------------------------------------------------
Preparing backporting package mongo
-----------------------------------------------------------------------------------
warning: Failed to find the configured license file 'licenses/*'
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: check: missing required meta-data: url
warning: build_py: byte-compiling is disabled, skipping.
warning: install_lib: byte-compiling is disabled, skipping.
gitpython not found: Cannot compute the git version.
running clean
removing 'build/lib' (and everything under it)
removing 'build/bdist.linux-x86_64' (and everything under it)
'build/scripts-3.6' does not exist -- can't clean it
removing 'build'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-587815757
@ashb @mik-laj ? Can I merge the backporting packages?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-585218154
> I've got the wheel files built by CI and am going to check out they install and uninstall in all the different permutations I am think of
Hey @ashb -> FYI - you can test the wheel packages by switching to #7389 -> there you can install any version of airflow 1.10.1 -> 1.10.9 inside Breeze with `--install-airflow-version N.N.N` and then pip install <wheel>.whl.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk edited a comment on issue #7391: [AIRFLOW-6663]
Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-589539027
I actually am going to try a slightly different, nicer approach with better separation between the main and backport packages. Bare with me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r382954347
##########
File path: backport_packages/setup_backport_packages.py
##########
@@ -0,0 +1,251 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Setup.py for the Backport packages of Airflow project."""
+
+import io
+import itertools
+import logging
+import os
+import sys
+import textwrap
+from importlib import util
+from os.path import dirname
+from shutil import copytree, rmtree
+from typing import List
+
+from setuptools import Command, find_packages, setup as setuptools_setup
+
+sys.path.append(os.path.join(dirname(__file__), os.pardir))
+
+
+logger = logging.getLogger(__name__)
+
+# Kept manually in sync with airflow.__version__
+# noinspection PyUnresolvedReferences
+spec = util.spec_from_file_location("airflow.version", os.path.join('airflow', 'version.py'))
+# noinspection PyUnresolvedReferences
+mod = util.module_from_spec(spec)
+spec.loader.exec_module(mod) # type: ignore
+version = mod.version # type: ignore
+
+PY3 = sys.version_info[0] == 3
+
+# noinspection PyUnboundLocalVariable
+try:
+ with io.open('README.md', encoding='utf-8') as f:
+ long_description = f.read()
+except FileNotFoundError:
+ long_description = ''
+
+
+class CleanCommand(Command):
+ """
+ Command to tidy up the project root.
+ Registered as cmdclass in setup() so it can be called with ``python setup.py extra_clean``.
+ """
+
+ description = "Tidy up the project root"
+ user_options = [] # type: List[str]
+
+ def initialize_options(self):
+ """Set default values for options."""
+
+ def finalize_options(self):
+ """Set final values for options."""
+
+ # noinspection PyMethodMayBeStatic
+ def run(self):
+ """Run command to remove temporary files and directories."""
+ os.chdir(dirname(__file__))
+ os.system('rm -vrf ./build ./dist ./*.pyc ./*.tgz ./*.egg-info')
+
+
+def get_providers_dependencies():
+ import setup # From AIRFLOW_SOURCES/setup.py
+
+ return {
+ "amazon": [setup.aws],
+ "apache.cassandra": [setup.cassandra],
+ "apache.druid": [setup.druid],
+ "apache.hdfs": [setup.hdfs],
+ "apache.hive": [setup.hive],
+ "apache.pig": [],
+ "apache.pinot": [setup.pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [setup.celery],
+ "cloudant": [setup.cloudant],
+ "cncf.kubernetes": [setup.kubernetes],
+ "databricks": [setup.databricks],
+ "datadog": [setup.datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [setup.docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [setup.gcp],
+ "google.marketing_platform": [setup.gcp],
+ "google.suite": [setup.gcp],
+ "grpc": [setup.grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [setup.jdbc],
+ "jenkins": [setup.jenkins],
+ "jira": [setup.jira],
+ "microsoft.azure": [setup.azure],
+ "microsoft.mssql": [setup.mssql],
+ "microsoft.winrm": [setup.winrm],
+ "mongo": [setup.mongo],
+ "mysql": [setup.mysql],
+ "odbc": [setup.odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [setup.oracle],
+ "pagerduty": [setup.pagerduty],
+ "papermill": [setup.papermill],
+ "postgres": [setup.postgres],
+ "presto": [setup.presto],
+ "qubole": [setup.qds],
+ "redis": [setup.redis],
+ "salesforce": [setup.salesforce],
+ "samba": [setup.samba],
+ "segment": [setup.segment],
+ "sftp": [setup.ssh],
+ "slack": [setup.slack],
+ "snowflake": [setup.snowflake],
+ "sqlite": [],
+ "ssh": [setup.ssh],
+ "vertica": [setup.vertica],
+ "zendesk": [setup.zendesk],
+ }
+
+
+PROVIDERS_DEPENDENCIES = get_providers_dependencies()
+
+
+def copy_provider_sources():
+ build_dir = os.path.join(dirname(__file__), "build")
+ if os.path.isdir(build_dir):
+ rmtree(build_dir)
+ package_providers_dir = os.path.join(dirname(__file__), "airflow", "providers")
+ if os.path.isdir(package_providers_dir):
+ rmtree(package_providers_dir)
+ copytree(os.path.join(*[dirname(__file__), os.pardir, "airflow", "providers"]),
Review comment:
```suggestion
copytree(os.path.join(dirname(__file__), os.pardir, "airflow", "providers"),
```
should work too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r382963732
##########
File path: backport_packages/setup_backport_packages.py
##########
@@ -0,0 +1,251 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Setup.py for the Backport packages of Airflow project."""
+
+import io
+import itertools
+import logging
+import os
+import sys
+import textwrap
+from importlib import util
+from os.path import dirname
+from shutil import copytree, rmtree
+from typing import List
+
+from setuptools import Command, find_packages, setup as setuptools_setup
+
+sys.path.append(os.path.join(dirname(__file__), os.pardir))
+
+
+logger = logging.getLogger(__name__)
+
+# Kept manually in sync with airflow.__version__
+# noinspection PyUnresolvedReferences
+spec = util.spec_from_file_location("airflow.version", os.path.join('airflow', 'version.py'))
+# noinspection PyUnresolvedReferences
+mod = util.module_from_spec(spec)
+spec.loader.exec_module(mod) # type: ignore
+version = mod.version # type: ignore
+
+PY3 = sys.version_info[0] == 3
+
+# noinspection PyUnboundLocalVariable
+try:
+ with io.open('README.md', encoding='utf-8') as f:
+ long_description = f.read()
+except FileNotFoundError:
+ long_description = ''
+
+
+class CleanCommand(Command):
+ """
+ Command to tidy up the project root.
+ Registered as cmdclass in setup() so it can be called with ``python setup.py extra_clean``.
+ """
+
+ description = "Tidy up the project root"
+ user_options = [] # type: List[str]
+
+ def initialize_options(self):
+ """Set default values for options."""
+
+ def finalize_options(self):
+ """Set final values for options."""
+
+ # noinspection PyMethodMayBeStatic
+ def run(self):
+ """Run command to remove temporary files and directories."""
+ os.chdir(dirname(__file__))
+ os.system('rm -vrf ./build ./dist ./*.pyc ./*.tgz ./*.egg-info')
+
+
+def get_providers_dependencies():
+ import setup # From AIRFLOW_SOURCES/setup.py
+
+ return {
+ "amazon": [setup.aws],
+ "apache.cassandra": [setup.cassandra],
+ "apache.druid": [setup.druid],
+ "apache.hdfs": [setup.hdfs],
+ "apache.hive": [setup.hive],
+ "apache.pig": [],
+ "apache.pinot": [setup.pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [setup.celery],
+ "cloudant": [setup.cloudant],
+ "cncf.kubernetes": [setup.kubernetes],
+ "databricks": [setup.databricks],
+ "datadog": [setup.datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [setup.docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [setup.gcp],
+ "google.marketing_platform": [setup.gcp],
+ "google.suite": [setup.gcp],
+ "grpc": [setup.grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [setup.jdbc],
+ "jenkins": [setup.jenkins],
+ "jira": [setup.jira],
+ "microsoft.azure": [setup.azure],
+ "microsoft.mssql": [setup.mssql],
+ "microsoft.winrm": [setup.winrm],
+ "mongo": [setup.mongo],
+ "mysql": [setup.mysql],
+ "odbc": [setup.odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [setup.oracle],
+ "pagerduty": [setup.pagerduty],
+ "papermill": [setup.papermill],
+ "postgres": [setup.postgres],
+ "presto": [setup.presto],
+ "qubole": [setup.qds],
+ "redis": [setup.redis],
+ "salesforce": [setup.salesforce],
+ "samba": [setup.samba],
+ "segment": [setup.segment],
+ "sftp": [setup.ssh],
+ "slack": [setup.slack],
+ "snowflake": [setup.snowflake],
+ "sqlite": [],
+ "ssh": [setup.ssh],
+ "vertica": [setup.vertica],
+ "zendesk": [setup.zendesk],
+ }
+
+
+PROVIDERS_DEPENDENCIES = get_providers_dependencies()
+
+
+def copy_provider_sources():
+ build_dir = os.path.join(dirname(__file__), "build")
+ if os.path.isdir(build_dir):
+ rmtree(build_dir)
+ package_providers_dir = os.path.join(dirname(__file__), "airflow", "providers")
+ if os.path.isdir(package_providers_dir):
+ rmtree(package_providers_dir)
+ copytree(os.path.join(*[dirname(__file__), os.pardir, "airflow", "providers"]),
+ os.path.join(*[dirname(__file__), "airflow", "providers"]))
Review comment:
and here too :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-584201606
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=h1) Report
> Merging [#7391](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/61a8bb65818521ccbb846e647103535b3e36b26d?src=pr&el=desc) will **decrease** coverage by `0.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7391 +/- ##
==========================================
- Coverage 86.82% 86.65% -0.18%
==========================================
Files 891 891
Lines 42095 42095
==========================================
- Hits 36549 36477 -72
- Misses 5546 5618 +72
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0%> (-45.66%)` | :arrow_down: |
| [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55% <0%> (-45%)` | :arrow_down: |
| [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `76.02% <0%> (-1.54%)` | :arrow_down: |
| [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `90.9% <0%> (-0.83%)` | :arrow_down: |
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.83% <0%> (-0.29%)` | :arrow_down: |
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.09% <0%> (-0.15%)` | :arrow_down: |
| [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.12% <0%> (+0.19%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=footer). Last update [61a8bb6...7688100](https://codecov.io/gh/apache/airflow/pull/7391?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-590026357
All fixed @kaxil
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r380939740
##########
File path: MANIFEST-packages.in
##########
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+include NOTICE
+include LICENSE
+include CHANGELOG.txt
+include README.md
+graft licenses/
Review comment:
This is warning is still showing up on CI @potiuk
https://travis-ci.org/apache/airflow/jobs/651778738
```
-----------------------------------------------------------------------------------
Preparing backporting package mongo
-----------------------------------------------------------------------------------
warning: Failed to find the configured license file 'licenses/*'
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: check: missing required meta-data: url
warning: build_py: byte-compiling is disabled, skipping.
warning: install_lib: byte-compiling is disabled, skipping.
gitpython not found: Cannot compute the git version.
running clean
removing 'build/lib' (and everything under it)
removing 'build/bdist.linux-x86_64' (and everything under it)
'build/scripts-3.6' does not exist -- can't clean it
removing 'build'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r380941844
##########
File path: README.md
##########
@@ -107,6 +108,33 @@ unit of work and continuity.
+## Airflow Versions, backporting packages and future upgrade path
+
+
+Currently stable versions of Apache Airflow are released in 1.10.* series. We are working on the
+future, incompatible version of Airflow from the 2.0.* series. It is going to be released in
+in 2020. However the exact time of release depends on many factors and is yet unknown.
+We have already a lot of changes in the hooks/operators/sensors for many external systems
+and they are not used because they are part of the master/2.0 release.
+
+In the Airflow 2.0 - following AIP-21 "change in import paths" all the non-core operators/hooks/sensors
+of Apache Airflow have been moved to the "airflow.providers" package. This opened a possibility to
+use the operators from Airflow 2.0 in Airflow 1.10 - with the constraint that those
+packages can only be used in python3.6+ environment.
+
+Therefore we decided to prepare and release backport packages that can be installed
+for older Airflow versions. Those backport packages are released more frequently. Users do not
+have to upgrade their Airflow version to use those packages. There are a number of incompatibilities
+between Airflow 2.0 and 1.10.* - documented in [UPDATING.md](UPDATING.md). With backported
+providers package users can migrate their DAGs to the new providers package incrementally
+and once they convert to the new operators/sensors/hooks they can seamlessly migrate their
+environments to Airflow 2.0.
+
+More information about the status and releases of the back-ported packages are available
+at https://cwiki.apache.org/confluence/display/AIRFLOW/Backported+providers+packages+for+Airflow+1.10.*+series
+
+You
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r376827479
##########
File path: setup.py
##########
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+ "amazon": [aws],
+ "apache.cassandra": [cassandra],
+ "apache.druid": [druid],
+ "apache.hdfs": [hdfs],
+ "apache.hive": [hive],
+ "apache.pig": [],
+ "apache.pinot": [pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [celery],
+ "cloudant": [cloudant],
+ "cncf.kubernetes": [kubernetes],
+ "databricks": [databricks],
+ "datadog": [datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [gcp],
Review comment:
For me it seems reasonable to release it as one package because all integration requires a common hook - CloudBaseHook. The division into various departments e.g. cloud, ads, marketing_platform, analytics can also change over time. Currently, this is not a big problem, because we focused mainly on GCP, where the division into departments is clear, but other services do not have a clear division. For example: Google Ads is not related to Google Marketing Platform, Google Analytics is also not related to GMP. Why? I don't know that, but for new people, understanding these structures will be problematic. Very often, when a company decides on one provider it tries to use many of its services, and so it seems sensible to give such an opportunity right away. It will also facilitate testing when we implement AIP-8, because we will not have packages whose dependencies are very close to each other.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-589733090
@ashb @mik-laj - As promise, I rewrote the backport scripts to be much more separated and independent. I think this is WAAAAAY better now, and it actually is even better when it comes to the "apply_defaults" problem we discussed earlier today. What happens during the build now is that all the "providers" sources are simply copied over to "airflow/providers" inside the "backport_packages" folder before packaging is done.
This is where we can plug-in any modification of the sources we can think of, including the metaclass modifications.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391:
[AIRFLOW-6663] Prepare backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r382963718
##########
File path: backport_packages/setup_backport_packages.py
##########
@@ -0,0 +1,251 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Setup.py for the Backport packages of Airflow project."""
+
+import io
+import itertools
+import logging
+import os
+import sys
+import textwrap
+from importlib import util
+from os.path import dirname
+from shutil import copytree, rmtree
+from typing import List
+
+from setuptools import Command, find_packages, setup as setuptools_setup
+
+sys.path.append(os.path.join(dirname(__file__), os.pardir))
+
+
+logger = logging.getLogger(__name__)
+
+# Kept manually in sync with airflow.__version__
+# noinspection PyUnresolvedReferences
+spec = util.spec_from_file_location("airflow.version", os.path.join('airflow', 'version.py'))
+# noinspection PyUnresolvedReferences
+mod = util.module_from_spec(spec)
+spec.loader.exec_module(mod) # type: ignore
+version = mod.version # type: ignore
+
+PY3 = sys.version_info[0] == 3
+
+# noinspection PyUnboundLocalVariable
+try:
+ with io.open('README.md', encoding='utf-8') as f:
+ long_description = f.read()
+except FileNotFoundError:
+ long_description = ''
+
+
+class CleanCommand(Command):
+ """
+ Command to tidy up the project root.
+ Registered as cmdclass in setup() so it can be called with ``python setup.py extra_clean``.
+ """
+
+ description = "Tidy up the project root"
+ user_options = [] # type: List[str]
+
+ def initialize_options(self):
+ """Set default values for options."""
+
+ def finalize_options(self):
+ """Set final values for options."""
+
+ # noinspection PyMethodMayBeStatic
+ def run(self):
+ """Run command to remove temporary files and directories."""
+ os.chdir(dirname(__file__))
+ os.system('rm -vrf ./build ./dist ./*.pyc ./*.tgz ./*.egg-info')
+
+
+def get_providers_dependencies():
+ import setup # From AIRFLOW_SOURCES/setup.py
+
+ return {
+ "amazon": [setup.aws],
+ "apache.cassandra": [setup.cassandra],
+ "apache.druid": [setup.druid],
+ "apache.hdfs": [setup.hdfs],
+ "apache.hive": [setup.hive],
+ "apache.pig": [],
+ "apache.pinot": [setup.pinot],
+ "apache.spark": [],
+ "apache.sqoop": [],
+ "celery": [setup.celery],
+ "cloudant": [setup.cloudant],
+ "cncf.kubernetes": [setup.kubernetes],
+ "databricks": [setup.databricks],
+ "datadog": [setup.datadog],
+ "dingding": [],
+ "discord": [],
+ "docker": [setup.docker],
+ "email": [],
+ "ftp": [],
+ "google.cloud": [setup.gcp],
+ "google.marketing_platform": [setup.gcp],
+ "google.suite": [setup.gcp],
+ "grpc": [setup.grpc],
+ "http": [],
+ "imap": [],
+ "jdbc": [setup.jdbc],
+ "jenkins": [setup.jenkins],
+ "jira": [setup.jira],
+ "microsoft.azure": [setup.azure],
+ "microsoft.mssql": [setup.mssql],
+ "microsoft.winrm": [setup.winrm],
+ "mongo": [setup.mongo],
+ "mysql": [setup.mysql],
+ "odbc": [setup.odbc],
+ "openfass": [],
+ "opsgenie": [],
+ "oracle": [setup.oracle],
+ "pagerduty": [setup.pagerduty],
+ "papermill": [setup.papermill],
+ "postgres": [setup.postgres],
+ "presto": [setup.presto],
+ "qubole": [setup.qds],
+ "redis": [setup.redis],
+ "salesforce": [setup.salesforce],
+ "samba": [setup.samba],
+ "segment": [setup.segment],
+ "sftp": [setup.ssh],
+ "slack": [setup.slack],
+ "snowflake": [setup.snowflake],
+ "sqlite": [],
+ "ssh": [setup.ssh],
+ "vertica": [setup.vertica],
+ "zendesk": [setup.zendesk],
+ }
+
+
+PROVIDERS_DEPENDENCIES = get_providers_dependencies()
+
+
+def copy_provider_sources():
+ build_dir = os.path.join(dirname(__file__), "build")
+ if os.path.isdir(build_dir):
+ rmtree(build_dir)
+ package_providers_dir = os.path.join(dirname(__file__), "airflow", "providers")
+ if os.path.isdir(package_providers_dir):
+ rmtree(package_providers_dir)
+ copytree(os.path.join(*[dirname(__file__), os.pardir, "airflow", "providers"]),
Review comment:
indeed!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-584152145
> Something slightly odd is going on with the naming -- some are using `-`, others are using `_`
>
Yeah. That's default naming conventions used by sdist/bdist :(. They are not consistent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare
backporting packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
URL: https://github.com/apache/airflow/pull/7391#issuecomment-589979050
All warnings and Travis fixed with this one - and it is really nicer and more separated. @ashb @mik-laj @nuclearpinguin @kaxil . I would love to merge it - then we can start running automated tests on GCP providers package (that's the last blocker for us to automate all the tests).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services