You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2023/11/18 16:02:33 UTC
(airflow) branch add-reproducible-builds-support updated (3211525594 -> 3060c4ec1b)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch add-reproducible-builds-support
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit 3211525594 Use reproducible builds for provider packages
new 3060c4ec1b Use reproducible builds for provider packages
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (3211525594)
\
N -- N -- N refs/heads/add-reproducible-builds-support (3060c4ec1b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/actions/build-prod-images/action.yml | 2 +-
.github/workflows/ci.yml | 2 +-
dev/breeze/src/airflow_breeze/prepare_providers/provider_packages.py | 3 ---
dev/refresh_images.sh | 2 +-
4 files changed, 3 insertions(+), 6 deletions(-)
(airflow) 01/01: Use reproducible builds for provider packages
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch add-reproducible-builds-support
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3060c4ec1b120886de2ab02598ddc34787342844
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Nov 16 16:29:53 2023 +0100
Use reproducible builds for provider packages
Flit allows to build reproducible packages (packages that can be
compared bit-by-bit) providing that source date epoch is set to
repeatable value when package is built. This PR implements
reproducibility of our builds by freezing the documentation preparation
time in provider.yaml as "source date epoch" and always using it when
building the package. This way anyone using breeze to build the package
will have exactly the same binary package produced, which will make it
way easier to verify if the packages are ready for release by the PMC
member.
We will no longer have to check the sources, PMC members will simply
need to build the same packages locally using breeze and see if the
generated packages are exactly the same.
That also includes permissions bits - flit sets permissions of
the generated packages to "no permissions for other/group" in
order to get bit-to-bit reproducibility - because on some systems
umask is set differently by default and created file would be
different because of the permission bit. This caused problems
when building docker images - so we had it changed when moving
packages to 'dist' folder, but this PR changes it so that CI
build does it when moving packages from dist to docker-context-files
instead
The "source-date-epoch" fields have been regenerated in this PR as
well. Also this PR replaces `lru_cache` method of storing output
of `get_provider_metadata_packages` with custom-stored dictionary -
thanks to that instead of invalidating whole cache of providers
metadata refreshed from yaml files we can refresh individual provider
metadata entries after they have been updated. This saves a lot
of time for validation - because every time when provider yaml is
updated we need to re-read it and re-validate it with json schema,
with this change we only do it for the updated provider yaml - which
saves about 0.5 a second per provider yaml update and when you
update all provides it is done way faster.
---
.github/actions/build-prod-images/action.yml | 2 +-
.github/workflows/ci.yml | 2 +-
airflow/provider.yaml.schema.json | 11 ++++++
airflow/providers/airbyte/provider.yaml | 1 +
airflow/providers/alibaba/provider.yaml | 1 +
airflow/providers/amazon/provider.yaml | 1 +
airflow/providers/apache/beam/provider.yaml | 1 +
airflow/providers/apache/cassandra/provider.yaml | 1 +
airflow/providers/apache/drill/provider.yaml | 1 +
airflow/providers/apache/druid/provider.yaml | 1 +
airflow/providers/apache/flink/provider.yaml | 1 +
airflow/providers/apache/hdfs/provider.yaml | 1 +
airflow/providers/apache/hive/provider.yaml | 1 +
airflow/providers/apache/impala/provider.yaml | 1 +
airflow/providers/apache/kafka/provider.yaml | 1 +
airflow/providers/apache/kylin/provider.yaml | 1 +
airflow/providers/apache/livy/provider.yaml | 1 +
airflow/providers/apache/pig/provider.yaml | 1 +
airflow/providers/apache/pinot/provider.yaml | 1 +
airflow/providers/apache/spark/provider.yaml | 1 +
airflow/providers/apache/sqoop/provider.yaml | 1 +
airflow/providers/apprise/provider.yaml | 1 +
airflow/providers/arangodb/provider.yaml | 1 +
airflow/providers/asana/provider.yaml | 1 +
airflow/providers/atlassian/jira/provider.yaml | 1 +
airflow/providers/celery/provider.yaml | 1 +
airflow/providers/cloudant/provider.yaml | 1 +
airflow/providers/cncf/kubernetes/provider.yaml | 1 +
airflow/providers/cohere/provider.yaml | 1 +
airflow/providers/common/io/provider.yaml | 1 +
airflow/providers/common/sql/provider.yaml | 1 +
airflow/providers/daskexecutor/provider.yaml | 1 +
airflow/providers/databricks/provider.yaml | 1 +
airflow/providers/datadog/provider.yaml | 1 +
airflow/providers/dbt/cloud/provider.yaml | 1 +
airflow/providers/dingding/provider.yaml | 1 +
airflow/providers/discord/provider.yaml | 1 +
airflow/providers/docker/provider.yaml | 1 +
airflow/providers/elasticsearch/provider.yaml | 1 +
airflow/providers/exasol/provider.yaml | 1 +
airflow/providers/facebook/provider.yaml | 1 +
airflow/providers/ftp/provider.yaml | 1 +
airflow/providers/github/provider.yaml | 1 +
airflow/providers/google/provider.yaml | 1 +
airflow/providers/grpc/provider.yaml | 1 +
airflow/providers/hashicorp/provider.yaml | 1 +
airflow/providers/http/provider.yaml | 1 +
airflow/providers/imap/provider.yaml | 1 +
airflow/providers/influxdb/provider.yaml | 1 +
airflow/providers/jdbc/provider.yaml | 1 +
airflow/providers/jenkins/provider.yaml | 1 +
airflow/providers/microsoft/azure/provider.yaml | 1 +
airflow/providers/microsoft/mssql/provider.yaml | 1 +
airflow/providers/microsoft/psrp/provider.yaml | 1 +
airflow/providers/microsoft/winrm/provider.yaml | 1 +
airflow/providers/mongo/provider.yaml | 1 +
airflow/providers/mysql/provider.yaml | 1 +
airflow/providers/neo4j/provider.yaml | 1 +
airflow/providers/odbc/provider.yaml | 1 +
airflow/providers/openai/provider.yaml | 1 +
airflow/providers/openfaas/provider.yaml | 1 +
airflow/providers/openlineage/provider.yaml | 1 +
airflow/providers/opensearch/provider.yaml | 1 +
airflow/providers/opsgenie/provider.yaml | 1 +
airflow/providers/oracle/provider.yaml | 1 +
airflow/providers/pagerduty/provider.yaml | 1 +
airflow/providers/papermill/provider.yaml | 1 +
airflow/providers/pgvector/provider.yaml | 1 +
airflow/providers/pinecone/provider.yaml | 1 +
airflow/providers/plexus/provider.yaml | 1 +
airflow/providers/postgres/provider.yaml | 1 +
airflow/providers/presto/provider.yaml | 1 +
airflow/providers/redis/provider.yaml | 1 +
airflow/providers/salesforce/provider.yaml | 1 +
airflow/providers/samba/provider.yaml | 1 +
airflow/providers/segment/provider.yaml | 1 +
airflow/providers/sendgrid/provider.yaml | 1 +
airflow/providers/sftp/provider.yaml | 1 +
airflow/providers/singularity/provider.yaml | 1 +
airflow/providers/slack/provider.yaml | 1 +
airflow/providers/smtp/provider.yaml | 1 +
airflow/providers/snowflake/provider.yaml | 1 +
airflow/providers/sqlite/provider.yaml | 1 +
airflow/providers/ssh/provider.yaml | 1 +
airflow/providers/tableau/provider.yaml | 1 +
airflow/providers/tabular/provider.yaml | 1 +
airflow/providers/telegram/provider.yaml | 1 +
airflow/providers/trino/provider.yaml | 1 +
airflow/providers/vertica/provider.yaml | 1 +
airflow/providers/weaviate/provider.yaml | 1 +
airflow/providers/yandex/provider.yaml | 1 +
airflow/providers/zendesk/provider.yaml | 1 +
.../commands/release_management_commands.py | 1 -
.../prepare_providers/provider_documentation.py | 33 +++++++++++++----
.../prepare_providers/provider_packages.py | 16 +++++----
dev/breeze/src/airflow_breeze/utils/packages.py | 41 +++++++++++++++-------
dev/refresh_images.sh | 2 +-
97 files changed, 166 insertions(+), 31 deletions(-)
diff --git a/.github/actions/build-prod-images/action.yml b/.github/actions/build-prod-images/action.yml
index 74667f0de1..882265a212 100644
--- a/.github/actions/build-prod-images/action.yml
+++ b/.github/actions/build-prod-images/action.yml
@@ -48,7 +48,7 @@ runs:
--package-format wheel --version-suffix-for-pypi dev0
- name: "Move dist packages to docker-context files"
shell: bash
- run: mv -v ./dist/*.whl ./docker-context-files
+ run: mv -v ./dist/*.whl ./docker-context-files && chmod a+r ./docker-context-files/*
- name: "Download constraints from the CI build"
uses: actions/download-artifact@v3
with:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 7a9ffb2251..374ea142ad 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -1931,7 +1931,7 @@ jobs:
COMMIT_SHA: ${{ github.sha }}
if: matrix.platform == 'linux/amd64'
- name: "Move dist packages to docker-context files"
- run: mv -v ./dist/*.whl ./docker-context-files
+ run: mv -v ./dist/*.whl ./docker-context-files && chmod a+r ./docker-context-files/*
- name: "Push PROD cache ${{ matrix.python-version }} ${{ matrix.platform }}"
run: >
breeze prod-image build
diff --git a/airflow/provider.yaml.schema.json b/airflow/provider.yaml.schema.json
index da32cd1c40..be589f5490 100644
--- a/airflow/provider.yaml.schema.json
+++ b/airflow/provider.yaml.schema.json
@@ -380,6 +380,16 @@
"type": "string"
}
}
+ },
+ "source-date-epoch": {
+ "type": "integer",
+ "description": "Source date epoch - seconds since epoch (gmtime) when the release documentation was prepared. Used to generate reproducible package builds with flint.",
+
+ "minimum": 0,
+ "default": 0,
+ "examples": [
+ 1609459200
+ ]
}
},
"additionalProperties": false,
@@ -442,6 +452,7 @@
"package-name",
"description",
"suspended",
+ "source-date-epoch",
"dependencies",
"versions"
]
diff --git a/airflow/providers/airbyte/provider.yaml b/airflow/providers/airbyte/provider.yaml
index fb80c9872e..7eabe2cfbd 100644
--- a/airflow/providers/airbyte/provider.yaml
+++ b/airflow/providers/airbyte/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Airbyte <https://airbyte.io/>`__
suspended: false
+source-date-epoch: 1700148043
versions:
- 3.4.0
- 3.3.2
diff --git a/airflow/providers/alibaba/provider.yaml b/airflow/providers/alibaba/provider.yaml
index 5a596fd31a..2e0664e1b0 100644
--- a/airflow/providers/alibaba/provider.yaml
+++ b/airflow/providers/alibaba/provider.yaml
@@ -22,6 +22,7 @@ description: |
Alibaba Cloud integration (including `Alibaba Cloud <https://www.alibabacloud.com//>`__).
suspended: false
+source-date-epoch: 1700148044
versions:
- 2.6.0
- 2.5.3
diff --git a/airflow/providers/amazon/provider.yaml b/airflow/providers/amazon/provider.yaml
index 3b357520a0..7873c40f31 100644
--- a/airflow/providers/amazon/provider.yaml
+++ b/airflow/providers/amazon/provider.yaml
@@ -22,6 +22,7 @@ description: |
Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
suspended: false
+source-date-epoch: 1700148045
versions:
- 8.11.0
- 8.10.0
diff --git a/airflow/providers/apache/beam/provider.yaml b/airflow/providers/apache/beam/provider.yaml
index dc42d34e8c..6910e281fa 100644
--- a/airflow/providers/apache/beam/provider.yaml
+++ b/airflow/providers/apache/beam/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Beam <https://beam.apache.org/>`__.
suspended: false
+source-date-epoch: 1700148046
versions:
- 5.3.0
- 5.2.3
diff --git a/airflow/providers/apache/cassandra/provider.yaml b/airflow/providers/apache/cassandra/provider.yaml
index e430d34534..a427c959e8 100644
--- a/airflow/providers/apache/cassandra/provider.yaml
+++ b/airflow/providers/apache/cassandra/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Cassandra <http://cassandra.apache.org/>`__.
suspended: false
+source-date-epoch: 1700148046
versions:
- 3.3.0
- 3.2.1
diff --git a/airflow/providers/apache/drill/provider.yaml b/airflow/providers/apache/drill/provider.yaml
index fb9314c262..c5b50280b4 100644
--- a/airflow/providers/apache/drill/provider.yaml
+++ b/airflow/providers/apache/drill/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Drill <https://drill.apache.org/>`__.
suspended: false
+source-date-epoch: 1700148047
versions:
- 2.5.0
- 2.4.4
diff --git a/airflow/providers/apache/druid/provider.yaml b/airflow/providers/apache/druid/provider.yaml
index 0e982d6727..e9c46340dd 100644
--- a/airflow/providers/apache/druid/provider.yaml
+++ b/airflow/providers/apache/druid/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Druid <https://druid.apache.org/>`__.
suspended: false
+source-date-epoch: 1700148047
versions:
- 3.6.0
- 3.5.0
diff --git a/airflow/providers/apache/flink/provider.yaml b/airflow/providers/apache/flink/provider.yaml
index 0d6cda3412..99c8041841 100644
--- a/airflow/providers/apache/flink/provider.yaml
+++ b/airflow/providers/apache/flink/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Flink <https://flink.apache.org/>`__
suspended: false
+source-date-epoch: 1700148048
versions:
- 1.2.0
- 1.1.3
diff --git a/airflow/providers/apache/hdfs/provider.yaml b/airflow/providers/apache/hdfs/provider.yaml
index bfa836c134..5a52e6ebb4 100644
--- a/airflow/providers/apache/hdfs/provider.yaml
+++ b/airflow/providers/apache/hdfs/provider.yaml
@@ -23,6 +23,7 @@ description: |
and `WebHDFS <https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html>`__.
suspended: false
+source-date-epoch: 1700148048
versions:
- 4.2.0
- 4.1.1
diff --git a/airflow/providers/apache/hive/provider.yaml b/airflow/providers/apache/hive/provider.yaml
index 82e8f1e5e0..4b2dbf3958 100644
--- a/airflow/providers/apache/hive/provider.yaml
+++ b/airflow/providers/apache/hive/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Hive <https://hive.apache.org/>`__
suspended: false
+source-date-epoch: 1700148049
versions:
- 6.2.0
- 6.1.6
diff --git a/airflow/providers/apache/impala/provider.yaml b/airflow/providers/apache/impala/provider.yaml
index 77a6ec4924..a997f3053f 100644
--- a/airflow/providers/apache/impala/provider.yaml
+++ b/airflow/providers/apache/impala/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Impala <https://impala.apache.org/>`__.
suspended: false
+source-date-epoch: 1700148050
versions:
- 1.2.0
- 1.1.3
diff --git a/airflow/providers/apache/kafka/provider.yaml b/airflow/providers/apache/kafka/provider.yaml
index ab8ad48f04..674b48380c 100644
--- a/airflow/providers/apache/kafka/provider.yaml
+++ b/airflow/providers/apache/kafka/provider.yaml
@@ -20,6 +20,7 @@ package-name: apache-airflow-providers-apache-kafka
name: Apache Kafka
suspended: false
+source-date-epoch: 1700148050
description: |
`Apache Kafka <https://kafka.apache.org/>`__
versions:
diff --git a/airflow/providers/apache/kylin/provider.yaml b/airflow/providers/apache/kylin/provider.yaml
index 07af03e1c6..fe36480fa3 100644
--- a/airflow/providers/apache/kylin/provider.yaml
+++ b/airflow/providers/apache/kylin/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Kylin <https://kylin.apache.org/>`__
suspended: false
+source-date-epoch: 1700148051
versions:
- 3.3.0
- 3.2.1
diff --git a/airflow/providers/apache/livy/provider.yaml b/airflow/providers/apache/livy/provider.yaml
index 85f1d03973..b62796e547 100644
--- a/airflow/providers/apache/livy/provider.yaml
+++ b/airflow/providers/apache/livy/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Livy <https://livy.apache.org/>`__
suspended: false
+source-date-epoch: 1700148051
versions:
- 3.6.0
- 3.5.4
diff --git a/airflow/providers/apache/pig/provider.yaml b/airflow/providers/apache/pig/provider.yaml
index e2f3571320..0dbe8af52a 100644
--- a/airflow/providers/apache/pig/provider.yaml
+++ b/airflow/providers/apache/pig/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Pig <https://pig.apache.org/>`__
suspended: false
+source-date-epoch: 1700148052
versions:
- 4.2.0
- 4.1.2
diff --git a/airflow/providers/apache/pinot/provider.yaml b/airflow/providers/apache/pinot/provider.yaml
index 347c5c3d59..23c6e8a3fa 100644
--- a/airflow/providers/apache/pinot/provider.yaml
+++ b/airflow/providers/apache/pinot/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Pinot <https://pinot.apache.org/>`__
suspended: false
+source-date-epoch: 1700148052
versions:
- 4.2.0
- 4.1.4
diff --git a/airflow/providers/apache/spark/provider.yaml b/airflow/providers/apache/spark/provider.yaml
index 31f2728a5c..d819d56a38 100644
--- a/airflow/providers/apache/spark/provider.yaml
+++ b/airflow/providers/apache/spark/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Spark <https://spark.apache.org/>`__
suspended: false
+source-date-epoch: 1700148053
versions:
- 4.4.0
- 4.3.0
diff --git a/airflow/providers/apache/sqoop/provider.yaml b/airflow/providers/apache/sqoop/provider.yaml
index f959f50cb3..80934eb587 100644
--- a/airflow/providers/apache/sqoop/provider.yaml
+++ b/airflow/providers/apache/sqoop/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Sqoop <https://sqoop.apache.org/>`__
suspended: false
+source-date-epoch: 1700148053
versions:
- 4.1.0
- 4.0.0
diff --git a/airflow/providers/apprise/provider.yaml b/airflow/providers/apprise/provider.yaml
index 90f3cbcdb4..248a546bb0 100644
--- a/airflow/providers/apprise/provider.yaml
+++ b/airflow/providers/apprise/provider.yaml
@@ -24,6 +24,7 @@ description: |
`Apprise <https://github.com/caronc/apprise>`__
suspended: false
+source-date-epoch: 1700148054
versions:
- 1.1.0
diff --git a/airflow/providers/arangodb/provider.yaml b/airflow/providers/arangodb/provider.yaml
index 5e4b602b3f..b5a83e7e90 100644
--- a/airflow/providers/arangodb/provider.yaml
+++ b/airflow/providers/arangodb/provider.yaml
@@ -26,6 +26,7 @@ dependencies:
- python-arango>=7.3.2
suspended: false
+source-date-epoch: 1700148054
versions:
- 2.3.0
- 2.2.2
diff --git a/airflow/providers/asana/provider.yaml b/airflow/providers/asana/provider.yaml
index 8c1b8a0430..4a8bd8777c 100644
--- a/airflow/providers/asana/provider.yaml
+++ b/airflow/providers/asana/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Asana <https://app.asana.com/>`__
suspended: false
+source-date-epoch: 1700148055
versions:
- 2.3.0
- 2.2.2
diff --git a/airflow/providers/atlassian/jira/provider.yaml b/airflow/providers/atlassian/jira/provider.yaml
index 6d3e1c9cc9..04d0a7a36d 100644
--- a/airflow/providers/atlassian/jira/provider.yaml
+++ b/airflow/providers/atlassian/jira/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Atlassian Jira <https://www.atlassian.com/>`__
suspended: false
+source-date-epoch: 1700148055
versions:
- 2.2.0
- 2.1.1
diff --git a/airflow/providers/celery/provider.yaml b/airflow/providers/celery/provider.yaml
index ff9e767c9d..74f63bb4e6 100644
--- a/airflow/providers/celery/provider.yaml
+++ b/airflow/providers/celery/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Celery <http://www.celeryproject.org/>`__
suspended: false
+source-date-epoch: 1700148056
versions:
- 3.4.1
- 3.4.0
diff --git a/airflow/providers/cloudant/provider.yaml b/airflow/providers/cloudant/provider.yaml
index 32f79df6c1..11a99600d1 100644
--- a/airflow/providers/cloudant/provider.yaml
+++ b/airflow/providers/cloudant/provider.yaml
@@ -22,6 +22,7 @@ description: |
`IBM Cloudant <https://www.ibm.com/cloud/cloudant>`__
suspended: false
+source-date-epoch: 1700148056
versions:
- 3.3.0
- 3.2.1
diff --git a/airflow/providers/cncf/kubernetes/provider.yaml b/airflow/providers/cncf/kubernetes/provider.yaml
index 4ea8a84087..cf1c107843 100644
--- a/airflow/providers/cncf/kubernetes/provider.yaml
+++ b/airflow/providers/cncf/kubernetes/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Kubernetes <https://kubernetes.io/>`__
suspended: false
+source-date-epoch: 1700148057
versions:
- 7.9.0
- 7.8.0
diff --git a/airflow/providers/cohere/provider.yaml b/airflow/providers/cohere/provider.yaml
index 55500e0e76..8a6b12f73a 100644
--- a/airflow/providers/cohere/provider.yaml
+++ b/airflow/providers/cohere/provider.yaml
@@ -24,6 +24,7 @@ description: |
`Cohere <https://docs.cohere.com/docs>`__
suspended: false
+source-date-epoch: 1700148058
versions:
- 1.0.0
diff --git a/airflow/providers/common/io/provider.yaml b/airflow/providers/common/io/provider.yaml
index 1e1ecb392e..fdc8134090 100644
--- a/airflow/providers/common/io/provider.yaml
+++ b/airflow/providers/common/io/provider.yaml
@@ -22,6 +22,7 @@ description: |
``Common IO Provider``
suspended: false
+source-date-epoch: 1700148058
versions:
- 1.0.1
- 1.0.0
diff --git a/airflow/providers/common/sql/provider.yaml b/airflow/providers/common/sql/provider.yaml
index 4dc1e70326..c448889250 100644
--- a/airflow/providers/common/sql/provider.yaml
+++ b/airflow/providers/common/sql/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Common SQL Provider <https://en.wikipedia.org/wiki/SQL>`__
suspended: false
+source-date-epoch: 1700148058
versions:
- 1.8.1
- 1.8.0
diff --git a/airflow/providers/daskexecutor/provider.yaml b/airflow/providers/daskexecutor/provider.yaml
index 76e67944b8..5516edf827 100644
--- a/airflow/providers/daskexecutor/provider.yaml
+++ b/airflow/providers/daskexecutor/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Dask <https://www.dask.org/>`__
suspended: false
+source-date-epoch: 1700148059
versions:
- 1.1.0
- 1.0.1
diff --git a/airflow/providers/databricks/provider.yaml b/airflow/providers/databricks/provider.yaml
index 2890561e38..81ce8ee1b8 100644
--- a/airflow/providers/databricks/provider.yaml
+++ b/airflow/providers/databricks/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Databricks <https://databricks.com/>`__
suspended: false
+source-date-epoch: 1700148060
versions:
- 5.0.0
- 4.7.0
diff --git a/airflow/providers/datadog/provider.yaml b/airflow/providers/datadog/provider.yaml
index 7286c718fe..6a757e377e 100644
--- a/airflow/providers/datadog/provider.yaml
+++ b/airflow/providers/datadog/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Datadog <https://www.datadoghq.com/>`__
suspended: false
+source-date-epoch: 1700148060
versions:
- 3.4.0
- 3.3.2
diff --git a/airflow/providers/dbt/cloud/provider.yaml b/airflow/providers/dbt/cloud/provider.yaml
index 76d181253f..acc74672c2 100644
--- a/airflow/providers/dbt/cloud/provider.yaml
+++ b/airflow/providers/dbt/cloud/provider.yaml
@@ -22,6 +22,7 @@ description: |
`dbt Cloud <https://www.getdbt.com/product/what-is-dbt/>`__
suspended: false
+source-date-epoch: 1700148061
versions:
- 3.4.0
- 3.3.0
diff --git a/airflow/providers/dingding/provider.yaml b/airflow/providers/dingding/provider.yaml
index 17a92cac77..7a418f95a6 100644
--- a/airflow/providers/dingding/provider.yaml
+++ b/airflow/providers/dingding/provider.yaml
@@ -22,6 +22,7 @@ description: |
`DingTalk <https://www.dingtalk.com/>`__
suspended: false
+source-date-epoch: 1700148061
versions:
- 3.3.0
- 3.2.1
diff --git a/airflow/providers/discord/provider.yaml b/airflow/providers/discord/provider.yaml
index dc91840d8a..465067a78c 100644
--- a/airflow/providers/discord/provider.yaml
+++ b/airflow/providers/discord/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Discord <https://discordapp.com/>`__
suspended: false
+source-date-epoch: 1700148062
versions:
- 3.4.1
- 3.4.0
diff --git a/airflow/providers/docker/provider.yaml b/airflow/providers/docker/provider.yaml
index d54f2db7ae..bbf4553fb2 100644
--- a/airflow/providers/docker/provider.yaml
+++ b/airflow/providers/docker/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Docker <https://docs.docker.com/install/>`__
suspended: false
+source-date-epoch: 1700148062
versions:
- 3.8.1
- 3.8.0
diff --git a/airflow/providers/elasticsearch/provider.yaml b/airflow/providers/elasticsearch/provider.yaml
index dc7e7829c1..8b77048926 100644
--- a/airflow/providers/elasticsearch/provider.yaml
+++ b/airflow/providers/elasticsearch/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Elasticsearch <https://www.elastic.co/elasticsearch>`__
suspended: false
+source-date-epoch: 1700148063
versions:
- 5.1.1
- 5.1.0
diff --git a/airflow/providers/exasol/provider.yaml b/airflow/providers/exasol/provider.yaml
index 8314c8960a..f193510a7e 100644
--- a/airflow/providers/exasol/provider.yaml
+++ b/airflow/providers/exasol/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Exasol <https://docs.exasol.com/home.htm>`__
suspended: false
+source-date-epoch: 1700148064
versions:
- 4.3.0
- 4.2.5
diff --git a/airflow/providers/facebook/provider.yaml b/airflow/providers/facebook/provider.yaml
index 46aa33f244..a1427c9a6e 100644
--- a/airflow/providers/facebook/provider.yaml
+++ b/airflow/providers/facebook/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Facebook Ads <http://business.facebook.com/>`__
suspended: false
+source-date-epoch: 1700148064
versions:
- 3.3.0
- 3.2.2
diff --git a/airflow/providers/ftp/provider.yaml b/airflow/providers/ftp/provider.yaml
index 226caf7ca8..dcdeb06e7b 100644
--- a/airflow/providers/ftp/provider.yaml
+++ b/airflow/providers/ftp/provider.yaml
@@ -22,6 +22,7 @@ description: |
`File Transfer Protocol (FTP) <https://tools.ietf.org/html/rfc114>`__
suspended: false
+source-date-epoch: 1700148065
versions:
- 3.6.1
- 3.6.0
diff --git a/airflow/providers/github/provider.yaml b/airflow/providers/github/provider.yaml
index 962acdb067..5737413b19 100644
--- a/airflow/providers/github/provider.yaml
+++ b/airflow/providers/github/provider.yaml
@@ -29,6 +29,7 @@ dependencies:
- PyGithub!=1.58
suspended: false
+source-date-epoch: 1700148065
versions:
- 2.4.0
- 2.3.2
diff --git a/airflow/providers/google/provider.yaml b/airflow/providers/google/provider.yaml
index 4287801d2b..0f72165a30 100644
--- a/airflow/providers/google/provider.yaml
+++ b/airflow/providers/google/provider.yaml
@@ -29,6 +29,7 @@ description: |
- `Google Workspace <https://workspace.google.com/>`__ (formerly Google Suite)
suspended: false
+source-date-epoch: 1700148066
versions:
- 10.11.1
- 10.11.0
diff --git a/airflow/providers/grpc/provider.yaml b/airflow/providers/grpc/provider.yaml
index e5f3451748..9f6b19c3ca 100644
--- a/airflow/providers/grpc/provider.yaml
+++ b/airflow/providers/grpc/provider.yaml
@@ -22,6 +22,7 @@ description: |
`gRPC <https://grpc.io/>`__
suspended: false
+source-date-epoch: 1700148067
versions:
- 3.3.0
- 3.2.2
diff --git a/airflow/providers/hashicorp/provider.yaml b/airflow/providers/hashicorp/provider.yaml
index 68c26ccb6f..81974af832 100644
--- a/airflow/providers/hashicorp/provider.yaml
+++ b/airflow/providers/hashicorp/provider.yaml
@@ -22,6 +22,7 @@ description: |
Hashicorp including `Hashicorp Vault <https://www.vaultproject.io/>`__
suspended: false
+source-date-epoch: 1700148068
versions:
- 3.5.0
- 3.4.3
diff --git a/airflow/providers/http/provider.yaml b/airflow/providers/http/provider.yaml
index ab9ca16c06..183ecf1351 100644
--- a/airflow/providers/http/provider.yaml
+++ b/airflow/providers/http/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Hypertext Transfer Protocol (HTTP) <https://www.w3.org/Protocols/>`__
suspended: false
+source-date-epoch: 1700148068
versions:
- 4.7.0
- 4.6.0
diff --git a/airflow/providers/imap/provider.yaml b/airflow/providers/imap/provider.yaml
index eae2b360cb..9701bd392b 100644
--- a/airflow/providers/imap/provider.yaml
+++ b/airflow/providers/imap/provider.yaml
@@ -23,6 +23,7 @@ description: |
`Internet Message Access Protocol (IMAP) <https://tools.ietf.org/html/rfc3501>`__
suspended: false
+source-date-epoch: 1700148069
versions:
- 3.4.0
- 3.3.2
diff --git a/airflow/providers/influxdb/provider.yaml b/airflow/providers/influxdb/provider.yaml
index 0282c73931..9a8f105afb 100644
--- a/airflow/providers/influxdb/provider.yaml
+++ b/airflow/providers/influxdb/provider.yaml
@@ -29,6 +29,7 @@ dependencies:
- requests>=2.26.0
suspended: false
+source-date-epoch: 1700148070
versions:
- 2.3.0
- 2.2.3
diff --git a/airflow/providers/jdbc/provider.yaml b/airflow/providers/jdbc/provider.yaml
index 8948dfeb50..4182086dd3 100644
--- a/airflow/providers/jdbc/provider.yaml
+++ b/airflow/providers/jdbc/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Java Database Connectivity (JDBC) <https://docs.oracle.com/javase/8/docs/technotes/guides/jdbc/>`__
suspended: false
+source-date-epoch: 1700148070
versions:
- 4.1.0
- 4.0.2
diff --git a/airflow/providers/jenkins/provider.yaml b/airflow/providers/jenkins/provider.yaml
index 12538e796f..0ed5868dbf 100644
--- a/airflow/providers/jenkins/provider.yaml
+++ b/airflow/providers/jenkins/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Jenkins <https://jenkins.io/>`__
suspended: false
+source-date-epoch: 1700148071
versions:
- 3.4.0
- 3.3.2
diff --git a/airflow/providers/microsoft/azure/provider.yaml b/airflow/providers/microsoft/azure/provider.yaml
index 6679c8b4de..5d8b449bed 100644
--- a/airflow/providers/microsoft/azure/provider.yaml
+++ b/airflow/providers/microsoft/azure/provider.yaml
@@ -20,6 +20,7 @@ name: Microsoft Azure
description: |
`Microsoft Azure <https://azure.microsoft.com/>`__
suspended: false
+source-date-epoch: 1700148071
versions:
- 8.2.0
- 8.1.0
diff --git a/airflow/providers/microsoft/mssql/provider.yaml b/airflow/providers/microsoft/mssql/provider.yaml
index ef61b852d4..f0aa626c25 100644
--- a/airflow/providers/microsoft/mssql/provider.yaml
+++ b/airflow/providers/microsoft/mssql/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Microsoft SQL Server (MSSQL) <https://www.microsoft.com/en-us/sql-server/sql-server-downloads>`__
suspended: false
+source-date-epoch: 1700148072
versions:
- 3.5.0
- 3.4.2
diff --git a/airflow/providers/microsoft/psrp/provider.yaml b/airflow/providers/microsoft/psrp/provider.yaml
index bb283157a3..a7dc604433 100644
--- a/airflow/providers/microsoft/psrp/provider.yaml
+++ b/airflow/providers/microsoft/psrp/provider.yaml
@@ -24,6 +24,7 @@ description: |
<https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-psrp/>`__.
suspended: false
+source-date-epoch: 1700148073
versions:
- 2.4.0
- 2.3.2
diff --git a/airflow/providers/microsoft/winrm/provider.yaml b/airflow/providers/microsoft/winrm/provider.yaml
index c56b2b6dcf..7fbd5f8cc3 100644
--- a/airflow/providers/microsoft/winrm/provider.yaml
+++ b/airflow/providers/microsoft/winrm/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Windows Remote Management (WinRM) <https://docs.microsoft.com/en-us/windows/win32/winrm/portal>`__
suspended: false
+source-date-epoch: 1700148073
versions:
- 3.3.0
- 3.2.2
diff --git a/airflow/providers/mongo/provider.yaml b/airflow/providers/mongo/provider.yaml
index 53a441627c..32834c5587 100644
--- a/airflow/providers/mongo/provider.yaml
+++ b/airflow/providers/mongo/provider.yaml
@@ -22,6 +22,7 @@ description: |
`MongoDB <https://www.mongodb.com/what-is-mongodb>`__
suspended: false
+source-date-epoch: 1700148074
versions:
- 3.4.0
- 3.3.0
diff --git a/airflow/providers/mysql/provider.yaml b/airflow/providers/mysql/provider.yaml
index 7b73fef4f7..b2a744b8e6 100644
--- a/airflow/providers/mysql/provider.yaml
+++ b/airflow/providers/mysql/provider.yaml
@@ -22,6 +22,7 @@ description: |
`MySQL <https://www.mysql.com/products/>`__
suspended: false
+source-date-epoch: 1700148074
versions:
- 5.4.0
- 5.3.1
diff --git a/airflow/providers/neo4j/provider.yaml b/airflow/providers/neo4j/provider.yaml
index b00407208c..c9f5d9457c 100644
--- a/airflow/providers/neo4j/provider.yaml
+++ b/airflow/providers/neo4j/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Neo4j <https://neo4j.com/>`__
suspended: false
+source-date-epoch: 1700148075
versions:
- 3.4.0
- 3.3.3
diff --git a/airflow/providers/odbc/provider.yaml b/airflow/providers/odbc/provider.yaml
index 513d089b22..21facc2fd0 100644
--- a/airflow/providers/odbc/provider.yaml
+++ b/airflow/providers/odbc/provider.yaml
@@ -22,6 +22,7 @@ description: |
`ODBC <https://github.com/mkleehammer/pyodbc/wiki>`__
suspended: false
+source-date-epoch: 1700148076
versions:
- 4.1.0
- 4.0.0
diff --git a/airflow/providers/openai/provider.yaml b/airflow/providers/openai/provider.yaml
index ffb07de7bb..8884108ca6 100644
--- a/airflow/providers/openai/provider.yaml
+++ b/airflow/providers/openai/provider.yaml
@@ -24,6 +24,7 @@ description: |
`OpenAI <https://platform.openai.com/docs/introduction>`__
suspended: false
+source-date-epoch: 1700148076
versions:
- 1.0.0
diff --git a/airflow/providers/openfaas/provider.yaml b/airflow/providers/openfaas/provider.yaml
index 3eb521d163..3c6b73f514 100644
--- a/airflow/providers/openfaas/provider.yaml
+++ b/airflow/providers/openfaas/provider.yaml
@@ -22,6 +22,7 @@ description: |
`OpenFaaS <https://www.openfaas.com/>`__
suspended: false
+source-date-epoch: 1700148077
versions:
- 3.3.0
- 3.2.1
diff --git a/airflow/providers/openlineage/provider.yaml b/airflow/providers/openlineage/provider.yaml
index 5dd878d5fd..8fed707c37 100644
--- a/airflow/providers/openlineage/provider.yaml
+++ b/airflow/providers/openlineage/provider.yaml
@@ -22,6 +22,7 @@ description: |
`OpenLineage <https://openlineage.io/>`__
suspended: false
+source-date-epoch: 1700148077
versions:
- 1.2.1
- 1.2.0
diff --git a/airflow/providers/opensearch/provider.yaml b/airflow/providers/opensearch/provider.yaml
index 270d7cd208..a0bddd79d5 100644
--- a/airflow/providers/opensearch/provider.yaml
+++ b/airflow/providers/opensearch/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Opensearch <https://opensearch.org/>`__
suspended: false
+source-date-epoch: 1700148078
versions:
- 1.0.0
diff --git a/airflow/providers/opsgenie/provider.yaml b/airflow/providers/opsgenie/provider.yaml
index da89196f34..d60f55f6d7 100644
--- a/airflow/providers/opsgenie/provider.yaml
+++ b/airflow/providers/opsgenie/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Opsgenie <https://www.opsgenie.com/>`__
suspended: false
+source-date-epoch: 1700148078
versions:
- 5.2.0
- 5.1.1
diff --git a/airflow/providers/oracle/provider.yaml b/airflow/providers/oracle/provider.yaml
index 3befc752e6..2a95c4138a 100644
--- a/airflow/providers/oracle/provider.yaml
+++ b/airflow/providers/oracle/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Oracle <https://www.oracle.com/en/database/>`__
suspended: false
+source-date-epoch: 1700148079
versions:
- 3.8.0
- 3.7.4
diff --git a/airflow/providers/pagerduty/provider.yaml b/airflow/providers/pagerduty/provider.yaml
index 1a03a52dee..8d01628236 100644
--- a/airflow/providers/pagerduty/provider.yaml
+++ b/airflow/providers/pagerduty/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Pagerduty <https://www.pagerduty.com/>`__
suspended: false
+source-date-epoch: 1700148079
versions:
- 3.4.0
- 3.3.1
diff --git a/airflow/providers/papermill/provider.yaml b/airflow/providers/papermill/provider.yaml
index c85fde1677..8a7155ef58 100644
--- a/airflow/providers/papermill/provider.yaml
+++ b/airflow/providers/papermill/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Papermill <https://github.com/nteract/papermill>`__
suspended: false
+source-date-epoch: 1700148080
versions:
- 3.4.0
- 3.2.1
diff --git a/airflow/providers/pgvector/provider.yaml b/airflow/providers/pgvector/provider.yaml
index 1fedfe141b..37a6471994 100644
--- a/airflow/providers/pgvector/provider.yaml
+++ b/airflow/providers/pgvector/provider.yaml
@@ -24,6 +24,7 @@ description: |
`pgvector <https://github.com/pgvector/pgvector>`__
suspended: false
+source-date-epoch: 1700148080
versions:
- 1.0.0
diff --git a/airflow/providers/pinecone/provider.yaml b/airflow/providers/pinecone/provider.yaml
index f3050cfe22..853c223ae6 100644
--- a/airflow/providers/pinecone/provider.yaml
+++ b/airflow/providers/pinecone/provider.yaml
@@ -24,6 +24,7 @@ description: |
`Pinecone <https://docs.pinecone.io/docs/overview>`__
suspended: false
+source-date-epoch: 1700148081
versions:
- 1.0.0
diff --git a/airflow/providers/plexus/provider.yaml b/airflow/providers/plexus/provider.yaml
index bb25339651..70d2251e8f 100644
--- a/airflow/providers/plexus/provider.yaml
+++ b/airflow/providers/plexus/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Plexus <https://plexus.corescientific.com/>`__
suspended: false
+source-date-epoch: 1700148081
versions:
- 3.3.0
- 3.2.2
diff --git a/airflow/providers/postgres/provider.yaml b/airflow/providers/postgres/provider.yaml
index 78e716bfca..4afc8bc695 100644
--- a/airflow/providers/postgres/provider.yaml
+++ b/airflow/providers/postgres/provider.yaml
@@ -22,6 +22,7 @@ description: |
`PostgreSQL <https://www.postgresql.org/>`__
suspended: false
+source-date-epoch: 1700148082
versions:
- 5.8.0
- 5.7.1
diff --git a/airflow/providers/presto/provider.yaml b/airflow/providers/presto/provider.yaml
index 107375a3c8..fa20f97a7d 100644
--- a/airflow/providers/presto/provider.yaml
+++ b/airflow/providers/presto/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Presto <https://prestodb.github.io/>`__
suspended: false
+source-date-epoch: 1700148082
versions:
- 5.2.1
- 5.2.0
diff --git a/airflow/providers/redis/provider.yaml b/airflow/providers/redis/provider.yaml
index 486555ea54..052fc17a18 100644
--- a/airflow/providers/redis/provider.yaml
+++ b/airflow/providers/redis/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Redis <https://redis.io/>`__
suspended: false
+source-date-epoch: 1700148083
versions:
- 3.4.0
- 3.3.2
diff --git a/airflow/providers/salesforce/provider.yaml b/airflow/providers/salesforce/provider.yaml
index a494a6dc5c..e68d51ab0b 100644
--- a/airflow/providers/salesforce/provider.yaml
+++ b/airflow/providers/salesforce/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Salesforce <https://www.salesforce.com/>`__
suspended: false
+source-date-epoch: 1700148084
versions:
- 5.5.1
- 5.5.0
diff --git a/airflow/providers/samba/provider.yaml b/airflow/providers/samba/provider.yaml
index 27d673e8e3..0822edb681 100644
--- a/airflow/providers/samba/provider.yaml
+++ b/airflow/providers/samba/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Samba <https://www.samba.org/>`__
suspended: false
+source-date-epoch: 1700148084
versions:
- 4.3.0
- 4.2.2
diff --git a/airflow/providers/segment/provider.yaml b/airflow/providers/segment/provider.yaml
index 22b17d9c33..81e68982f6 100644
--- a/airflow/providers/segment/provider.yaml
+++ b/airflow/providers/segment/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Segment <https://segment.com/>`__
suspended: false
+source-date-epoch: 1700148085
versions:
- 3.3.0
- 3.2.1
diff --git a/airflow/providers/sendgrid/provider.yaml b/airflow/providers/sendgrid/provider.yaml
index f1e7e90c8f..e897e7af8c 100644
--- a/airflow/providers/sendgrid/provider.yaml
+++ b/airflow/providers/sendgrid/provider.yaml
@@ -26,6 +26,7 @@ dependencies:
- sendgrid>=6.0.0
suspended: false
+source-date-epoch: 1700148085
versions:
- 3.3.0
- 3.2.2
diff --git a/airflow/providers/sftp/provider.yaml b/airflow/providers/sftp/provider.yaml
index 073700ff35..af2be40c4a 100644
--- a/airflow/providers/sftp/provider.yaml
+++ b/airflow/providers/sftp/provider.yaml
@@ -22,6 +22,7 @@ description: |
`SSH File Transfer Protocol (SFTP) <https://tools.ietf.org/wg/secsh/draft-ietf-secsh-filexfer/>`__
suspended: false
+source-date-epoch: 1700148086
versions:
- 4.7.0
- 4.6.1
diff --git a/airflow/providers/singularity/provider.yaml b/airflow/providers/singularity/provider.yaml
index 443ed1a98a..52a83b896a 100644
--- a/airflow/providers/singularity/provider.yaml
+++ b/airflow/providers/singularity/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Singularity <https://sylabs.io/guides/latest/user-guide/>`__
suspended: false
+source-date-epoch: 1700148086
versions:
- 3.3.0
- 3.2.2
diff --git a/airflow/providers/slack/provider.yaml b/airflow/providers/slack/provider.yaml
index e1283fde7b..73b7dc3e4e 100644
--- a/airflow/providers/slack/provider.yaml
+++ b/airflow/providers/slack/provider.yaml
@@ -25,6 +25,7 @@ description: |
- `Slack Incoming Webhook <https://api.slack.com/messaging/webhooks>`__
suspended: false
+source-date-epoch: 1700148087
versions:
- 8.4.0
- 8.3.0
diff --git a/airflow/providers/smtp/provider.yaml b/airflow/providers/smtp/provider.yaml
index 8ff02eb9fa..096eeed141 100644
--- a/airflow/providers/smtp/provider.yaml
+++ b/airflow/providers/smtp/provider.yaml
@@ -23,6 +23,7 @@ description: |
`Simple Mail Transfer Protocol (SMTP) <https://tools.ietf.org/html/rfc5321>`__
suspended: false
+source-date-epoch: 1700148088
versions:
- 1.4.1
- 1.4.0
diff --git a/airflow/providers/snowflake/provider.yaml b/airflow/providers/snowflake/provider.yaml
index bdc5b3ef07..026ec5c14d 100644
--- a/airflow/providers/snowflake/provider.yaml
+++ b/airflow/providers/snowflake/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Snowflake <https://www.snowflake.com/>`__
suspended: false
+source-date-epoch: 1700148088
versions:
- 5.1.1
- 5.1.0
diff --git a/airflow/providers/sqlite/provider.yaml b/airflow/providers/sqlite/provider.yaml
index 6bde2aca1c..4a20baf342 100644
--- a/airflow/providers/sqlite/provider.yaml
+++ b/airflow/providers/sqlite/provider.yaml
@@ -22,6 +22,7 @@ description: |
`SQLite <https://www.sqlite.org/>`__
suspended: false
+source-date-epoch: 1700148089
versions:
- 3.5.0
- 3.4.3
diff --git a/airflow/providers/ssh/provider.yaml b/airflow/providers/ssh/provider.yaml
index 0944c7dda3..0330f00b94 100644
--- a/airflow/providers/ssh/provider.yaml
+++ b/airflow/providers/ssh/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Secure Shell (SSH) <https://tools.ietf.org/html/rfc4251>`__
suspended: false
+source-date-epoch: 1700148090
versions:
- 3.8.1
- 3.8.0
diff --git a/airflow/providers/tableau/provider.yaml b/airflow/providers/tableau/provider.yaml
index ef591d060b..8e456d3140 100644
--- a/airflow/providers/tableau/provider.yaml
+++ b/airflow/providers/tableau/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Tableau <https://www.tableau.com/>`__
suspended: false
+source-date-epoch: 1700148090
versions:
- 4.3.0
- 4.2.2
diff --git a/airflow/providers/tabular/provider.yaml b/airflow/providers/tabular/provider.yaml
index 267068b71d..1fc5dc5770 100644
--- a/airflow/providers/tabular/provider.yaml
+++ b/airflow/providers/tabular/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Tabular <https://tabular.io/>`__
suspended: false
+source-date-epoch: 1700148091
versions:
- 1.3.0
- 1.2.1
diff --git a/airflow/providers/telegram/provider.yaml b/airflow/providers/telegram/provider.yaml
index 36c59720f9..8241dc4656 100644
--- a/airflow/providers/telegram/provider.yaml
+++ b/airflow/providers/telegram/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Telegram <https://telegram.org/>`__
suspended: false
+source-date-epoch: 1700148091
versions:
- 4.2.0
- 4.1.1
diff --git a/airflow/providers/trino/provider.yaml b/airflow/providers/trino/provider.yaml
index 562304635a..5ae593b9e1 100644
--- a/airflow/providers/trino/provider.yaml
+++ b/airflow/providers/trino/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Trino <https://trino.io/>`__
suspended: false
+source-date-epoch: 1700148092
versions:
- 5.4.0
- 5.3.1
diff --git a/airflow/providers/vertica/provider.yaml b/airflow/providers/vertica/provider.yaml
index 09ef356ecd..f809a8a1ba 100644
--- a/airflow/providers/vertica/provider.yaml
+++ b/airflow/providers/vertica/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Vertica <https://www.vertica.com/>`__
suspended: false
+source-date-epoch: 1700148092
versions:
- 3.6.0
- 3.5.2
diff --git a/airflow/providers/weaviate/provider.yaml b/airflow/providers/weaviate/provider.yaml
index 94224031da..7301f5021c 100644
--- a/airflow/providers/weaviate/provider.yaml
+++ b/airflow/providers/weaviate/provider.yaml
@@ -24,6 +24,7 @@ description: |
`Weaviate <https://weaviate.io/developers/weaviate>`__
suspended: false
+source-date-epoch: 1700148093
versions:
- 1.0.0
diff --git a/airflow/providers/yandex/provider.yaml b/airflow/providers/yandex/provider.yaml
index 8073992a25..c275924076 100644
--- a/airflow/providers/yandex/provider.yaml
+++ b/airflow/providers/yandex/provider.yaml
@@ -21,6 +21,7 @@ name: Yandex
description: |
Yandex including `Yandex.Cloud <https://cloud.yandex.com/>`__
suspended: false
+source-date-epoch: 1700148093
versions:
- 3.6.0
- 3.5.0
diff --git a/airflow/providers/zendesk/provider.yaml b/airflow/providers/zendesk/provider.yaml
index 04fa4423c9..25f03bd351 100644
--- a/airflow/providers/zendesk/provider.yaml
+++ b/airflow/providers/zendesk/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Zendesk <https://www.zendesk.com/>`__
suspended: false
+source-date-epoch: 1700148094
versions:
- 4.4.0
- 4.3.2
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index cdc40645ab..fe3494541b 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -481,7 +481,6 @@ def prepare_provider_packages(
cleanup_build_remnants(target_provider_root_sources_path)
build_provider_package(
provider_id=provider_id,
- version_suffix=version_suffix_for_pypi,
package_format=package_format,
target_provider_root_sources_path=target_provider_root_sources_path,
)
diff --git a/dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py b/dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py
index 3c0370f33e..0193f9e0ab 100644
--- a/dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py
+++ b/dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py
@@ -28,6 +28,7 @@ from copy import deepcopy
from enum import Enum
from pathlib import Path
from shutil import copyfile
+from time import time
from typing import Any, NamedTuple
import jinja2
@@ -42,8 +43,9 @@ from airflow_breeze.utils.packages import (
ProviderPackageDetails,
get_provider_details,
get_provider_jinja_context,
- get_provider_packages_metadata,
get_source_package_path,
+ refresh_provider_metadata_from_yaml_file,
+ refresh_provider_metadata_with_provider_id,
render_template,
)
from airflow_breeze.utils.run_utils import run_command
@@ -458,13 +460,29 @@ def _update_version_in_provider_yaml(
original_text = provider_yaml_path.read_text()
new_text = re.sub(r"versions:", f"versions:\n - {v}", original_text, 1)
provider_yaml_path.write_text(new_text)
- # IMPORTANT!!! Whenever we update provider.yaml files, we MUST clear cache for
- # get_provider_packages_metadata function, because otherwise anything next will not use it
- get_provider_packages_metadata.cache_clear()
get_console().print(f"[special]Bumped version to {v}\n")
return with_breaking_changes, maybe_with_new_features
+def _update_source_date_epoch_in_provider_yaml(
+ provider_package_id: str,
+) -> None:
+ """
+ Updates source date epoch in provider yaml that then can be used to generate reproducible packages.
+
+ :param provider_package_id: provider package
+ """
+ provider_yaml_path = get_source_package_path(provider_package_id) / "provider.yaml"
+ original_text = provider_yaml_path.read_text()
+ source_date_epoch = int(time())
+ new_text = re.sub(
+ r"source-date-epoch: [0-9]*", f"source-date-epoch: {source_date_epoch}", original_text, 1
+ )
+ provider_yaml_path.write_text(new_text)
+ refresh_provider_metadata_with_provider_id(provider_package_id)
+ get_console().print(f"[special]Updated source-date-epoch to {source_date_epoch}\n")
+
+
def _verify_changelog_exists(package: str) -> Path:
provider_details = get_provider_details(package)
changelog_path = Path(provider_details.source_provider_package_path) / "CHANGELOG.rst"
@@ -687,11 +705,14 @@ def update_release_notes(
with_breaking_changes, maybe_with_new_features = _update_version_in_provider_yaml(
provider_package_id=provider_package_id, type_of_change=type_of_change
)
+ _update_source_date_epoch_in_provider_yaml(provider_package_id)
proceed, list_of_list_of_changes, changes_as_table = _get_all_changes_for_package(
provider_package_id=provider_package_id,
base_branch=base_branch,
reapply_templates_only=reapply_templates_only,
)
+ else:
+ _update_source_date_epoch_in_provider_yaml(provider_package_id)
provider_details = get_provider_details(provider_package_id)
_verify_changelog_exists(provider_details.provider_id)
jinja_context = get_provider_documentation_jinja_context(
@@ -951,9 +972,7 @@ def _replace_min_airflow_version_in_provider_yaml(
provider_yaml_txt,
)
provider_yaml_path.write_text(provider_yaml_txt)
- # IMPORTANT!!! Whenever we update provider.yaml files, we MUST clear cache for
- # get_provider_packages_metadata function, because otherwise anything next will not use it
- get_provider_packages_metadata.cache_clear()
+ refresh_provider_metadata_from_yaml_file(provider_yaml_path)
def update_min_airflow_version(
diff --git a/dev/breeze/src/airflow_breeze/prepare_providers/provider_packages.py b/dev/breeze/src/airflow_breeze/prepare_providers/provider_packages.py
index 5b004eb27d..2f3b7eface 100644
--- a/dev/breeze/src/airflow_breeze/prepare_providers/provider_packages.py
+++ b/dev/breeze/src/airflow_breeze/prepare_providers/provider_packages.py
@@ -22,7 +22,6 @@ import subprocess
import sys
from pathlib import Path
from shutil import copytree, rmtree
-from stat import S_IRGRP, S_IROTH, S_IRUSR, S_IWUSR
from typing import IO, Any
from airflow_breeze.utils.console import get_console
@@ -179,9 +178,7 @@ def cleanup_build_remnants(target_provider_root_sources_path: Path):
get_console().print(f"[info]Cleaned remnants in {target_provider_root_sources_path}\n")
-def build_provider_package(
- provider_id: str, version_suffix: str, target_provider_root_sources_path: Path, package_format: str
-):
+def build_provider_package(provider_id: str, target_provider_root_sources_path: Path, package_format: str):
get_console().print(
f"\n[info]Building provider package: {provider_id} in format {package_format} in "
f"{target_provider_root_sources_path}\n"
@@ -190,7 +187,14 @@ def build_provider_package(
if package_format != "both":
command.extend(["--format", package_format])
try:
- run_command(command, check=True, cwd=target_provider_root_sources_path)
+ run_command(
+ command,
+ check=True,
+ cwd=target_provider_root_sources_path,
+ env={
+ "SOURCE_DATE_EPOCH": str(get_provider_details(provider_id).source_date_epoch),
+ },
+ )
except subprocess.CalledProcessError as ex:
get_console().print("[error]The command returned an error %s", ex)
raise PrepareReleasePackageErrorBuildingPackageException()
@@ -207,8 +211,6 @@ def move_built_packages_and_cleanup(
# Shutil can move packages also between filesystems
target_file = dist_folder / file.name
target_file.unlink(missing_ok=True)
- # Change ownership to group/other to read the file
- file.chmod(S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH)
shutil.move(file.as_posix(), dist_folder.as_posix())
if skip_cleanup:
diff --git a/dev/breeze/src/airflow_breeze/utils/packages.py b/dev/breeze/src/airflow_breeze/utils/packages.py
index df2c4b1d51..79904f3c73 100644
--- a/dev/breeze/src/airflow_breeze/utils/packages.py
+++ b/dev/breeze/src/airflow_breeze/utils/packages.py
@@ -84,6 +84,7 @@ class PluginInfo(NamedTuple):
class ProviderPackageDetails(NamedTuple):
provider_id: str
+ source_date_epoch: int
full_package_name: str
pypi_package_name: str
source_provider_package_path: Path
@@ -130,28 +131,41 @@ class PipRequirements(NamedTuple):
return cls(package=package, version_required=version_required.strip())
+PROVIDER_METADATA: dict[str, dict[str, Any]] = {}
+
+
+def refresh_provider_metadata_from_yaml_file(provider_yaml_path: Path):
+ import jsonschema
+ import yaml
+
+ schema = _load_schema()
+ with open(provider_yaml_path) as yaml_file:
+ provider = yaml.safe_load(yaml_file)
+ try:
+ jsonschema.validate(provider, schema=schema)
+ except jsonschema.ValidationError:
+ raise Exception(f"Unable to parse: {provider_yaml_path}.")
+ PROVIDER_METADATA[get_short_package_name(provider["package-name"])] = provider
+
+
+def refresh_provider_metadata_with_provider_id(provider_id: str):
+ provider_yaml_path = get_source_package_path(provider_id) / "provider.yaml"
+ refresh_provider_metadata_from_yaml_file(provider_yaml_path)
+
-@lru_cache
def get_provider_packages_metadata() -> dict[str, dict[str, Any]]:
"""
Load all data from providers files
:return: A list containing the contents of all provider.yaml files.
"""
- import jsonschema
- import yaml
- schema = _load_schema()
- result: dict[str, dict[str, Any]] = {}
+ if PROVIDER_METADATA:
+ return PROVIDER_METADATA
+
for provider_yaml_path in get_provider_yaml_paths():
- with open(provider_yaml_path) as yaml_file:
- provider = yaml.safe_load(yaml_file)
- try:
- jsonschema.validate(provider, schema=schema)
- except jsonschema.ValidationError:
- raise Exception(f"Unable to parse: {provider_yaml_path}.")
- result[get_short_package_name(provider["package-name"])] = provider
- return result
+ refresh_provider_metadata_from_yaml_file(provider_yaml_path)
+ return PROVIDER_METADATA
def validate_provider_info_with_runtime_schema(provider_info: dict[str, Any]) -> None:
@@ -440,6 +454,7 @@ def get_provider_details(provider_id: str) -> ProviderPackageDetails:
)
return ProviderPackageDetails(
provider_id=provider_id,
+ source_date_epoch=provider_info["source-date-epoch"],
full_package_name=f"airflow.providers.{provider_id}",
pypi_package_name=f"apache-airflow-providers-{provider_id.replace('.', '-')}",
source_provider_package_path=get_source_package_path(provider_id),
diff --git a/dev/refresh_images.sh b/dev/refresh_images.sh
index 638e86930c..2b11229aa0 100755
--- a/dev/refresh_images.sh
+++ b/dev/refresh_images.sh
@@ -42,7 +42,7 @@ breeze release-management prepare-provider-packages \
breeze release-management prepare-airflow-package --package-format wheel --version-suffix-for-pypi dev0
-mv -v ./dist/*.whl ./docker-context-files
+mv -v ./dist/*.whl ./docker-context-files && chmod a+r ./docker-context-files/*
breeze prod-image build \
--builder airflow_cache \