(airflow) branch main updated: Get rid of pyarrow-hotfix for CVE-2023-47248 (#36697)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new d105c7115f Get rid of pyarrow-hotfix for CVE-2023-47248 (#36697)
d105c7115f is described below

commit d105c7115f56f88d48a2888484a0ed7d1c01576f
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Jan 9 21:41:45 2024 +0100

    Get rid of pyarrow-hotfix for CVE-2023-47248 (#36697)
    
    The #35650 introduced a hotfix for Pyarrow CVE-2023-47248. So far
    we have been blocked from removing it by Apache Beam that limited
    Airflow from bumping pyarrow to a version that was not vulnerable.
    
    This is now possible since Apache Beam relesed 2.53.0 version on
    4th of January 2023 that allows to use non-vulnerable pyarrow.
    
    We are now bumping both Pyarrow and Beam minimum versions to
    reflect that and remove pyarrow hotfix.
---
 airflow/providers/apache/beam/provider.yaml | 4 +++-
 generated/provider_dependencies.json        | 3 ++-
 setup.py                                    | 4 ----
 3 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/airflow/providers/apache/beam/provider.yaml b/airflow/providers/apache/beam/provider.yaml
index 0facc9cd45..ddf8100ede 100644
--- a/airflow/providers/apache/beam/provider.yaml
+++ b/airflow/providers/apache/beam/provider.yaml
@@ -52,7 +52,9 @@ versions:
 
 dependencies:
   - apache-airflow>=2.6.0
-  - apache-beam>=2.47.0
+  # Apache Beam > 2.53.0 and pyarrow > 14.0.1 fix https://nvd.nist.gov/vuln/detail/CVE-2023-47248.
+  - apache-beam>=2.53.0
+  - pyarrow>=14.0.1
 
 integrations:
   - integration-name: Apache Beam
diff --git a/generated/provider_dependencies.json b/generated/provider_dependencies.json
index a0813d7979..cbbb567a52 100644
--- a/generated/provider_dependencies.json
+++ b/generated/provider_dependencies.json
@@ -56,7 +56,8 @@
   "apache.beam": {
     "deps": [
       "apache-airflow>=2.6.0",
-      "apache-beam>=2.47.0"
+      "apache-beam>=2.53.0",
+      "pyarrow>=14.0.1"
     ],
     "cross-providers-deps": [
       "google"
diff --git a/setup.py b/setup.py
index 047326f0c6..ae3848b069 100644
--- a/setup.py
+++ b/setup.py
@@ -351,10 +351,6 @@ leveldb = ["plyvel"]
 otel = ["opentelemetry-exporter-prometheus"]
 pandas = [
     "pandas>=0.17.1",
-    # Use pyarrow-hotfix to fix https://nvd.nist.gov/vuln/detail/CVE-2023-47248.
-    # We should remove it once Apache Beam frees us to upgrade to pyarrow 14.0.1
-    "pyarrow-hotfix",
-    "pyarrow>=9.0.0",
 ]
 password = [
     "bcrypt>=2.0.0",