You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dwreeves (via GitHub)" <gi...@apache.org> on 2026/04/28 14:59:52 UTC

[I] [Feat] [Slack Provider] Lazy import of Slack modules in snowflake/hooks/* [airflow]

dwreeves opened a new issue, #66025:
URL: https://github.com/apache/airflow/issues/66025

   ### Under which category would you file this issue?
   
   Providers
   
   ### Apache Airflow version
   
   2.10.5
   
   ### What happened and how to reproduce it?
   
   The `SlackWebhookHook` import uses an incredibly large amount of memory and occasionally causes my workers to OOM. I don't have this issue elsewhere on my workers.
   
   I've discovered that the import of `from slack_sdk.webhook.async_client import AsyncWebhookClient` causes most of the added overhead. I believe this may have to do with the import of `aiohttp`, which adds significant overhead.
   
   In most cases, users just need the sync client, not the async client. So lazy loading should hopefully cause this to OOM less frequently.
   
   ### What you think should happen instead?
   
   Lazy-loading of the Slack client reduces the amount of memory used by about 10% (relevant for global imports of the SlackWebhookHook at the scheduler level), and lazy loading the asyncio SDK is a 7% improvement (relevant for task execution that doesn't touch the asyncio SDK):
   
   ```
     Python: 3.12.11 (main, Jun  4 2025, 17:42:58) [Clang 20.1.4 ]
     Runs per scenario: 20  |  also benchmark w/o .pyc: no
   
            avg ms (±stdev)  |  avg RSS MB  —  20 runs each          
   ╭──────────────────────────┬───────────────────┬──────────────────╮
   │ Scenario                 │ Time w/ .pyc (ms) │ RSS w/ .pyc (MB) │
   ├──────────────────────────┼───────────────────┼──────────────────┤
   │ Base imports             │           966 ±60 │             86.5 │
   │ Base imports + SDK       │   1026 ±98  (+60) │     88.3  (+1.8) │
   │ Base imports + Async SDK │  1124 ±77  (+158) │     94.9  (+8.4) │
   ╰──────────────────────────┴───────────────────┴──────────────────╯
      (+N) = delta vs first scenario  |  ±N = stdev across runs
   ```
   
   The delta is a little smaller than I would have expected given how often jobs associated with the SlackWebhookHook OOMs. Still, this is a pretty marked improvement from just lazy loading some modules.
   
   ### Operating System
   
   _No response_
   
   ### Deployment
   
   None
   
   ### Apache Airflow Provider(s)
   
   _No response_
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow==2.10.5
   apache-airflow-providers-common-compat==1.7.0
   apache-airflow-providers-common-io==1.6.0
   apache-airflow-providers-common-sql==1.27.1
   apache-airflow-providers-slack==9.1.0
   
   
   ### Official Helm Chart version
   
   Not Applicable
   
   ### Kubernetes Version
   
   _No response_
   
   ### Helm Chart configuration
   
   _No response_
   
   ### Docker Image customizations
   
   _No response_
   
   ### Anything else?
   
   You can run the benchmark with my code below (note: AI generated).
   
   ```python
   #!/usr/bin/env python3
   # NOTE: AI generated script.
   # /// script
   # requires-python = ">=3.11"
   # dependencies = [
   #   "apache-airflow",
   #   "apache-airflow-providers-slack",
   #   "rich-click",
   #   "rich",
   #   "requests",
   #   "tenacity",
   #   "cryptography",
   #   "slack-sdk",
   #   "aiohttp",
   #   "sqlalchemy",
   # ]
   # ///
   from __future__ import annotations
   
   import os
   import shutil
   import statistics
   import subprocess
   import sys
   import tempfile
   from dataclasses import dataclass, field
   
   import rich_click as click
   from rich import box
   from rich.console import Console
   from rich.table import Table
   from rich.text import Text
   
   console = Console()
   
   BASE_IMPORTS = """\
   import json
   import warnings
   from collections.abc import Callable
   from functools import cached_property, wraps
   from typing import TYPE_CHECKING, Any
   
   from airflow.providers.common.compat.sdk import AirflowException, AirflowNotFoundException, BaseHook
   """
   
   BASE_IMPORTS_PLUS_SDK = BASE_IMPORTS + """
   from slack_sdk import WebhookClient
   """
   
   # effectively equivalent to:
   # from airflow.providers.slack.hooks.slack_webhook import *
   BASE_IMPORTS_PLUS_ASYNC_SDK = BASE_IMPORTS_PLUS_SDK + BASE_IMPORTS + """
   from airflow.providers.common.compat.connection import get_async_connection
   from slack_sdk.webhook.async_client import AsyncWebhookClient
   
   # ^ effectively equivalent to:
   from airflow.providers.slack.hooks.slack_webhook import *
   """
   
   SCENARIOS = [
       ("Base imports", BASE_IMPORTS),
       ("Base imports + SDK", BASE_IMPORTS_PLUS_SDK),
       ("Base import + Async SDK", BASE_IMPORTS_PLUS_ASYNC_SDK),
   ]
   
   
   @dataclass
   class RunStats:
       times_ms: list[float] = field(default_factory=list)
       rss_mb: list[float] = field(default_factory=list)
   
       @property
       def avg_ms(self) -> float:
           return statistics.mean(self.times_ms)
   
       @property
       def avg_rss(self) -> float:
           return statistics.mean(self.rss_mb)
   
       @property
       def stdev_ms(self) -> float:
           return statistics.stdev(self.times_ms) if len(self.times_ms) > 1 else 0.0
   
   
   @dataclass
   class ScenarioResult:
       label: str
       with_pyc: RunStats
       without_pyc: RunStats | None = None
   
   
   SUBPROCESS_CODE_TEMPLATE = """\
   import time, os, sys
   import psutil
   
   proc = psutil.Process(os.getpid())
   rss_before = proc.memory_info().rss
   t0 = time.perf_counter()
   try:
   {indented}
       elapsed = time.perf_counter() - t0
       rss_after = proc.memory_info().rss
       print(f"{{elapsed:.6f}} {{(rss_after - rss_before) / 1024 / 1024:.4f}}")
   except Exception as e:
       print(f"ERROR: {{e}}", file=sys.stderr)
       sys.exit(1)
   """
   
   
   def _single_run(import_block: str, use_pyc: bool) -> tuple[float, float]:
       """Spawn a fresh interpreter and return (elapsed_ms, rss_mb)."""
       indented = "\n".join("    " + line for line in import_block.splitlines())
       code = SUBPROCESS_CODE_TEMPLATE.format(indented=indented)
   
       env = None
       tmp_dir = None
       if not use_pyc:
           # Redirect bytecode cache to a fresh empty dir so Python finds no .pyc
           # files and must compile every module from source.
           tmp_dir = tempfile.mkdtemp(prefix="no_pyc_bench_")
           env = {**os.environ, "PYTHONPYCACHEPREFIX": tmp_dir}
   
       try:
           result = subprocess.run(
               [sys.executable, "-c", code],
               capture_output=True,
               text=True,
               env=env,
           )
       finally:
           if tmp_dir:
               shutil.rmtree(tmp_dir, ignore_errors=True)
   
       if result.returncode != 0:
           console.print(f"[red]ERROR:[/] {result.stderr.strip()}")
           sys.exit(1)
   
       elapsed_s, rss_mb = result.stdout.strip().split()
       return float(elapsed_s) * 1000, float(rss_mb)
   
   
   def measure(label: str, import_block: str, n: int, include_no_pyc: bool) -> ScenarioResult:
       with_pyc = RunStats()
       without_pyc = RunStats() if include_no_pyc else None
   
       # Warm-up run to ensure .pyc files are written before timed runs.
       with console.status(f"  [italic]{label}[/]  warming up .pyc cache…", spinner="dots"):
           _single_run(import_block, use_pyc=True)
   
       for i in range(n):
           with console.status(f"  [italic]{label}[/]  run {i + 1}/{n} (w/ .pyc)…", spinner="dots"):
               ms, mb = _single_run(import_block, use_pyc=True)
           with_pyc.times_ms.append(ms)
           with_pyc.rss_mb.append(mb)
   
       if include_no_pyc:
           for i in range(n):
               with console.status(f"  [italic]{label}[/]  run {i + 1}/{n} (w/o .pyc)…", spinner="dots"):
                   ms, mb = _single_run(import_block, use_pyc=False)
               without_pyc.times_ms.append(ms)  # type: ignore[union-attr]
               without_pyc.rss_mb.append(mb)  # type: ignore[union-attr]
   
       return ScenarioResult(label=label, with_pyc=with_pyc, without_pyc=without_pyc)
   
   
   def _time_cell(stats: RunStats, baseline_ms: float) -> Text:
       t = Text(f"{stats.avg_ms:.0f}", style="white")
       t.append(f" ±{stats.stdev_ms:.0f}", style="dim")
       if stats.avg_ms != baseline_ms:
           t.append(f"  (+{stats.avg_ms - baseline_ms:.0f})", style="dim cyan")
       return t
   
   
   def _rss_cell(stats: RunStats, baseline_mb: float) -> Text:
       t = Text(f"{stats.avg_rss:.1f}", style="white")
       if stats.avg_rss != baseline_mb:
           t.append(f"  (+{stats.avg_rss - baseline_mb:.1f})", style="dim cyan")
       return t
   
   
   @click.command()
   @click.option(
       "-n", "--runs",
       default=10,
       show_default=True,
       help="Number of cold-cache subprocess runs to average per scenario.",
   )
   @click.option(
       "--no-pyc",
       is_flag=True,
       default=False,
       help="Also run benchmarks without .pyc bytecode caches.",
   )
   @click.rich_config({"theme": "cargo-slim"})
   def main(runs: int, no_pyc: bool) -> None:
       console.print()
       console.rule("[bold cyan]Snowflake Hook — Import Cost Benchmark[/]")
       console.print(f"  [dim]Python: {sys.version}[/]")
       console.print(f"  [dim]Runs per scenario: {runs}  |  also benchmark w/o .pyc: {'yes' if no_pyc else 'no'}[/]")
   
       console.print()
   
       results = [measure(label, block, runs, no_pyc) for label, block in SCENARIOS]
   
       table = Table(
           box=box.ROUNDED,
           show_footer=False,
           title_style="bold cyan",
           border_style="grey50",
           title=f"avg ms (±stdev)  |  avg RSS MB  —  {runs} runs each",
       )
       table.add_column("Scenario", style="white", no_wrap=True)
       table.add_column("Time w/ .pyc (ms)", justify="right")
       table.add_column("RSS w/ .pyc (MB)", justify="right")
       if no_pyc:
           table.add_column("Time w/o .pyc (ms)", justify="right")
           table.add_column("RSS w/o .pyc (MB)", justify="right")
   
       baseline_pyc_ms = results[0].with_pyc.avg_ms
       baseline_pyc_mb = results[0].with_pyc.avg_rss
       baseline_nopyc_ms = results[0].without_pyc.avg_ms if no_pyc else 0.0
       baseline_nopyc_mb = results[0].without_pyc.avg_rss if no_pyc else 0.0
   
       for r in results:
           row = [
               r.label,
               _time_cell(r.with_pyc, baseline_pyc_ms),
               _rss_cell(r.with_pyc, baseline_pyc_mb),
           ]
           if no_pyc and r.without_pyc is not None:
               row += [
                   _time_cell(r.without_pyc, baseline_nopyc_ms),
                   _rss_cell(r.without_pyc, baseline_nopyc_mb),
               ]
           table.add_row(*row)
   
       console.print(table)
       console.print("  [dim](+N) = delta vs first scenario  |  ±N = stdev across runs[/]")
       console.print()
   
   
   if __name__ == "__main__":
       main()
   ```
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feat] [Slack Provider] Lazy import of Slack modules in slack/hooks/* [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #66025:
URL: https://github.com/apache/airflow/issues/66025#issuecomment-4548974624

   Closed for same reasons as https://github.com/apache/airflow/pull/67479#issuecomment-4548348694


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feat] [Slack Provider] Lazy import of Slack modules in slack/hooks/* [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed issue #66025: [Feat] [Slack Provider] Lazy import of Slack modules in slack/hooks/*
URL: https://github.com/apache/airflow/issues/66025


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org