You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dwreeves (via GitHub)" <gi...@apache.org> on 2026/04/28 14:59:52 UTC
[I] [Feat] [Slack Provider] Lazy import of Slack modules in snowflake/hooks/* [airflow]
dwreeves opened a new issue, #66025:
URL: https://github.com/apache/airflow/issues/66025
### Under which category would you file this issue?
Providers
### Apache Airflow version
2.10.5
### What happened and how to reproduce it?
The `SlackWebhookHook` import uses an incredibly large amount of memory and occasionally causes my workers to OOM. I don't have this issue elsewhere on my workers.
I've discovered that the import of `from slack_sdk.webhook.async_client import AsyncWebhookClient` causes most of the added overhead. I believe this may have to do with the import of `aiohttp`, which adds significant overhead.
In most cases, users just need the sync client, not the async client. So lazy loading should hopefully cause this to OOM less frequently.
### What you think should happen instead?
Lazy-loading of the Slack client reduces the amount of memory used by about 10% (relevant for global imports of the SlackWebhookHook at the scheduler level), and lazy loading the asyncio SDK is a 7% improvement (relevant for task execution that doesn't touch the asyncio SDK):
```
Python: 3.12.11 (main, Jun 4 2025, 17:42:58) [Clang 20.1.4 ]
Runs per scenario: 20 | also benchmark w/o .pyc: no
avg ms (±stdev) | avg RSS MB — 20 runs each
╭──────────────────────────┬───────────────────┬──────────────────╮
│ Scenario │ Time w/ .pyc (ms) │ RSS w/ .pyc (MB) │
├──────────────────────────┼───────────────────┼──────────────────┤
│ Base imports │ 966 ±60 │ 86.5 │
│ Base imports + SDK │ 1026 ±98 (+60) │ 88.3 (+1.8) │
│ Base imports + Async SDK │ 1124 ±77 (+158) │ 94.9 (+8.4) │
╰──────────────────────────┴───────────────────┴──────────────────╯
(+N) = delta vs first scenario | ±N = stdev across runs
```
The delta is a little smaller than I would have expected given how often jobs associated with the SlackWebhookHook OOMs. Still, this is a pretty marked improvement from just lazy loading some modules.
### Operating System
_No response_
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
apache-airflow==2.10.5
apache-airflow-providers-common-compat==1.7.0
apache-airflow-providers-common-io==1.6.0
apache-airflow-providers-common-sql==1.27.1
apache-airflow-providers-slack==9.1.0
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
You can run the benchmark with my code below (note: AI generated).
```python
#!/usr/bin/env python3
# NOTE: AI generated script.
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "apache-airflow",
# "apache-airflow-providers-slack",
# "rich-click",
# "rich",
# "requests",
# "tenacity",
# "cryptography",
# "slack-sdk",
# "aiohttp",
# "sqlalchemy",
# ]
# ///
from __future__ import annotations
import os
import shutil
import statistics
import subprocess
import sys
import tempfile
from dataclasses import dataclass, field
import rich_click as click
from rich import box
from rich.console import Console
from rich.table import Table
from rich.text import Text
console = Console()
BASE_IMPORTS = """\
import json
import warnings
from collections.abc import Callable
from functools import cached_property, wraps
from typing import TYPE_CHECKING, Any
from airflow.providers.common.compat.sdk import AirflowException, AirflowNotFoundException, BaseHook
"""
BASE_IMPORTS_PLUS_SDK = BASE_IMPORTS + """
from slack_sdk import WebhookClient
"""
# effectively equivalent to:
# from airflow.providers.slack.hooks.slack_webhook import *
BASE_IMPORTS_PLUS_ASYNC_SDK = BASE_IMPORTS_PLUS_SDK + BASE_IMPORTS + """
from airflow.providers.common.compat.connection import get_async_connection
from slack_sdk.webhook.async_client import AsyncWebhookClient
# ^ effectively equivalent to:
from airflow.providers.slack.hooks.slack_webhook import *
"""
SCENARIOS = [
("Base imports", BASE_IMPORTS),
("Base imports + SDK", BASE_IMPORTS_PLUS_SDK),
("Base import + Async SDK", BASE_IMPORTS_PLUS_ASYNC_SDK),
]
@dataclass
class RunStats:
times_ms: list[float] = field(default_factory=list)
rss_mb: list[float] = field(default_factory=list)
@property
def avg_ms(self) -> float:
return statistics.mean(self.times_ms)
@property
def avg_rss(self) -> float:
return statistics.mean(self.rss_mb)
@property
def stdev_ms(self) -> float:
return statistics.stdev(self.times_ms) if len(self.times_ms) > 1 else 0.0
@dataclass
class ScenarioResult:
label: str
with_pyc: RunStats
without_pyc: RunStats | None = None
SUBPROCESS_CODE_TEMPLATE = """\
import time, os, sys
import psutil
proc = psutil.Process(os.getpid())
rss_before = proc.memory_info().rss
t0 = time.perf_counter()
try:
{indented}
elapsed = time.perf_counter() - t0
rss_after = proc.memory_info().rss
print(f"{{elapsed:.6f}} {{(rss_after - rss_before) / 1024 / 1024:.4f}}")
except Exception as e:
print(f"ERROR: {{e}}", file=sys.stderr)
sys.exit(1)
"""
def _single_run(import_block: str, use_pyc: bool) -> tuple[float, float]:
"""Spawn a fresh interpreter and return (elapsed_ms, rss_mb)."""
indented = "\n".join(" " + line for line in import_block.splitlines())
code = SUBPROCESS_CODE_TEMPLATE.format(indented=indented)
env = None
tmp_dir = None
if not use_pyc:
# Redirect bytecode cache to a fresh empty dir so Python finds no .pyc
# files and must compile every module from source.
tmp_dir = tempfile.mkdtemp(prefix="no_pyc_bench_")
env = {**os.environ, "PYTHONPYCACHEPREFIX": tmp_dir}
try:
result = subprocess.run(
[sys.executable, "-c", code],
capture_output=True,
text=True,
env=env,
)
finally:
if tmp_dir:
shutil.rmtree(tmp_dir, ignore_errors=True)
if result.returncode != 0:
console.print(f"[red]ERROR:[/] {result.stderr.strip()}")
sys.exit(1)
elapsed_s, rss_mb = result.stdout.strip().split()
return float(elapsed_s) * 1000, float(rss_mb)
def measure(label: str, import_block: str, n: int, include_no_pyc: bool) -> ScenarioResult:
with_pyc = RunStats()
without_pyc = RunStats() if include_no_pyc else None
# Warm-up run to ensure .pyc files are written before timed runs.
with console.status(f" [italic]{label}[/] warming up .pyc cache…", spinner="dots"):
_single_run(import_block, use_pyc=True)
for i in range(n):
with console.status(f" [italic]{label}[/] run {i + 1}/{n} (w/ .pyc)…", spinner="dots"):
ms, mb = _single_run(import_block, use_pyc=True)
with_pyc.times_ms.append(ms)
with_pyc.rss_mb.append(mb)
if include_no_pyc:
for i in range(n):
with console.status(f" [italic]{label}[/] run {i + 1}/{n} (w/o .pyc)…", spinner="dots"):
ms, mb = _single_run(import_block, use_pyc=False)
without_pyc.times_ms.append(ms) # type: ignore[union-attr]
without_pyc.rss_mb.append(mb) # type: ignore[union-attr]
return ScenarioResult(label=label, with_pyc=with_pyc, without_pyc=without_pyc)
def _time_cell(stats: RunStats, baseline_ms: float) -> Text:
t = Text(f"{stats.avg_ms:.0f}", style="white")
t.append(f" ±{stats.stdev_ms:.0f}", style="dim")
if stats.avg_ms != baseline_ms:
t.append(f" (+{stats.avg_ms - baseline_ms:.0f})", style="dim cyan")
return t
def _rss_cell(stats: RunStats, baseline_mb: float) -> Text:
t = Text(f"{stats.avg_rss:.1f}", style="white")
if stats.avg_rss != baseline_mb:
t.append(f" (+{stats.avg_rss - baseline_mb:.1f})", style="dim cyan")
return t
@click.command()
@click.option(
"-n", "--runs",
default=10,
show_default=True,
help="Number of cold-cache subprocess runs to average per scenario.",
)
@click.option(
"--no-pyc",
is_flag=True,
default=False,
help="Also run benchmarks without .pyc bytecode caches.",
)
@click.rich_config({"theme": "cargo-slim"})
def main(runs: int, no_pyc: bool) -> None:
console.print()
console.rule("[bold cyan]Snowflake Hook — Import Cost Benchmark[/]")
console.print(f" [dim]Python: {sys.version}[/]")
console.print(f" [dim]Runs per scenario: {runs} | also benchmark w/o .pyc: {'yes' if no_pyc else 'no'}[/]")
console.print()
results = [measure(label, block, runs, no_pyc) for label, block in SCENARIOS]
table = Table(
box=box.ROUNDED,
show_footer=False,
title_style="bold cyan",
border_style="grey50",
title=f"avg ms (±stdev) | avg RSS MB — {runs} runs each",
)
table.add_column("Scenario", style="white", no_wrap=True)
table.add_column("Time w/ .pyc (ms)", justify="right")
table.add_column("RSS w/ .pyc (MB)", justify="right")
if no_pyc:
table.add_column("Time w/o .pyc (ms)", justify="right")
table.add_column("RSS w/o .pyc (MB)", justify="right")
baseline_pyc_ms = results[0].with_pyc.avg_ms
baseline_pyc_mb = results[0].with_pyc.avg_rss
baseline_nopyc_ms = results[0].without_pyc.avg_ms if no_pyc else 0.0
baseline_nopyc_mb = results[0].without_pyc.avg_rss if no_pyc else 0.0
for r in results:
row = [
r.label,
_time_cell(r.with_pyc, baseline_pyc_ms),
_rss_cell(r.with_pyc, baseline_pyc_mb),
]
if no_pyc and r.without_pyc is not None:
row += [
_time_cell(r.without_pyc, baseline_nopyc_ms),
_rss_cell(r.without_pyc, baseline_nopyc_mb),
]
table.add_row(*row)
console.print(table)
console.print(" [dim](+N) = delta vs first scenario | ±N = stdev across runs[/]")
console.print()
if __name__ == "__main__":
main()
```
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feat] [Slack Provider] Lazy import of Slack modules in slack/hooks/* [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #66025:
URL: https://github.com/apache/airflow/issues/66025#issuecomment-4548974624
Closed for same reasons as https://github.com/apache/airflow/pull/67479#issuecomment-4548348694
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feat] [Slack Provider] Lazy import of Slack modules in slack/hooks/* [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed issue #66025: [Feat] [Slack Provider] Lazy import of Slack modules in slack/hooks/*
URL: https://github.com/apache/airflow/issues/66025
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org