You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "BentsiLeviav (via GitHub)" <gi...@apache.org> on 2026/05/18 02:45:21 UTC

[PR] Add ClickHouse Provider [airflow]

BentsiLeviav opened a new pull request, #67080:
URL: https://github.com/apache/airflow/pull/67080

   ### Description
   Adds a new apache-airflow-providers-clickhouse provider that integrates Airflow with ClickHouse via the HTTP interface using the `clickhouse-connect` library.
   
   ### Scope of this implementation
   
     - `ClickHouseHook` - the core integration, extending `DbApiHook` so all standard `SQLExecuteQueryOperator` features work out of the box (templating, handler, split_statements, etc.)
     - Connection form UI with dedicated fields for TLS, timeouts, compression, session settings, and client kwargs
     - `bulk_insert_rows()` for more performant inserts using clickhouse-connect's native insert path
     - `get_uri()` for SQLAlchemy-compatible connection strings (`clickhousedb://` / `clickhousedbs://`)
     - Connection type docs, operator how-to guide, and integration logo
     - 95 unit tests
   
   ### Implementation decisions
   
     - `DB-API 2.0` adapter (`ClickHouseConnection`): clickhouse-connect doesn't expose a DB-API connection natively - we wrap its Client in a thin adapter so `DbApiHook.run()` works unmodified. `commit()`
     and `rollback()` are intentional no-ops since ClickHouse has no transactions.
     - Two-level settings merge: both `session_settings` and `client_kwargs` can be set at the connection level (via the extra JSON field) and overridden at the task level (via hook constructor arguments), with the constructor taking precedence on conflicts.
     - Hook-managed kwargs protection: keys that the hook owns (host, port, username, password, database, secure, verify, client_name, settings) are stripped from any user-supplied client_kwargs so hook-managed values always win.
     - Client name: every query is tagged with `apache-airflow/<version> apache-airflow-providers-clickhouse/<version>` in the HTTP User-Agent (system.query_log), making queries traceable back to their Airflow source. Users can append a custom label via the client_name extra field.
     - No dedicated operators are added - `SQLExecuteQueryOperator` from `common.sql` covers all standard SQL use cases.
   
   
   
   ## File structure (generated with Claude)
   
     | File(s) | Purpose |
     |---|---|
     | `provider.yaml` | Provider metadata: name, version, integrations, connection types, UI field behaviour, and `conn-fields` schema used to generate the connection form |
     | `pyproject.toml` | Package build config and dependencies (`clickhouse-connect >=0.7.0`, `common-sql >=1.32.0`) — auto-generated from the Breeze template |
     | `src/.../hooks/clickhouse.py` | Core implementation: `ClickHouseHook` (extends `DbApiHook`) and `ClickHouseConnection` (thin DB-API 2.0 adapter wrapping the `clickhouse-connect` client) |
     | `src/.../get_provider_info.py` | Auto-generated from `provider.yaml` by the Breeze release tooling — do not edit manually |
     | `src/airflow/__init__.py`, `src/airflow/providers/__init__.py` | Namespace package declarations required for the `airflow.providers` implicit namespace |
     | `src/.../clickhouse/__init__.py` | Version file (`__version__ = "1.0.0"`) with minimum Airflow version guard — auto-generated |
     | `docs/connections/clickhouse.rst` | Connection configuration reference: all fields, their types, defaults, and JSON/URI examples |
     | `docs/operators/clickhouse.rst` | How-to guide: using `SQLExecuteQueryOperator` and `ClickHouseHook` directly, including `session_settings` and `bulk_insert_rows` examples |
     | `docs/index.rst`, `docs/conf.py`, `docs/changelog.rst`, `docs/security.rst` | Standard provider docs scaffold — mostly auto-generated |
     | `docs/integration-logos/ClickHouse.png` | Official ClickHouse logo used by the Apache Airflow website |
     | `tests/unit/clickhouse/hooks/test_clickhouse.py` | 95 unit tests covering connection building, settings/kwargs merge logic, database override, URI generation, bulk insert, UI widgets, and
     autocommit semantics |
     | `tests/system/clickhouse/example_clickhouse.py` | System test / example DAG: create table → bulk insert → read rows → drop table |
     | `.github/boring-cyborg.yml` | Adds `provider:clickhouse` label rule for automatic PR labelling |
     | `scripts/ci/docker-compose/remove-sources.yml`, `tests-sources.yml` | Auto-updated by prek to mount the clickhouse provider sources/tests into the CI Docker environment |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "BentsiLeviav (via GitHub)" <gi...@apache.org>.
BentsiLeviav commented on code in PR #67080:
URL: https://github.com/apache/airflow/pull/67080#discussion_r3312455932


##########
providers/clickhouse/src/airflow/providers/clickhouse/hooks/clickhouse.py:
##########
@@ -0,0 +1,463 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any
+from urllib.parse import quote_plus
+
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+    import clickhouse_connect.driver.client
+    from clickhouse_connect.dbapi.cursor import Cursor as ClickHouseDbApiCursor
+
+# Optional scalar parameters forwarded verbatim to clickhouse_connect.get_client()
+# Note: client_name is handled separately by _build_client_name() so Airflow version
+# info is always embedded in the HTTP User-Agent / system.query_log entry.
+_OPTIONAL_CLIENT_KWARGS = (
+    "connect_timeout",
+    "send_receive_timeout",
+    "compress",
+)
+
+# kwargs that the hook always manages itself.  User-supplied client_kwargs that
+# overlap with these keys are dropped and logged at DEBUG so the hook's own values always win.
+_HOOK_MANAGED_KWARGS: frozenset[str] = frozenset(
+    {
+        "host",
+        "port",
+        "username",
+        "password",
+        "database",
+        "secure",
+        "verify",
+        "client_name",
+        "settings",
+    }
+)
+
+
+def _build_client_name(custom: str | None = None) -> str:
+    """
+    Return the ``client_name`` string passed to ``clickhouse_connect.get_client()``.
+
+    The value is prepended to the HTTP ``User-Agent`` header that ClickHouse records
+    in ``system.query_log``, so every query is traceable back to its Airflow source.
+
+    Format (no custom label)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version>
+
+    Format (with the ``client_name`` extra field set to ``"my-pipeline"``)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version> (my-pipeline)
+
+    The ``clickhouse_connect`` library appends its own token and OS information, so
+    the full ``User-Agent`` looks like::
+
+        apache-airflow/X.Y.Z apache-airflow-providers-clickhouse/X.Y.Z
+        clickhouse-connect/X.Y.Z (lv:py/X.Y.Z; mode:sync; os:linux)
+    """
+    import airflow
+    from airflow.providers.clickhouse import __version__ as provider_version
+
+    name = f"apache-airflow/{airflow.__version__} apache-airflow-providers-clickhouse/{provider_version}"
+    if custom:
+        name = f"{name} ({custom.strip()})"
+    return name
+
+
+class ClickHouseConnection:
+    """
+    Minimal DB-API 2.0 connection adapter wrapping a ``clickhouse_connect`` Client.
+
+    SQL execution is delegated to ``clickhouse_connect.dbapi.cursor.Cursor``, which
+    routes each statement to ``client.query()`` or ``client.command()`` automatically
+    by inspecting the SQL keyword after stripping comments — the same logic used by
+    the ``clickhouse-connect`` SQLAlchemy dialect.
+
+    ClickHouse has no multi-statement transactions.  Every statement is effectively
+    auto-committed, so ``commit()`` and ``rollback()`` are intentional no-ops and
+    the ``autocommit`` attribute is always ``True``.  ``DbApiHook.run()`` checks
+    ``conn.autocommit`` via :meth:`ClickHouseHook.get_autocommit` and skips the
+    ``conn.commit()`` call when it is truthy.
+    """
+
+    # Signals to DbApiHook.get_autocommit() that no explicit commit is needed.
+    autocommit: bool = True
+
+    def __init__(self, client: clickhouse_connect.driver.client.Client) -> None:
+        self._client = client
+
+    def cursor(self) -> ClickHouseDbApiCursor:
+        from clickhouse_connect.dbapi.cursor import Cursor
+
+        return Cursor(self._client)
+
+    def close(self) -> None:
+        self._client.close()
+
+    def commit(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+    def rollback(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+
+class ClickHouseHook(DbApiHook):
+    """
+    Interact with ClickHouse via the HTTP interface (``clickhouse-connect``).
+
+    This hook wraps ``clickhouse_connect.get_client()`` behind a thin DB-API 2.0
+    adapter (:class:`ClickHouseConnection`), so all standard
+    :class:`~airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`
+    features work out of the box (templating, ``handler``, ``split_statements``,
+    etc.).
+
+    :param database: Optional database name.  Overrides the ``schema`` field of the
+        Airflow connection.  Useful when one connection points to a ClickHouse cluster
+        and individual tasks need to target different databases.
+    :param session_settings: Optional dict of ClickHouse session-level settings
+        (e.g. ``{"max_execution_time": 60, "max_threads": 4}``).  Values supplied
+        here are **merged on top of** any ``session_settings`` dict already present
+        in the connection's ``extra`` JSON field, with the constructor argument
+        taking precedence. For a full list of available session settings visit https://clickhouse.com/docs/operations/settings/settings
+    :param client_kwargs: Optional dict of additional keyword arguments forwarded
+        verbatim to ``clickhouse_connect.get_client()``.  Use this to pass any
+        parameter not otherwise exposed by the hook (e.g. ``http_proxy``,
+        ``pool_mgr_params``).  Values supplied here are **merged on top of** any
+        ``client_kwargs`` dict already present in the connection's ``extra`` JSON
+        field, with the constructor argument taking precedence on conflicting keys.
+        Keys that the hook manages itself (``host``, ``port``, ``username``,
+        ``password``, ``database``, ``secure``, ``verify``, ``client_name``,
+        ``settings``) are silently ignored so that hook-managed values always
+        take precedence.
+
+    .. seealso::
+        - `clickhouse-connect documentation <https://clickhouse.com/docs/en/integrations/python>`_
+        - `clickhouse session settings documentation <https://clickhouse.com/docs/operations/settings/settings>`_
+        - :ref:`howto/connection:clickhouse`
+
+    """
+
+    conn_name_attr = "clickhouse_conn_id"
+    default_conn_name = "clickhouse_default"
+    conn_type = "clickhouse"
+    hook_name = "ClickHouse"
+
+    # ClickHouse has no multi-statement transactions; every statement is auto-committed.
+    supports_autocommit = True
+    _test_connection_sql = "SELECT 1"
+
+    def set_autocommit(self, conn: ClickHouseConnection, autocommit: bool) -> None:
+        """No-op: ClickHouse has no transaction support."""
+
+    def get_autocommit(self, conn: ClickHouseConnection) -> bool:
+        """Return ``True``: ClickHouse auto-commits every statement."""
+        return True
+
+    def __init__(
+        self,
+        *args,
+        database: str | None = None,
+        session_settings: dict[str, Any] | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.database = database
+        self.session_settings: dict[str, Any] = session_settings or {}
+        self.client_kwargs: dict[str, Any] = client_kwargs or {}
+
+    @classmethod
+    def get_connection_form_widgets(cls) -> dict[str, Any]:
+        """Return extra connection form widgets shown in the Airflow UI."""
+        from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+        from flask_babel import lazy_gettext
+        from wtforms import BooleanField, IntegerField, StringField
+
+        return {
+            "secure": BooleanField(
+                label=lazy_gettext("Use TLS (HTTPS)"),
+                description="Enable TLS/HTTPS for the connection (default: False).",
+            ),
+            "verify": BooleanField(
+                label=lazy_gettext("Verify SSL Certificate"),
+                description="Verify the server SSL certificate when TLS is enabled (default: True).",
+            ),
+            "connect_timeout": IntegerField(
+                lazy_gettext("Connection Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "send_receive_timeout": IntegerField(
+                lazy_gettext("Query Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "compress": BooleanField(
+                label=lazy_gettext("Enable LZ4 Compression"),
+                description="Compress query results with LZ4 (default: True).",
+            ),
+            "client_name": StringField(
+                lazy_gettext("Client Name (label)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Optional label appended to the Airflow version identifier in the "
+                    "ClickHouse HTTP User-Agent and system.query_log client_name field. "
+                    "The hook always prepends apache-airflow/<version> "
+                    "apache-airflow-providers-clickhouse/<version>."
+                ),
+            ),
+            "session_settings": StringField(
+                lazy_gettext("Session Settings (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "ClickHouse session-level settings as a JSON object, "
+                    'e.g. {"max_execution_time": 60, "max_threads": 4}.'
+                ),
+            ),
+            "client_kwargs": StringField(
+                lazy_gettext("Client kwargs (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Additional keyword arguments forwarded to clickhouse_connect.get_client() "
+                    'as a JSON object, e.g. {"http_proxy": "http://proxy:8080"}. '
+                    "Hook-managed keys (host, port, username, password, database, secure, "
+                    "verify, client_name, settings) are ignored."
+                ),
+            ),
+        }
+
+    @classmethod
+    def get_ui_field_behaviour(cls) -> dict[str, Any]:
+        """Return custom field behaviour for the connection form."""
+        return {
+            "hidden_fields": [],
+            "relabeling": {
+                "schema": "Database",
+            },
+            "placeholders": {
+                "host": "localhost",
+                "port": "8123",
+                "login": "default",
+                "schema": "default",
+                "extra": json.dumps(
+                    {
+                        "secure": False,
+                        "verify": True,
+                        "connect_timeout": 10,
+                        "send_receive_timeout": 300,
+                        "compress": True,
+                        "client_name": "airflow",
+                        "session_settings": {
+                            "max_execution_time": 300,
+                            "max_memory_usage": 10000000000,
+                        },
+                    },
+                    indent=1,
+                ),
+                "secure": "false",
+                "verify": "true",
+                "connect_timeout": "10",
+                "send_receive_timeout": "300",
+                "compress": "true",
+                "client_name": "airflow",
+                "session_settings": '{"max_execution_time": 300}',
+                "client_kwargs": '{"http_proxy": "http://proxy:8080"}',
+            },
+        }

Review Comment:
   done. Added the missing options to `provider.yaml`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "BentsiLeviav (via GitHub)" <gi...@apache.org>.
BentsiLeviav commented on code in PR #67080:
URL: https://github.com/apache/airflow/pull/67080#discussion_r3293936930


##########
providers/clickhouse/src/airflow/providers/clickhouse/hooks/clickhouse.py:
##########
@@ -0,0 +1,463 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any
+from urllib.parse import quote_plus
+
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+    import clickhouse_connect.driver.client
+    from clickhouse_connect.dbapi.cursor import Cursor as ClickHouseDbApiCursor
+
+# Optional scalar parameters forwarded verbatim to clickhouse_connect.get_client()
+# Note: client_name is handled separately by _build_client_name() so Airflow version
+# info is always embedded in the HTTP User-Agent / system.query_log entry.
+_OPTIONAL_CLIENT_KWARGS = (
+    "connect_timeout",
+    "send_receive_timeout",
+    "compress",
+)
+
+# kwargs that the hook always manages itself.  User-supplied client_kwargs that
+# overlap with these keys are dropped and logged at DEBUG so the hook's own values always win.
+_HOOK_MANAGED_KWARGS: frozenset[str] = frozenset(
+    {
+        "host",
+        "port",
+        "username",
+        "password",
+        "database",
+        "secure",
+        "verify",
+        "client_name",
+        "settings",
+    }
+)
+
+
+def _build_client_name(custom: str | None = None) -> str:
+    """
+    Return the ``client_name`` string passed to ``clickhouse_connect.get_client()``.
+
+    The value is prepended to the HTTP ``User-Agent`` header that ClickHouse records
+    in ``system.query_log``, so every query is traceable back to its Airflow source.
+
+    Format (no custom label)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version>
+
+    Format (with the ``client_name`` extra field set to ``"my-pipeline"``)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version> (my-pipeline)
+
+    The ``clickhouse_connect`` library appends its own token and OS information, so
+    the full ``User-Agent`` looks like::
+
+        apache-airflow/X.Y.Z apache-airflow-providers-clickhouse/X.Y.Z
+        clickhouse-connect/X.Y.Z (lv:py/X.Y.Z; mode:sync; os:linux)
+    """
+    import airflow
+    from airflow.providers.clickhouse import __version__ as provider_version
+
+    name = f"apache-airflow/{airflow.__version__} apache-airflow-providers-clickhouse/{provider_version}"
+    if custom:
+        name = f"{name} ({custom.strip()})"
+    return name
+
+
+class ClickHouseConnection:
+    """
+    Minimal DB-API 2.0 connection adapter wrapping a ``clickhouse_connect`` Client.
+
+    SQL execution is delegated to ``clickhouse_connect.dbapi.cursor.Cursor``, which
+    routes each statement to ``client.query()`` or ``client.command()`` automatically
+    by inspecting the SQL keyword after stripping comments — the same logic used by
+    the ``clickhouse-connect`` SQLAlchemy dialect.
+
+    ClickHouse has no multi-statement transactions.  Every statement is effectively
+    auto-committed, so ``commit()`` and ``rollback()`` are intentional no-ops and
+    the ``autocommit`` attribute is always ``True``.  ``DbApiHook.run()`` checks
+    ``conn.autocommit`` via :meth:`ClickHouseHook.get_autocommit` and skips the
+    ``conn.commit()`` call when it is truthy.
+    """
+
+    # Signals to DbApiHook.get_autocommit() that no explicit commit is needed.
+    autocommit: bool = True
+
+    def __init__(self, client: clickhouse_connect.driver.client.Client) -> None:
+        self._client = client
+
+    def cursor(self) -> ClickHouseDbApiCursor:
+        from clickhouse_connect.dbapi.cursor import Cursor
+
+        return Cursor(self._client)
+
+    def close(self) -> None:
+        self._client.close()
+
+    def commit(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+    def rollback(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+
+class ClickHouseHook(DbApiHook):
+    """
+    Interact with ClickHouse via the HTTP interface (``clickhouse-connect``).
+
+    This hook wraps ``clickhouse_connect.get_client()`` behind a thin DB-API 2.0
+    adapter (:class:`ClickHouseConnection`), so all standard
+    :class:`~airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`
+    features work out of the box (templating, ``handler``, ``split_statements``,
+    etc.).
+
+    :param database: Optional database name.  Overrides the ``schema`` field of the
+        Airflow connection.  Useful when one connection points to a ClickHouse cluster
+        and individual tasks need to target different databases.
+    :param session_settings: Optional dict of ClickHouse session-level settings
+        (e.g. ``{"max_execution_time": 60, "max_threads": 4}``).  Values supplied
+        here are **merged on top of** any ``session_settings`` dict already present
+        in the connection's ``extra`` JSON field, with the constructor argument
+        taking precedence. For a full list of available session settings visit https://clickhouse.com/docs/operations/settings/settings
+    :param client_kwargs: Optional dict of additional keyword arguments forwarded
+        verbatim to ``clickhouse_connect.get_client()``.  Use this to pass any
+        parameter not otherwise exposed by the hook (e.g. ``http_proxy``,
+        ``pool_mgr_params``).  Values supplied here are **merged on top of** any
+        ``client_kwargs`` dict already present in the connection's ``extra`` JSON
+        field, with the constructor argument taking precedence on conflicting keys.
+        Keys that the hook manages itself (``host``, ``port``, ``username``,
+        ``password``, ``database``, ``secure``, ``verify``, ``client_name``,
+        ``settings``) are silently ignored so that hook-managed values always
+        take precedence.
+
+    .. seealso::
+        - `clickhouse-connect documentation <https://clickhouse.com/docs/en/integrations/python>`_
+        - `clickhouse session settings documentation <https://clickhouse.com/docs/operations/settings/settings>`_
+        - :ref:`howto/connection:clickhouse`
+
+    """
+
+    conn_name_attr = "clickhouse_conn_id"
+    default_conn_name = "clickhouse_default"
+    conn_type = "clickhouse"
+    hook_name = "ClickHouse"
+
+    # ClickHouse has no multi-statement transactions; every statement is auto-committed.
+    supports_autocommit = True
+    _test_connection_sql = "SELECT 1"
+
+    def set_autocommit(self, conn: ClickHouseConnection, autocommit: bool) -> None:
+        """No-op: ClickHouse has no transaction support."""
+
+    def get_autocommit(self, conn: ClickHouseConnection) -> bool:
+        """Return ``True``: ClickHouse auto-commits every statement."""
+        return True
+
+    def __init__(
+        self,
+        *args,
+        database: str | None = None,
+        session_settings: dict[str, Any] | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.database = database
+        self.session_settings: dict[str, Any] = session_settings or {}
+        self.client_kwargs: dict[str, Any] = client_kwargs or {}
+
+    @classmethod
+    def get_connection_form_widgets(cls) -> dict[str, Any]:
+        """Return extra connection form widgets shown in the Airflow UI."""
+        from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+        from flask_babel import lazy_gettext
+        from wtforms import BooleanField, IntegerField, StringField
+
+        return {
+            "secure": BooleanField(
+                label=lazy_gettext("Use TLS (HTTPS)"),
+                description="Enable TLS/HTTPS for the connection (default: False).",
+            ),
+            "verify": BooleanField(
+                label=lazy_gettext("Verify SSL Certificate"),
+                description="Verify the server SSL certificate when TLS is enabled (default: True).",
+            ),
+            "connect_timeout": IntegerField(
+                lazy_gettext("Connection Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "send_receive_timeout": IntegerField(
+                lazy_gettext("Query Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "compress": BooleanField(
+                label=lazy_gettext("Enable LZ4 Compression"),
+                description="Compress query results with LZ4 (default: True).",
+            ),
+            "client_name": StringField(
+                lazy_gettext("Client Name (label)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Optional label appended to the Airflow version identifier in the "
+                    "ClickHouse HTTP User-Agent and system.query_log client_name field. "
+                    "The hook always prepends apache-airflow/<version> "
+                    "apache-airflow-providers-clickhouse/<version>."
+                ),
+            ),
+            "session_settings": StringField(
+                lazy_gettext("Session Settings (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "ClickHouse session-level settings as a JSON object, "
+                    'e.g. {"max_execution_time": 60, "max_threads": 4}.'
+                ),
+            ),
+            "client_kwargs": StringField(
+                lazy_gettext("Client kwargs (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Additional keyword arguments forwarded to clickhouse_connect.get_client() "
+                    'as a JSON object, e.g. {"http_proxy": "http://proxy:8080"}. '
+                    "Hook-managed keys (host, port, username, password, database, secure, "
+                    "verify, client_name, settings) are ignored."
+                ),
+            ),
+        }
+
+    @classmethod
+    def get_ui_field_behaviour(cls) -> dict[str, Any]:
+        """Return custom field behaviour for the connection form."""
+        return {
+            "hidden_fields": [],
+            "relabeling": {
+                "schema": "Database",
+            },
+            "placeholders": {
+                "host": "localhost",
+                "port": "8123",
+                "login": "default",
+                "schema": "default",
+                "extra": json.dumps(
+                    {
+                        "secure": False,
+                        "verify": True,
+                        "connect_timeout": 10,
+                        "send_receive_timeout": 300,
+                        "compress": True,
+                        "client_name": "airflow",
+                        "session_settings": {
+                            "max_execution_time": 300,
+                            "max_memory_usage": 10000000000,
+                        },
+                    },
+                    indent=1,
+                ),
+                "secure": "false",
+                "verify": "true",
+                "connect_timeout": "10",
+                "send_receive_timeout": "300",
+                "compress": "true",
+                "client_name": "airflow",
+                "session_settings": '{"max_execution_time": 300}',
+                "client_kwargs": '{"http_proxy": "http://proxy:8080"}',
+            },
+        }
+
+    def _get_client_kwargs(self) -> dict[str, Any]:
+        """
+        Build the keyword-argument dict passed to ``clickhouse_connect.get_client()``.
+
+        Construction order (last write wins):
+
+        1. ``client_kwargs`` from the connection ``extra`` JSON — connection-level
+           passthrough kwargs.
+        2. ``client_kwargs`` constructor argument — task-level overrides merged on top.
+        3. Hook-managed keys (``host``, ``port``, ``username``, ``password``,
+           ``database``, ``secure``, ``verify``, ``client_name``, ``settings``) —
+           always override anything in ``client_kwargs``.
+
+        Optional tuning parameters (``connect_timeout``, ``send_receive_timeout``,
+        ``compress``) are forwarded only when explicitly set in the ``extra`` JSON
+        field so that the driver's own defaults apply otherwise.
+
+        ``client_name`` is always set to a string that identifies the Airflow
+        installation (see :func:`_build_client_name`).  Any ``client_name`` value in
+        ``extra`` is treated as a human-readable label and appended as a comment so
+        it appears alongside the Airflow version info in ``system.query_log``.
+
+        ``session_settings`` from ``extra`` and from the constructor ``session_settings``
+        argument are **merged**, with the constructor argument taking precedence on
+        conflicting keys.
+        """
+        conn = self.get_connection(self.get_conn_id())
+        extra: dict[str, Any] = conn.extra_dejson
+
+        # Merge client_kwargs: extra values are the base, constructor values override.
+        raw_client_kwargs = extra.get("client_kwargs")

Review Comment:
   I saw [postgres](https://github.com/apache/airflow/blob/main/providers/postgres/src/airflow/providers/postgres/hooks/postgres.py#L261-L263) passes arbitrary extras through to the underlying library as well, which Airflow's security model explicitly accepts for trusted Connection configuration users.
   
   As a previous Airflow user, such capability would have helped a lot (where you could update critical settings via the connections and they would be applied to all relevant Dags, rather than going one by one and updating the constructor. The same goes for common settings that could be shared in a single place, instead of each Dag separately.)
     
     @eladkal could you or any other Airflow maintainer provide guidance on that? I'm ok with reverting this behaviour if needed, though I think it might be very useful. In addition, we can start with having this and revert the logic in the future if we get any user feedback about it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "BentsiLeviav (via GitHub)" <gi...@apache.org>.
BentsiLeviav commented on PR #67080:
URL: https://github.com/apache/airflow/pull/67080#issuecomment-4473848845

   @koletzilla @joe-clickhouse would you mind reviewing that as well?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #67080:
URL: https://github.com/apache/airflow/pull/67080#discussion_r3350396973


##########
providers/clickhousedb/src/airflow/providers/clickhousedb/hooks/clickhouse.py:
##########
@@ -0,0 +1,463 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any
+from urllib.parse import quote_plus
+
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+    import clickhouse_connect.driver.client
+    from clickhouse_connect.dbapi.cursor import Cursor as ClickHouseDbApiCursor
+
+# Optional scalar parameters forwarded verbatim to clickhouse_connect.get_client()
+# Note: client_name is handled separately by _build_client_name() so Airflow version
+# info is always embedded in the HTTP User-Agent / system.query_log entry.
+_OPTIONAL_CLIENT_KWARGS = (
+    "connect_timeout",
+    "send_receive_timeout",
+    "compress",
+)
+
+# kwargs that the hook always manages itself.  User-supplied client_kwargs that
+# overlap with these keys are dropped and logged at DEBUG so the hook's own values always win.
+_HOOK_MANAGED_KWARGS: frozenset[str] = frozenset(
+    {
+        "host",
+        "port",
+        "username",
+        "password",
+        "database",
+        "secure",
+        "verify",
+        "client_name",
+        "settings",
+    }
+)
+
+
+def _build_client_name(custom: str | None = None) -> str:
+    """
+    Return the ``client_name`` string passed to ``clickhouse_connect.get_client()``.
+
+    The value is prepended to the HTTP ``User-Agent`` header that ClickHouse records
+    in ``system.query_log``, so every query is traceable back to its Airflow source.
+
+    Format (no custom label)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version>
+
+    Format (with the ``client_name`` extra field set to ``"my-pipeline"``)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version> (my-pipeline)
+
+    The ``clickhouse_connect`` library appends its own token and OS information, so
+    the full ``User-Agent`` looks like::
+
+        apache-airflow/X.Y.Z apache-airflow-providers-clickhouse/X.Y.Z
+        clickhouse-connect/X.Y.Z (lv:py/X.Y.Z; mode:sync; os:linux)
+    """
+    import airflow
+    from airflow.providers.clickhouse import __version__ as provider_version
+
+    name = f"apache-airflow/{airflow.__version__} apache-airflow-providers-clickhouse/{provider_version}"
+    if custom:
+        name = f"{name} ({custom.strip()})"
+    return name
+
+
+class ClickHouseConnection:
+    """
+    Minimal DB-API 2.0 connection adapter wrapping a ``clickhouse_connect`` Client.
+
+    SQL execution is delegated to ``clickhouse_connect.dbapi.cursor.Cursor``, which
+    routes each statement to ``client.query()`` or ``client.command()`` automatically
+    by inspecting the SQL keyword after stripping comments — the same logic used by
+    the ``clickhouse-connect`` SQLAlchemy dialect.
+
+    ClickHouse has no multi-statement transactions.  Every statement is effectively
+    auto-committed, so ``commit()`` and ``rollback()`` are intentional no-ops and
+    the ``autocommit`` attribute is always ``True``.  ``DbApiHook.run()`` checks
+    ``conn.autocommit`` via :meth:`ClickHouseHook.get_autocommit` and skips the
+    ``conn.commit()`` call when it is truthy.
+    """
+
+    # Signals to DbApiHook.get_autocommit() that no explicit commit is needed.
+    autocommit: bool = True
+
+    def __init__(self, client: clickhouse_connect.driver.client.Client) -> None:
+        self._client = client
+
+    def cursor(self) -> ClickHouseDbApiCursor:
+        from clickhouse_connect.dbapi.cursor import Cursor
+
+        return Cursor(self._client)
+
+    def close(self) -> None:
+        self._client.close()
+
+    def commit(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+    def rollback(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+
+class ClickHouseHook(DbApiHook):
+    """
+    Interact with ClickHouse via the HTTP interface (``clickhouse-connect``).
+
+    This hook wraps ``clickhouse_connect.get_client()`` behind a thin DB-API 2.0
+    adapter (:class:`ClickHouseConnection`), so all standard
+    :class:`~airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`
+    features work out of the box (templating, ``handler``, ``split_statements``,
+    etc.).
+
+    :param database: Optional database name.  Overrides the ``schema`` field of the
+        Airflow connection.  Useful when one connection points to a ClickHouse cluster
+        and individual tasks need to target different databases.
+    :param session_settings: Optional dict of ClickHouse session-level settings
+        (e.g. ``{"max_execution_time": 60, "max_threads": 4}``).  Values supplied
+        here are **merged on top of** any ``session_settings`` dict already present
+        in the connection's ``extra`` JSON field, with the constructor argument
+        taking precedence. For a full list of available session settings visit https://clickhouse.com/docs/operations/settings/settings
+    :param client_kwargs: Optional dict of additional keyword arguments forwarded
+        verbatim to ``clickhouse_connect.get_client()``.  Use this to pass any
+        parameter not otherwise exposed by the hook (e.g. ``http_proxy``,
+        ``pool_mgr_params``).  Values supplied here are **merged on top of** any
+        ``client_kwargs`` dict already present in the connection's ``extra`` JSON
+        field, with the constructor argument taking precedence on conflicting keys.
+        Keys that the hook manages itself (``host``, ``port``, ``username``,
+        ``password``, ``database``, ``secure``, ``verify``, ``client_name``,
+        ``settings``) are silently ignored so that hook-managed values always
+        take precedence.
+
+    .. seealso::
+        - `clickhouse-connect documentation <https://clickhouse.com/docs/en/integrations/python>`_
+        - `clickhouse session settings documentation <https://clickhouse.com/docs/operations/settings/settings>`_
+        - :ref:`howto/connection:clickhouse`
+
+    """
+
+    conn_name_attr = "clickhouse_conn_id"
+    default_conn_name = "clickhouse_default"
+    conn_type = "clickhouse"
+    hook_name = "ClickHouse"
+
+    # ClickHouse has no multi-statement transactions; every statement is auto-committed.
+    supports_autocommit = True
+    _test_connection_sql = "SELECT 1"
+
+    def set_autocommit(self, conn: ClickHouseConnection, autocommit: bool) -> None:
+        """No-op: ClickHouse has no transaction support."""
+
+    def get_autocommit(self, conn: ClickHouseConnection) -> bool:
+        """Return ``True``: ClickHouse auto-commits every statement."""
+        return True
+
+    def __init__(
+        self,
+        *args,
+        database: str | None = None,
+        session_settings: dict[str, Any] | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.database = database
+        self.session_settings: dict[str, Any] = session_settings or {}
+        self.client_kwargs: dict[str, Any] = client_kwargs or {}
+
+    @classmethod
+    def get_connection_form_widgets(cls) -> dict[str, Any]:
+        """Return extra connection form widgets shown in the Airflow UI."""
+        from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+        from flask_babel import lazy_gettext
+        from wtforms import BooleanField, IntegerField, StringField
+
+        return {
+            "secure": BooleanField(
+                label=lazy_gettext("Use TLS (HTTPS)"),
+                description="Enable TLS/HTTPS for the connection (default: False).",
+            ),
+            "verify": BooleanField(
+                label=lazy_gettext("Verify SSL Certificate"),
+                description="Verify the server SSL certificate when TLS is enabled (default: True).",
+            ),
+            "connect_timeout": IntegerField(
+                lazy_gettext("Connection Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "send_receive_timeout": IntegerField(
+                lazy_gettext("Query Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "compress": BooleanField(
+                label=lazy_gettext("Enable LZ4 Compression"),
+                description="Compress query results with LZ4 (default: True).",
+            ),
+            "client_name": StringField(
+                lazy_gettext("Client Name (label)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Optional label appended to the Airflow version identifier in the "
+                    "ClickHouse HTTP User-Agent and system.query_log client_name field. "
+                    "The hook always prepends apache-airflow/<version> "
+                    "apache-airflow-providers-clickhouse/<version>."
+                ),
+            ),
+            "session_settings": StringField(
+                lazy_gettext("Session Settings (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "ClickHouse session-level settings as a JSON object, "
+                    'e.g. {"max_execution_time": 60, "max_threads": 4}.'
+                ),
+            ),
+            "client_kwargs": StringField(
+                lazy_gettext("Client kwargs (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Additional keyword arguments forwarded to clickhouse_connect.get_client() "
+                    'as a JSON object, e.g. {"http_proxy": "http://proxy:8080"}. '
+                    "Hook-managed keys (host, port, username, password, database, secure, "
+                    "verify, client_name, settings) are ignored."
+                ),
+            ),
+        }
+
+    @classmethod
+    def get_ui_field_behaviour(cls) -> dict[str, Any]:
+        """Return custom field behaviour for the connection form."""
+        return {
+            "hidden_fields": [],
+            "relabeling": {
+                "schema": "Database",
+            },
+            "placeholders": {
+                "host": "localhost",
+                "port": "8123",
+                "login": "default",
+                "schema": "default",
+                "extra": json.dumps(
+                    {
+                        "secure": False,
+                        "verify": True,
+                        "connect_timeout": 10,
+                        "send_receive_timeout": 300,
+                        "compress": True,
+                        "client_name": "airflow",
+                        "session_settings": {
+                            "max_execution_time": 300,
+                            "max_memory_usage": 10000000000,
+                        },
+                    },
+                    indent=1,
+                ),
+                "secure": "false",
+                "verify": "true",
+                "connect_timeout": "10",
+                "send_receive_timeout": "300",
+                "compress": "true",
+                "client_name": "airflow",
+                "session_settings": '{"max_execution_time": 300}',
+                "client_kwargs": '{"http_proxy": "http://proxy:8080"}',
+            },
+        }
+
+    def _get_client_kwargs(self) -> dict[str, Any]:
+        """
+        Build the keyword-argument dict passed to ``clickhouse_connect.get_client()``.
+
+        Construction order (last write wins):
+
+        1. ``client_kwargs`` from the connection ``extra`` JSON — connection-level
+           passthrough kwargs.
+        2. ``client_kwargs`` constructor argument — task-level overrides merged on top.
+        3. Hook-managed keys (``host``, ``port``, ``username``, ``password``,
+           ``database``, ``secure``, ``verify``, ``client_name``, ``settings``) —
+           always override anything in ``client_kwargs``.
+
+        Optional tuning parameters (``connect_timeout``, ``send_receive_timeout``,
+        ``compress``) are forwarded only when explicitly set in the ``extra`` JSON
+        field so that the driver's own defaults apply otherwise.
+
+        ``client_name`` is always set to a string that identifies the Airflow
+        installation (see :func:`_build_client_name`).  Any ``client_name`` value in
+        ``extra`` is treated as a human-readable label and appended as a comment so
+        it appears alongside the Airflow version info in ``system.query_log``.
+
+        ``session_settings`` from ``extra`` and from the constructor ``session_settings``
+        argument are **merged**, with the constructor argument taking precedence on
+        conflicting keys.
+        """
+        conn = self.get_connection(self.get_conn_id())
+        extra: dict[str, Any] = conn.extra_dejson
+
+        # Merge client_kwargs: extra values are the base, constructor values override.
+        raw_client_kwargs = extra.get("client_kwargs")

Review Comment:
   Example https://redirect.github.com/apache/airflow/pull/32345



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #67080:
URL: https://github.com/apache/airflow/pull/67080#discussion_r3305281619


##########
providers/clickhouse/src/airflow/providers/clickhouse/hooks/clickhouse.py:
##########
@@ -0,0 +1,463 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any
+from urllib.parse import quote_plus
+
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+    import clickhouse_connect.driver.client
+    from clickhouse_connect.dbapi.cursor import Cursor as ClickHouseDbApiCursor
+
+# Optional scalar parameters forwarded verbatim to clickhouse_connect.get_client()
+# Note: client_name is handled separately by _build_client_name() so Airflow version
+# info is always embedded in the HTTP User-Agent / system.query_log entry.
+_OPTIONAL_CLIENT_KWARGS = (
+    "connect_timeout",
+    "send_receive_timeout",
+    "compress",
+)
+
+# kwargs that the hook always manages itself.  User-supplied client_kwargs that
+# overlap with these keys are dropped and logged at DEBUG so the hook's own values always win.
+_HOOK_MANAGED_KWARGS: frozenset[str] = frozenset(
+    {
+        "host",
+        "port",
+        "username",
+        "password",
+        "database",
+        "secure",
+        "verify",
+        "client_name",
+        "settings",
+    }
+)
+
+
+def _build_client_name(custom: str | None = None) -> str:
+    """
+    Return the ``client_name`` string passed to ``clickhouse_connect.get_client()``.
+
+    The value is prepended to the HTTP ``User-Agent`` header that ClickHouse records
+    in ``system.query_log``, so every query is traceable back to its Airflow source.
+
+    Format (no custom label)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version>
+
+    Format (with the ``client_name`` extra field set to ``"my-pipeline"``)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version> (my-pipeline)
+
+    The ``clickhouse_connect`` library appends its own token and OS information, so
+    the full ``User-Agent`` looks like::
+
+        apache-airflow/X.Y.Z apache-airflow-providers-clickhouse/X.Y.Z
+        clickhouse-connect/X.Y.Z (lv:py/X.Y.Z; mode:sync; os:linux)
+    """
+    import airflow
+    from airflow.providers.clickhouse import __version__ as provider_version
+
+    name = f"apache-airflow/{airflow.__version__} apache-airflow-providers-clickhouse/{provider_version}"
+    if custom:
+        name = f"{name} ({custom.strip()})"
+    return name
+
+
+class ClickHouseConnection:
+    """
+    Minimal DB-API 2.0 connection adapter wrapping a ``clickhouse_connect`` Client.
+
+    SQL execution is delegated to ``clickhouse_connect.dbapi.cursor.Cursor``, which
+    routes each statement to ``client.query()`` or ``client.command()`` automatically
+    by inspecting the SQL keyword after stripping comments — the same logic used by
+    the ``clickhouse-connect`` SQLAlchemy dialect.
+
+    ClickHouse has no multi-statement transactions.  Every statement is effectively
+    auto-committed, so ``commit()`` and ``rollback()`` are intentional no-ops and
+    the ``autocommit`` attribute is always ``True``.  ``DbApiHook.run()`` checks
+    ``conn.autocommit`` via :meth:`ClickHouseHook.get_autocommit` and skips the
+    ``conn.commit()`` call when it is truthy.
+    """
+
+    # Signals to DbApiHook.get_autocommit() that no explicit commit is needed.
+    autocommit: bool = True
+
+    def __init__(self, client: clickhouse_connect.driver.client.Client) -> None:
+        self._client = client
+
+    def cursor(self) -> ClickHouseDbApiCursor:
+        from clickhouse_connect.dbapi.cursor import Cursor
+
+        return Cursor(self._client)
+
+    def close(self) -> None:
+        self._client.close()
+
+    def commit(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+    def rollback(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+
+class ClickHouseHook(DbApiHook):
+    """
+    Interact with ClickHouse via the HTTP interface (``clickhouse-connect``).
+
+    This hook wraps ``clickhouse_connect.get_client()`` behind a thin DB-API 2.0
+    adapter (:class:`ClickHouseConnection`), so all standard
+    :class:`~airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`
+    features work out of the box (templating, ``handler``, ``split_statements``,
+    etc.).
+
+    :param database: Optional database name.  Overrides the ``schema`` field of the
+        Airflow connection.  Useful when one connection points to a ClickHouse cluster
+        and individual tasks need to target different databases.
+    :param session_settings: Optional dict of ClickHouse session-level settings
+        (e.g. ``{"max_execution_time": 60, "max_threads": 4}``).  Values supplied
+        here are **merged on top of** any ``session_settings`` dict already present
+        in the connection's ``extra`` JSON field, with the constructor argument
+        taking precedence. For a full list of available session settings visit https://clickhouse.com/docs/operations/settings/settings
+    :param client_kwargs: Optional dict of additional keyword arguments forwarded
+        verbatim to ``clickhouse_connect.get_client()``.  Use this to pass any
+        parameter not otherwise exposed by the hook (e.g. ``http_proxy``,
+        ``pool_mgr_params``).  Values supplied here are **merged on top of** any
+        ``client_kwargs`` dict already present in the connection's ``extra`` JSON
+        field, with the constructor argument taking precedence on conflicting keys.
+        Keys that the hook manages itself (``host``, ``port``, ``username``,
+        ``password``, ``database``, ``secure``, ``verify``, ``client_name``,
+        ``settings``) are silently ignored so that hook-managed values always
+        take precedence.
+
+    .. seealso::
+        - `clickhouse-connect documentation <https://clickhouse.com/docs/en/integrations/python>`_
+        - `clickhouse session settings documentation <https://clickhouse.com/docs/operations/settings/settings>`_
+        - :ref:`howto/connection:clickhouse`
+
+    """
+
+    conn_name_attr = "clickhouse_conn_id"
+    default_conn_name = "clickhouse_default"
+    conn_type = "clickhouse"
+    hook_name = "ClickHouse"
+
+    # ClickHouse has no multi-statement transactions; every statement is auto-committed.
+    supports_autocommit = True
+    _test_connection_sql = "SELECT 1"
+
+    def set_autocommit(self, conn: ClickHouseConnection, autocommit: bool) -> None:
+        """No-op: ClickHouse has no transaction support."""
+
+    def get_autocommit(self, conn: ClickHouseConnection) -> bool:
+        """Return ``True``: ClickHouse auto-commits every statement."""
+        return True
+
+    def __init__(
+        self,
+        *args,
+        database: str | None = None,
+        session_settings: dict[str, Any] | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.database = database
+        self.session_settings: dict[str, Any] = session_settings or {}
+        self.client_kwargs: dict[str, Any] = client_kwargs or {}
+
+    @classmethod
+    def get_connection_form_widgets(cls) -> dict[str, Any]:
+        """Return extra connection form widgets shown in the Airflow UI."""
+        from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+        from flask_babel import lazy_gettext
+        from wtforms import BooleanField, IntegerField, StringField
+
+        return {
+            "secure": BooleanField(
+                label=lazy_gettext("Use TLS (HTTPS)"),
+                description="Enable TLS/HTTPS for the connection (default: False).",
+            ),
+            "verify": BooleanField(
+                label=lazy_gettext("Verify SSL Certificate"),
+                description="Verify the server SSL certificate when TLS is enabled (default: True).",
+            ),
+            "connect_timeout": IntegerField(
+                lazy_gettext("Connection Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "send_receive_timeout": IntegerField(
+                lazy_gettext("Query Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "compress": BooleanField(
+                label=lazy_gettext("Enable LZ4 Compression"),
+                description="Compress query results with LZ4 (default: True).",
+            ),
+            "client_name": StringField(
+                lazy_gettext("Client Name (label)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Optional label appended to the Airflow version identifier in the "
+                    "ClickHouse HTTP User-Agent and system.query_log client_name field. "
+                    "The hook always prepends apache-airflow/<version> "
+                    "apache-airflow-providers-clickhouse/<version>."
+                ),
+            ),
+            "session_settings": StringField(
+                lazy_gettext("Session Settings (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "ClickHouse session-level settings as a JSON object, "
+                    'e.g. {"max_execution_time": 60, "max_threads": 4}.'
+                ),
+            ),
+            "client_kwargs": StringField(
+                lazy_gettext("Client kwargs (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Additional keyword arguments forwarded to clickhouse_connect.get_client() "
+                    'as a JSON object, e.g. {"http_proxy": "http://proxy:8080"}. '
+                    "Hook-managed keys (host, port, username, password, database, secure, "
+                    "verify, client_name, settings) are ignored."
+                ),
+            ),
+        }
+
+    @classmethod
+    def get_ui_field_behaviour(cls) -> dict[str, Any]:
+        """Return custom field behaviour for the connection form."""
+        return {
+            "hidden_fields": [],
+            "relabeling": {
+                "schema": "Database",
+            },
+            "placeholders": {
+                "host": "localhost",
+                "port": "8123",
+                "login": "default",
+                "schema": "default",
+                "extra": json.dumps(
+                    {
+                        "secure": False,
+                        "verify": True,
+                        "connect_timeout": 10,
+                        "send_receive_timeout": 300,
+                        "compress": True,
+                        "client_name": "airflow",
+                        "session_settings": {
+                            "max_execution_time": 300,
+                            "max_memory_usage": 10000000000,
+                        },
+                    },
+                    indent=1,
+                ),
+                "secure": "false",
+                "verify": "true",
+                "connect_timeout": "10",
+                "send_receive_timeout": "300",
+                "compress": "true",
+                "client_name": "airflow",
+                "session_settings": '{"max_execution_time": 300}',
+                "client_kwargs": '{"http_proxy": "http://proxy:8080"}',
+            },
+        }

Review Comment:
   For context: https://github.com/apache/airflow/issues/62345 and https://github.com/apache/airflow/issues/63696



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #67080:
URL: https://github.com/apache/airflow/pull/67080#discussion_r3350385065


##########
providers/clickhousedb/src/airflow/providers/clickhousedb/hooks/clickhouse.py:
##########
@@ -0,0 +1,463 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any
+from urllib.parse import quote_plus
+
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+    import clickhouse_connect.driver.client
+    from clickhouse_connect.dbapi.cursor import Cursor as ClickHouseDbApiCursor
+
+# Optional scalar parameters forwarded verbatim to clickhouse_connect.get_client()
+# Note: client_name is handled separately by _build_client_name() so Airflow version
+# info is always embedded in the HTTP User-Agent / system.query_log entry.
+_OPTIONAL_CLIENT_KWARGS = (
+    "connect_timeout",
+    "send_receive_timeout",
+    "compress",
+)
+
+# kwargs that the hook always manages itself.  User-supplied client_kwargs that
+# overlap with these keys are dropped and logged at DEBUG so the hook's own values always win.
+_HOOK_MANAGED_KWARGS: frozenset[str] = frozenset(
+    {
+        "host",
+        "port",
+        "username",
+        "password",
+        "database",
+        "secure",
+        "verify",
+        "client_name",
+        "settings",
+    }
+)
+
+
+def _build_client_name(custom: str | None = None) -> str:
+    """
+    Return the ``client_name`` string passed to ``clickhouse_connect.get_client()``.
+
+    The value is prepended to the HTTP ``User-Agent`` header that ClickHouse records
+    in ``system.query_log``, so every query is traceable back to its Airflow source.
+
+    Format (no custom label)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version>
+
+    Format (with the ``client_name`` extra field set to ``"my-pipeline"``)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version> (my-pipeline)
+
+    The ``clickhouse_connect`` library appends its own token and OS information, so
+    the full ``User-Agent`` looks like::
+
+        apache-airflow/X.Y.Z apache-airflow-providers-clickhouse/X.Y.Z
+        clickhouse-connect/X.Y.Z (lv:py/X.Y.Z; mode:sync; os:linux)
+    """
+    import airflow
+    from airflow.providers.clickhouse import __version__ as provider_version
+
+    name = f"apache-airflow/{airflow.__version__} apache-airflow-providers-clickhouse/{provider_version}"
+    if custom:
+        name = f"{name} ({custom.strip()})"
+    return name
+
+
+class ClickHouseConnection:
+    """
+    Minimal DB-API 2.0 connection adapter wrapping a ``clickhouse_connect`` Client.
+
+    SQL execution is delegated to ``clickhouse_connect.dbapi.cursor.Cursor``, which
+    routes each statement to ``client.query()`` or ``client.command()`` automatically
+    by inspecting the SQL keyword after stripping comments — the same logic used by
+    the ``clickhouse-connect`` SQLAlchemy dialect.
+
+    ClickHouse has no multi-statement transactions.  Every statement is effectively
+    auto-committed, so ``commit()`` and ``rollback()`` are intentional no-ops and
+    the ``autocommit`` attribute is always ``True``.  ``DbApiHook.run()`` checks
+    ``conn.autocommit`` via :meth:`ClickHouseHook.get_autocommit` and skips the
+    ``conn.commit()`` call when it is truthy.
+    """
+
+    # Signals to DbApiHook.get_autocommit() that no explicit commit is needed.
+    autocommit: bool = True
+
+    def __init__(self, client: clickhouse_connect.driver.client.Client) -> None:
+        self._client = client
+
+    def cursor(self) -> ClickHouseDbApiCursor:
+        from clickhouse_connect.dbapi.cursor import Cursor
+
+        return Cursor(self._client)
+
+    def close(self) -> None:
+        self._client.close()
+
+    def commit(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+    def rollback(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+
+class ClickHouseHook(DbApiHook):
+    """
+    Interact with ClickHouse via the HTTP interface (``clickhouse-connect``).
+
+    This hook wraps ``clickhouse_connect.get_client()`` behind a thin DB-API 2.0
+    adapter (:class:`ClickHouseConnection`), so all standard
+    :class:`~airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`
+    features work out of the box (templating, ``handler``, ``split_statements``,
+    etc.).
+
+    :param database: Optional database name.  Overrides the ``schema`` field of the
+        Airflow connection.  Useful when one connection points to a ClickHouse cluster
+        and individual tasks need to target different databases.
+    :param session_settings: Optional dict of ClickHouse session-level settings
+        (e.g. ``{"max_execution_time": 60, "max_threads": 4}``).  Values supplied
+        here are **merged on top of** any ``session_settings`` dict already present
+        in the connection's ``extra`` JSON field, with the constructor argument
+        taking precedence. For a full list of available session settings visit https://clickhouse.com/docs/operations/settings/settings
+    :param client_kwargs: Optional dict of additional keyword arguments forwarded
+        verbatim to ``clickhouse_connect.get_client()``.  Use this to pass any
+        parameter not otherwise exposed by the hook (e.g. ``http_proxy``,
+        ``pool_mgr_params``).  Values supplied here are **merged on top of** any
+        ``client_kwargs`` dict already present in the connection's ``extra`` JSON
+        field, with the constructor argument taking precedence on conflicting keys.
+        Keys that the hook manages itself (``host``, ``port``, ``username``,
+        ``password``, ``database``, ``secure``, ``verify``, ``client_name``,
+        ``settings``) are silently ignored so that hook-managed values always
+        take precedence.
+
+    .. seealso::
+        - `clickhouse-connect documentation <https://clickhouse.com/docs/en/integrations/python>`_
+        - `clickhouse session settings documentation <https://clickhouse.com/docs/operations/settings/settings>`_
+        - :ref:`howto/connection:clickhouse`
+
+    """
+
+    conn_name_attr = "clickhouse_conn_id"
+    default_conn_name = "clickhouse_default"
+    conn_type = "clickhouse"
+    hook_name = "ClickHouse"
+
+    # ClickHouse has no multi-statement transactions; every statement is auto-committed.
+    supports_autocommit = True
+    _test_connection_sql = "SELECT 1"
+
+    def set_autocommit(self, conn: ClickHouseConnection, autocommit: bool) -> None:
+        """No-op: ClickHouse has no transaction support."""
+
+    def get_autocommit(self, conn: ClickHouseConnection) -> bool:
+        """Return ``True``: ClickHouse auto-commits every statement."""
+        return True
+
+    def __init__(
+        self,
+        *args,
+        database: str | None = None,
+        session_settings: dict[str, Any] | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.database = database
+        self.session_settings: dict[str, Any] = session_settings or {}
+        self.client_kwargs: dict[str, Any] = client_kwargs or {}
+
+    @classmethod
+    def get_connection_form_widgets(cls) -> dict[str, Any]:
+        """Return extra connection form widgets shown in the Airflow UI."""
+        from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+        from flask_babel import lazy_gettext
+        from wtforms import BooleanField, IntegerField, StringField
+
+        return {
+            "secure": BooleanField(
+                label=lazy_gettext("Use TLS (HTTPS)"),
+                description="Enable TLS/HTTPS for the connection (default: False).",
+            ),
+            "verify": BooleanField(
+                label=lazy_gettext("Verify SSL Certificate"),
+                description="Verify the server SSL certificate when TLS is enabled (default: True).",
+            ),
+            "connect_timeout": IntegerField(
+                lazy_gettext("Connection Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "send_receive_timeout": IntegerField(
+                lazy_gettext("Query Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "compress": BooleanField(
+                label=lazy_gettext("Enable LZ4 Compression"),
+                description="Compress query results with LZ4 (default: True).",
+            ),
+            "client_name": StringField(
+                lazy_gettext("Client Name (label)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Optional label appended to the Airflow version identifier in the "
+                    "ClickHouse HTTP User-Agent and system.query_log client_name field. "
+                    "The hook always prepends apache-airflow/<version> "
+                    "apache-airflow-providers-clickhouse/<version>."
+                ),
+            ),
+            "session_settings": StringField(
+                lazy_gettext("Session Settings (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "ClickHouse session-level settings as a JSON object, "
+                    'e.g. {"max_execution_time": 60, "max_threads": 4}.'
+                ),
+            ),
+            "client_kwargs": StringField(
+                lazy_gettext("Client kwargs (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Additional keyword arguments forwarded to clickhouse_connect.get_client() "
+                    'as a JSON object, e.g. {"http_proxy": "http://proxy:8080"}. '
+                    "Hook-managed keys (host, port, username, password, database, secure, "
+                    "verify, client_name, settings) are ignored."
+                ),
+            ),
+        }
+
+    @classmethod
+    def get_ui_field_behaviour(cls) -> dict[str, Any]:
+        """Return custom field behaviour for the connection form."""
+        return {
+            "hidden_fields": [],
+            "relabeling": {
+                "schema": "Database",
+            },
+            "placeholders": {
+                "host": "localhost",
+                "port": "8123",
+                "login": "default",
+                "schema": "default",
+                "extra": json.dumps(
+                    {
+                        "secure": False,
+                        "verify": True,
+                        "connect_timeout": 10,
+                        "send_receive_timeout": 300,
+                        "compress": True,
+                        "client_name": "airflow",
+                        "session_settings": {
+                            "max_execution_time": 300,
+                            "max_memory_usage": 10000000000,
+                        },
+                    },
+                    indent=1,
+                ),
+                "secure": "false",
+                "verify": "true",
+                "connect_timeout": "10",
+                "send_receive_timeout": "300",
+                "compress": "true",
+                "client_name": "airflow",
+                "session_settings": '{"max_execution_time": 300}',
+                "client_kwargs": '{"http_proxy": "http://proxy:8080"}',
+            },
+        }
+
+    def _get_client_kwargs(self) -> dict[str, Any]:
+        """
+        Build the keyword-argument dict passed to ``clickhouse_connect.get_client()``.
+
+        Construction order (last write wins):
+
+        1. ``client_kwargs`` from the connection ``extra`` JSON — connection-level
+           passthrough kwargs.
+        2. ``client_kwargs`` constructor argument — task-level overrides merged on top.
+        3. Hook-managed keys (``host``, ``port``, ``username``, ``password``,
+           ``database``, ``secure``, ``verify``, ``client_name``, ``settings``) —
+           always override anything in ``client_kwargs``.
+
+        Optional tuning parameters (``connect_timeout``, ``send_receive_timeout``,
+        ``compress``) are forwarded only when explicitly set in the ``extra`` JSON
+        field so that the driver's own defaults apply otherwise.
+
+        ``client_name`` is always set to a string that identifies the Airflow
+        installation (see :func:`_build_client_name`).  Any ``client_name`` value in
+        ``extra`` is treated as a human-readable label and appended as a comment so
+        it appears alongside the Airflow version info in ``system.query_log``.
+
+        ``session_settings`` from ``extra`` and from the constructor ``session_settings``
+        argument are **merged**, with the constructor argument taking precedence on
+        conflicting keys.
+        """
+        conn = self.get_connection(self.get_conn_id())
+        extra: dict[str, Any] = conn.extra_dejson
+
+        # Merge client_kwargs: extra values are the base, constructor values override.
+        raw_client_kwargs = extra.get("client_kwargs")

Review Comment:
   We normally use `hook_params` you can find many examples for that in the database related hooks across the source code. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on code in PR #67080:
URL: https://github.com/apache/airflow/pull/67080#discussion_r3312712416


##########
providers/clickhouse/src/airflow/providers/clickhouse/hooks/clickhouse.py:
##########
@@ -0,0 +1,366 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any
+from urllib.parse import quote_plus
+
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+    import clickhouse_connect.driver.client
+    from clickhouse_connect.dbapi.cursor import Cursor as ClickHouseDbApiCursor
+
+# Optional scalar parameters forwarded verbatim to clickhouse_connect.get_client()
+# Note: client_name is handled separately by _build_client_name() so Airflow version
+# info is always embedded in the HTTP User-Agent / system.query_log entry.
+_OPTIONAL_CLIENT_KWARGS = (
+    "connect_timeout",
+    "send_receive_timeout",
+    "compress",
+)
+
+# kwargs that the hook always manages itself.  User-supplied client_kwargs that
+# overlap with these keys are dropped and logged at DEBUG so the hook's own values always win.
+_HOOK_MANAGED_KWARGS: frozenset[str] = frozenset(
+    {
+        "host",
+        "port",
+        "username",
+        "password",
+        "database",
+        "secure",
+        "verify",
+        "client_name",
+        "settings",
+    }
+)
+
+
+def _build_client_name(custom: str | None = None) -> str:
+    """
+    Return the ``client_name`` string passed to ``clickhouse_connect.get_client()``.
+
+    The value is prepended to the HTTP ``User-Agent`` header that ClickHouse records
+    in ``system.query_log``, so every query is traceable back to its Airflow source.
+
+    Format (no custom label)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhousedb/<provider_version>
+
+    Format (with the ``client_name`` extra field set to ``"my-pipeline"``)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhousedb/<provider_version> (my-pipeline)
+
+    The ``clickhouse_connect`` library appends its own token and OS information, so
+    the full ``User-Agent`` looks like::
+
+        apache-airflow/X.Y.Z apache-airflow-providers-clickhousedb/X.Y.Z
+        clickhouse-connect/X.Y.Z (lv:py/X.Y.Z; mode:sync; os:linux)
+    """
+    import airflow
+    from airflow.providers.clickhouse import __version__ as provider_version
+
+    name = f"apache-airflow/{airflow.__version__} apache-airflow-providers-clickhousedb/{provider_version}"
+    if custom:
+        name = f"{name} ({custom.strip()})"
+    return name
+
+
+class ClickHouseConnection:
+    """
+    Minimal DB-API 2.0 connection adapter wrapping a ``clickhouse_connect`` Client.
+
+    SQL execution is delegated to ``clickhouse_connect.dbapi.cursor.Cursor``, which
+    routes each statement to ``client.query()`` or ``client.command()`` automatically
+    by inspecting the SQL keyword after stripping comments — the same logic used by
+    the ``clickhouse-connect`` SQLAlchemy dialect.
+
+    ClickHouse has no multi-statement transactions.  Every statement is effectively
+    auto-committed, so ``commit()`` and ``rollback()`` are intentional no-ops and
+    the ``autocommit`` attribute is always ``True``.  ``DbApiHook.run()`` checks
+    ``conn.autocommit`` via :meth:`ClickHouseHook.get_autocommit` and skips the
+    ``conn.commit()`` call when it is truthy.
+    """
+
+    # Signals to DbApiHook.get_autocommit() that no explicit commit is needed.
+    autocommit: bool = True
+
+    def __init__(self, client: clickhouse_connect.driver.client.Client) -> None:
+        self._client = client
+
+    def cursor(self) -> ClickHouseDbApiCursor:
+        from clickhouse_connect.dbapi.cursor import Cursor
+
+        return Cursor(self._client)
+
+    def close(self) -> None:
+        self._client.close()
+
+    def commit(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+    def rollback(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+
+class ClickHouseHook(DbApiHook):
+    """
+    Interact with ClickHouse via the HTTP interface (``clickhouse-connect``).
+
+    This hook wraps ``clickhouse_connect.get_client()`` behind a thin DB-API 2.0
+    adapter (:class:`ClickHouseConnection`), so all standard
+    :class:`~airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`
+    features work out of the box (templating, ``handler``, ``split_statements``,
+    etc.).
+
+    :param database: Optional database name.  Overrides the ``schema`` field of the
+        Airflow connection.  Useful when one connection points to a ClickHouse cluster
+        and individual tasks need to target different databases.
+    :param session_settings: Optional dict of ClickHouse session-level settings
+        (e.g. ``{"max_execution_time": 60, "max_threads": 4}``).  Values supplied
+        here are **merged on top of** any ``session_settings`` dict already present
+        in the connection's ``extra`` JSON field, with the constructor argument
+        taking precedence. For a full list of available session settings visit https://clickhouse.com/docs/operations/settings/settings
+    :param client_kwargs: Optional dict of additional keyword arguments forwarded
+        verbatim to ``clickhouse_connect.get_client()``.  Use this to pass any
+        parameter not otherwise exposed by the hook (e.g. ``http_proxy``,
+        ``pool_mgr_params``).  Values supplied here are **merged on top of** any
+        ``client_kwargs`` dict already present in the connection's ``extra`` JSON
+        field, with the constructor argument taking precedence on conflicting keys.
+        Keys that the hook manages itself (``host``, ``port``, ``username``,
+        ``password``, ``database``, ``secure``, ``verify``, ``client_name``,
+        ``settings``) are silently ignored so that hook-managed values always
+        take precedence.
+
+    .. seealso::
+        - `clickhouse-connect documentation <https://clickhouse.com/docs/en/integrations/python>`_
+        - `clickhouse session settings documentation <https://clickhouse.com/docs/operations/settings/settings>`_
+        - :ref:`howto/connection:clickhouse`
+
+    """
+
+    conn_name_attr = "clickhouse_conn_id"
+    default_conn_name = "clickhouse_default"
+    conn_type = "clickhouse"
+    hook_name = "ClickHouse"
+
+    # ClickHouse has no multi-statement transactions; every statement is auto-committed.
+    supports_autocommit = True
+    _test_connection_sql = "SELECT 1"
+
+    def set_autocommit(self, conn: ClickHouseConnection, autocommit: bool) -> None:
+        """No-op: ClickHouse has no transaction support."""
+
+    def get_autocommit(self, conn: ClickHouseConnection) -> bool:
+        """Return ``True``: ClickHouse auto-commits every statement."""
+        return True
+
+    def __init__(
+        self,
+        *args,
+        database: str | None = None,
+        session_settings: dict[str, Any] | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.database = database
+        self.session_settings: dict[str, Any] = session_settings or {}
+        self.client_kwargs: dict[str, Any] = client_kwargs or {}
+
+    def _get_client_kwargs(self) -> dict[str, Any]:
+        """
+        Build the keyword-argument dict passed to ``clickhouse_connect.get_client()``.
+
+        Construction order (last write wins):
+
+        1. ``client_kwargs`` from the connection ``extra`` JSON — connection-level
+           passthrough kwargs.
+        2. ``client_kwargs`` constructor argument — task-level overrides merged on top.
+        3. Hook-managed keys (``host``, ``port``, ``username``, ``password``,
+           ``database``, ``secure``, ``verify``, ``client_name``, ``settings``) —
+           always override anything in ``client_kwargs``.
+
+        Optional tuning parameters (``connect_timeout``, ``send_receive_timeout``,
+        ``compress``) are forwarded only when explicitly set in the ``extra`` JSON
+        field so that the driver's own defaults apply otherwise.
+
+        ``client_name`` is always set to a string that identifies the Airflow
+        installation (see :func:`_build_client_name`).  Any ``client_name`` value in
+        ``extra`` is treated as a human-readable label and appended as a comment so
+        it appears alongside the Airflow version info in ``system.query_log``.
+
+        ``session_settings`` from ``extra`` and from the constructor ``session_settings``
+        argument are **merged**, with the constructor argument taking precedence on
+        conflicting keys.
+        """
+        conn = self.get_connection(self.get_conn_id())
+        extra: dict[str, Any] = conn.extra_dejson
+
+        # Merge client_kwargs: extra values are the base, constructor values override.
+        raw_client_kwargs = extra.get("client_kwargs")
+        if isinstance(raw_client_kwargs, str):
+            raw_client_kwargs = json.loads(raw_client_kwargs)

Review Comment:
   json.loads() here will raise a raw JSONDecodeError if the user pastes invalid JSON into the connection “extra” fields (which is pretty likely), Can we catch this and raise with a clear message like “Invalid JSON in extra.client_kwargs/session_settings” ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #67080:
URL: https://github.com/apache/airflow/pull/67080#issuecomment-4620427098

   Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #67080:
URL: https://github.com/apache/airflow/pull/67080#issuecomment-4620420209

   For refrence mailing list discussion:
   
   [[PROPOSAL] New Community Provider: ClickHouse](https://lists.apache.org/thread/dpcfv6k60sw2xwmrf6mtokhfwr85n278) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "BentsiLeviav (via GitHub)" <gi...@apache.org>.
BentsiLeviav commented on code in PR #67080:
URL: https://github.com/apache/airflow/pull/67080#discussion_r3313554343


##########
providers/clickhouse/src/airflow/providers/clickhouse/hooks/clickhouse.py:
##########
@@ -0,0 +1,366 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any
+from urllib.parse import quote_plus
+
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+    import clickhouse_connect.driver.client
+    from clickhouse_connect.dbapi.cursor import Cursor as ClickHouseDbApiCursor
+
+# Optional scalar parameters forwarded verbatim to clickhouse_connect.get_client()
+# Note: client_name is handled separately by _build_client_name() so Airflow version
+# info is always embedded in the HTTP User-Agent / system.query_log entry.
+_OPTIONAL_CLIENT_KWARGS = (
+    "connect_timeout",
+    "send_receive_timeout",
+    "compress",
+)
+
+# kwargs that the hook always manages itself.  User-supplied client_kwargs that
+# overlap with these keys are dropped and logged at DEBUG so the hook's own values always win.
+_HOOK_MANAGED_KWARGS: frozenset[str] = frozenset(
+    {
+        "host",
+        "port",
+        "username",
+        "password",
+        "database",
+        "secure",
+        "verify",
+        "client_name",
+        "settings",
+    }
+)
+
+
+def _build_client_name(custom: str | None = None) -> str:
+    """
+    Return the ``client_name`` string passed to ``clickhouse_connect.get_client()``.
+
+    The value is prepended to the HTTP ``User-Agent`` header that ClickHouse records
+    in ``system.query_log``, so every query is traceable back to its Airflow source.
+
+    Format (no custom label)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhousedb/<provider_version>
+
+    Format (with the ``client_name`` extra field set to ``"my-pipeline"``)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhousedb/<provider_version> (my-pipeline)
+
+    The ``clickhouse_connect`` library appends its own token and OS information, so
+    the full ``User-Agent`` looks like::
+
+        apache-airflow/X.Y.Z apache-airflow-providers-clickhousedb/X.Y.Z
+        clickhouse-connect/X.Y.Z (lv:py/X.Y.Z; mode:sync; os:linux)
+    """
+    import airflow
+    from airflow.providers.clickhouse import __version__ as provider_version
+
+    name = f"apache-airflow/{airflow.__version__} apache-airflow-providers-clickhousedb/{provider_version}"
+    if custom:
+        name = f"{name} ({custom.strip()})"
+    return name
+
+
+class ClickHouseConnection:
+    """
+    Minimal DB-API 2.0 connection adapter wrapping a ``clickhouse_connect`` Client.
+
+    SQL execution is delegated to ``clickhouse_connect.dbapi.cursor.Cursor``, which
+    routes each statement to ``client.query()`` or ``client.command()`` automatically
+    by inspecting the SQL keyword after stripping comments — the same logic used by
+    the ``clickhouse-connect`` SQLAlchemy dialect.
+
+    ClickHouse has no multi-statement transactions.  Every statement is effectively
+    auto-committed, so ``commit()`` and ``rollback()`` are intentional no-ops and
+    the ``autocommit`` attribute is always ``True``.  ``DbApiHook.run()`` checks
+    ``conn.autocommit`` via :meth:`ClickHouseHook.get_autocommit` and skips the
+    ``conn.commit()`` call when it is truthy.
+    """
+
+    # Signals to DbApiHook.get_autocommit() that no explicit commit is needed.
+    autocommit: bool = True
+
+    def __init__(self, client: clickhouse_connect.driver.client.Client) -> None:
+        self._client = client
+
+    def cursor(self) -> ClickHouseDbApiCursor:
+        from clickhouse_connect.dbapi.cursor import Cursor
+
+        return Cursor(self._client)
+
+    def close(self) -> None:
+        self._client.close()
+
+    def commit(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+    def rollback(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+
+class ClickHouseHook(DbApiHook):
+    """
+    Interact with ClickHouse via the HTTP interface (``clickhouse-connect``).
+
+    This hook wraps ``clickhouse_connect.get_client()`` behind a thin DB-API 2.0
+    adapter (:class:`ClickHouseConnection`), so all standard
+    :class:`~airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`
+    features work out of the box (templating, ``handler``, ``split_statements``,
+    etc.).
+
+    :param database: Optional database name.  Overrides the ``schema`` field of the
+        Airflow connection.  Useful when one connection points to a ClickHouse cluster
+        and individual tasks need to target different databases.
+    :param session_settings: Optional dict of ClickHouse session-level settings
+        (e.g. ``{"max_execution_time": 60, "max_threads": 4}``).  Values supplied
+        here are **merged on top of** any ``session_settings`` dict already present
+        in the connection's ``extra`` JSON field, with the constructor argument
+        taking precedence. For a full list of available session settings visit https://clickhouse.com/docs/operations/settings/settings
+    :param client_kwargs: Optional dict of additional keyword arguments forwarded
+        verbatim to ``clickhouse_connect.get_client()``.  Use this to pass any
+        parameter not otherwise exposed by the hook (e.g. ``http_proxy``,
+        ``pool_mgr_params``).  Values supplied here are **merged on top of** any
+        ``client_kwargs`` dict already present in the connection's ``extra`` JSON
+        field, with the constructor argument taking precedence on conflicting keys.
+        Keys that the hook manages itself (``host``, ``port``, ``username``,
+        ``password``, ``database``, ``secure``, ``verify``, ``client_name``,
+        ``settings``) are silently ignored so that hook-managed values always
+        take precedence.
+
+    .. seealso::
+        - `clickhouse-connect documentation <https://clickhouse.com/docs/en/integrations/python>`_
+        - `clickhouse session settings documentation <https://clickhouse.com/docs/operations/settings/settings>`_
+        - :ref:`howto/connection:clickhouse`
+
+    """
+
+    conn_name_attr = "clickhouse_conn_id"
+    default_conn_name = "clickhouse_default"
+    conn_type = "clickhouse"
+    hook_name = "ClickHouse"
+
+    # ClickHouse has no multi-statement transactions; every statement is auto-committed.
+    supports_autocommit = True
+    _test_connection_sql = "SELECT 1"
+
+    def set_autocommit(self, conn: ClickHouseConnection, autocommit: bool) -> None:
+        """No-op: ClickHouse has no transaction support."""
+
+    def get_autocommit(self, conn: ClickHouseConnection) -> bool:
+        """Return ``True``: ClickHouse auto-commits every statement."""
+        return True
+
+    def __init__(
+        self,
+        *args,
+        database: str | None = None,
+        session_settings: dict[str, Any] | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.database = database
+        self.session_settings: dict[str, Any] = session_settings or {}
+        self.client_kwargs: dict[str, Any] = client_kwargs or {}
+
+    def _get_client_kwargs(self) -> dict[str, Any]:
+        """
+        Build the keyword-argument dict passed to ``clickhouse_connect.get_client()``.
+
+        Construction order (last write wins):
+
+        1. ``client_kwargs`` from the connection ``extra`` JSON — connection-level
+           passthrough kwargs.
+        2. ``client_kwargs`` constructor argument — task-level overrides merged on top.
+        3. Hook-managed keys (``host``, ``port``, ``username``, ``password``,
+           ``database``, ``secure``, ``verify``, ``client_name``, ``settings``) —
+           always override anything in ``client_kwargs``.
+
+        Optional tuning parameters (``connect_timeout``, ``send_receive_timeout``,
+        ``compress``) are forwarded only when explicitly set in the ``extra`` JSON
+        field so that the driver's own defaults apply otherwise.
+
+        ``client_name`` is always set to a string that identifies the Airflow
+        installation (see :func:`_build_client_name`).  Any ``client_name`` value in
+        ``extra`` is treated as a human-readable label and appended as a comment so
+        it appears alongside the Airflow version info in ``system.query_log``.
+
+        ``session_settings`` from ``extra`` and from the constructor ``session_settings``
+        argument are **merged**, with the constructor argument taking precedence on
+        conflicting keys.
+        """
+        conn = self.get_connection(self.get_conn_id())
+        extra: dict[str, Any] = conn.extra_dejson
+
+        # Merge client_kwargs: extra values are the base, constructor values override.
+        raw_client_kwargs = extra.get("client_kwargs")
+        if isinstance(raw_client_kwargs, str):
+            raw_client_kwargs = json.loads(raw_client_kwargs)

Review Comment:
   Done, adjusted also the relevant tests



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "koletzilla (via GitHub)" <gi...@apache.org>.
koletzilla commented on code in PR #67080:
URL: https://github.com/apache/airflow/pull/67080#discussion_r3304958452


##########
providers/clickhouse/src/airflow/providers/clickhouse/hooks/clickhouse.py:
##########
@@ -0,0 +1,463 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any
+from urllib.parse import quote_plus
+
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+    import clickhouse_connect.driver.client
+    from clickhouse_connect.dbapi.cursor import Cursor as ClickHouseDbApiCursor
+
+# Optional scalar parameters forwarded verbatim to clickhouse_connect.get_client()
+# Note: client_name is handled separately by _build_client_name() so Airflow version
+# info is always embedded in the HTTP User-Agent / system.query_log entry.
+_OPTIONAL_CLIENT_KWARGS = (
+    "connect_timeout",
+    "send_receive_timeout",
+    "compress",
+)
+
+# kwargs that the hook always manages itself.  User-supplied client_kwargs that
+# overlap with these keys are dropped and logged at DEBUG so the hook's own values always win.
+_HOOK_MANAGED_KWARGS: frozenset[str] = frozenset(
+    {
+        "host",
+        "port",
+        "username",
+        "password",
+        "database",
+        "secure",
+        "verify",
+        "client_name",
+        "settings",
+    }
+)
+
+
+def _build_client_name(custom: str | None = None) -> str:
+    """
+    Return the ``client_name`` string passed to ``clickhouse_connect.get_client()``.
+
+    The value is prepended to the HTTP ``User-Agent`` header that ClickHouse records
+    in ``system.query_log``, so every query is traceable back to its Airflow source.
+
+    Format (no custom label)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version>
+
+    Format (with the ``client_name`` extra field set to ``"my-pipeline"``)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version> (my-pipeline)
+
+    The ``clickhouse_connect`` library appends its own token and OS information, so
+    the full ``User-Agent`` looks like::
+
+        apache-airflow/X.Y.Z apache-airflow-providers-clickhouse/X.Y.Z
+        clickhouse-connect/X.Y.Z (lv:py/X.Y.Z; mode:sync; os:linux)
+    """
+    import airflow
+    from airflow.providers.clickhouse import __version__ as provider_version
+
+    name = f"apache-airflow/{airflow.__version__} apache-airflow-providers-clickhouse/{provider_version}"
+    if custom:
+        name = f"{name} ({custom.strip()})"
+    return name
+
+
+class ClickHouseConnection:
+    """
+    Minimal DB-API 2.0 connection adapter wrapping a ``clickhouse_connect`` Client.
+
+    SQL execution is delegated to ``clickhouse_connect.dbapi.cursor.Cursor``, which
+    routes each statement to ``client.query()`` or ``client.command()`` automatically
+    by inspecting the SQL keyword after stripping comments — the same logic used by
+    the ``clickhouse-connect`` SQLAlchemy dialect.
+
+    ClickHouse has no multi-statement transactions.  Every statement is effectively
+    auto-committed, so ``commit()`` and ``rollback()`` are intentional no-ops and
+    the ``autocommit`` attribute is always ``True``.  ``DbApiHook.run()`` checks
+    ``conn.autocommit`` via :meth:`ClickHouseHook.get_autocommit` and skips the
+    ``conn.commit()`` call when it is truthy.
+    """
+
+    # Signals to DbApiHook.get_autocommit() that no explicit commit is needed.
+    autocommit: bool = True
+
+    def __init__(self, client: clickhouse_connect.driver.client.Client) -> None:
+        self._client = client
+
+    def cursor(self) -> ClickHouseDbApiCursor:
+        from clickhouse_connect.dbapi.cursor import Cursor
+
+        return Cursor(self._client)
+
+    def close(self) -> None:
+        self._client.close()
+
+    def commit(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+    def rollback(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+
+class ClickHouseHook(DbApiHook):
+    """
+    Interact with ClickHouse via the HTTP interface (``clickhouse-connect``).
+
+    This hook wraps ``clickhouse_connect.get_client()`` behind a thin DB-API 2.0
+    adapter (:class:`ClickHouseConnection`), so all standard
+    :class:`~airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`
+    features work out of the box (templating, ``handler``, ``split_statements``,
+    etc.).
+
+    :param database: Optional database name.  Overrides the ``schema`` field of the
+        Airflow connection.  Useful when one connection points to a ClickHouse cluster
+        and individual tasks need to target different databases.
+    :param session_settings: Optional dict of ClickHouse session-level settings
+        (e.g. ``{"max_execution_time": 60, "max_threads": 4}``).  Values supplied
+        here are **merged on top of** any ``session_settings`` dict already present
+        in the connection's ``extra`` JSON field, with the constructor argument
+        taking precedence. For a full list of available session settings visit https://clickhouse.com/docs/operations/settings/settings
+    :param client_kwargs: Optional dict of additional keyword arguments forwarded
+        verbatim to ``clickhouse_connect.get_client()``.  Use this to pass any
+        parameter not otherwise exposed by the hook (e.g. ``http_proxy``,
+        ``pool_mgr_params``).  Values supplied here are **merged on top of** any
+        ``client_kwargs`` dict already present in the connection's ``extra`` JSON
+        field, with the constructor argument taking precedence on conflicting keys.
+        Keys that the hook manages itself (``host``, ``port``, ``username``,
+        ``password``, ``database``, ``secure``, ``verify``, ``client_name``,
+        ``settings``) are silently ignored so that hook-managed values always
+        take precedence.
+
+    .. seealso::
+        - `clickhouse-connect documentation <https://clickhouse.com/docs/en/integrations/python>`_
+        - `clickhouse session settings documentation <https://clickhouse.com/docs/operations/settings/settings>`_
+        - :ref:`howto/connection:clickhouse`
+
+    """
+
+    conn_name_attr = "clickhouse_conn_id"
+    default_conn_name = "clickhouse_default"
+    conn_type = "clickhouse"
+    hook_name = "ClickHouse"
+
+    # ClickHouse has no multi-statement transactions; every statement is auto-committed.
+    supports_autocommit = True
+    _test_connection_sql = "SELECT 1"
+
+    def set_autocommit(self, conn: ClickHouseConnection, autocommit: bool) -> None:
+        """No-op: ClickHouse has no transaction support."""
+
+    def get_autocommit(self, conn: ClickHouseConnection) -> bool:
+        """Return ``True``: ClickHouse auto-commits every statement."""
+        return True
+
+    def __init__(
+        self,
+        *args,
+        database: str | None = None,
+        session_settings: dict[str, Any] | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.database = database
+        self.session_settings: dict[str, Any] = session_settings or {}
+        self.client_kwargs: dict[str, Any] = client_kwargs or {}
+
+    @classmethod
+    def get_connection_form_widgets(cls) -> dict[str, Any]:
+        """Return extra connection form widgets shown in the Airflow UI."""
+        from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+        from flask_babel import lazy_gettext
+        from wtforms import BooleanField, IntegerField, StringField
+
+        return {
+            "secure": BooleanField(
+                label=lazy_gettext("Use TLS (HTTPS)"),
+                description="Enable TLS/HTTPS for the connection (default: False).",
+            ),
+            "verify": BooleanField(
+                label=lazy_gettext("Verify SSL Certificate"),
+                description="Verify the server SSL certificate when TLS is enabled (default: True).",
+            ),
+            "connect_timeout": IntegerField(
+                lazy_gettext("Connection Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "send_receive_timeout": IntegerField(
+                lazy_gettext("Query Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "compress": BooleanField(
+                label=lazy_gettext("Enable LZ4 Compression"),
+                description="Compress query results with LZ4 (default: True).",
+            ),
+            "client_name": StringField(
+                lazy_gettext("Client Name (label)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Optional label appended to the Airflow version identifier in the "
+                    "ClickHouse HTTP User-Agent and system.query_log client_name field. "
+                    "The hook always prepends apache-airflow/<version> "
+                    "apache-airflow-providers-clickhouse/<version>."
+                ),
+            ),
+            "session_settings": StringField(
+                lazy_gettext("Session Settings (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "ClickHouse session-level settings as a JSON object, "
+                    'e.g. {"max_execution_time": 60, "max_threads": 4}.'
+                ),
+            ),
+            "client_kwargs": StringField(
+                lazy_gettext("Client kwargs (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Additional keyword arguments forwarded to clickhouse_connect.get_client() "
+                    'as a JSON object, e.g. {"http_proxy": "http://proxy:8080"}. '
+                    "Hook-managed keys (host, port, username, password, database, secure, "
+                    "verify, client_name, settings) are ignored."
+                ),
+            ),
+        }
+
+    @classmethod
+    def get_ui_field_behaviour(cls) -> dict[str, Any]:
+        """Return custom field behaviour for the connection form."""
+        return {
+            "hidden_fields": [],
+            "relabeling": {
+                "schema": "Database",
+            },
+            "placeholders": {
+                "host": "localhost",
+                "port": "8123",
+                "login": "default",
+                "schema": "default",
+                "extra": json.dumps(
+                    {
+                        "secure": False,
+                        "verify": True,
+                        "connect_timeout": 10,
+                        "send_receive_timeout": 300,
+                        "compress": True,
+                        "client_name": "airflow",
+                        "session_settings": {
+                            "max_execution_time": 300,
+                            "max_memory_usage": 10000000000,
+                        },
+                    },
+                    indent=1,
+                ),
+                "secure": "false",
+                "verify": "true",
+                "connect_timeout": "10",
+                "send_receive_timeout": "300",
+                "compress": "true",
+                "client_name": "airflow",
+                "session_settings": '{"max_execution_time": 300}',
+                "client_kwargs": '{"http_proxy": "http://proxy:8080"}',
+            },
+        }

Review Comment:
   Seems like both `get_connection_form_widgets` and `get_ui_field_behaviour` are deprecated and from Airflow 3.2, we don't need to redefine them as we already have that info in providers.yaml. So looks like we should remove that code from here 🤔 
   
   Sources: 
   - https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#custom-connection-fields
   - https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#defining-connection-ui-metadata-in-python-legacy
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "joe-clickhouse (via GitHub)" <gi...@apache.org>.
joe-clickhouse commented on code in PR #67080:
URL: https://github.com/apache/airflow/pull/67080#discussion_r3277609320


##########
providers/clickhouse/src/airflow/providers/clickhouse/hooks/clickhouse.py:
##########
@@ -0,0 +1,463 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any
+from urllib.parse import quote_plus
+
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+    import clickhouse_connect.driver.client
+    from clickhouse_connect.dbapi.cursor import Cursor as ClickHouseDbApiCursor
+
+# Optional scalar parameters forwarded verbatim to clickhouse_connect.get_client()
+# Note: client_name is handled separately by _build_client_name() so Airflow version
+# info is always embedded in the HTTP User-Agent / system.query_log entry.
+_OPTIONAL_CLIENT_KWARGS = (
+    "connect_timeout",
+    "send_receive_timeout",
+    "compress",
+)
+
+# kwargs that the hook always manages itself.  User-supplied client_kwargs that
+# overlap with these keys are dropped and logged at DEBUG so the hook's own values always win.
+_HOOK_MANAGED_KWARGS: frozenset[str] = frozenset(
+    {
+        "host",
+        "port",
+        "username",
+        "password",
+        "database",
+        "secure",
+        "verify",
+        "client_name",
+        "settings",
+    }
+)
+
+
+def _build_client_name(custom: str | None = None) -> str:
+    """
+    Return the ``client_name`` string passed to ``clickhouse_connect.get_client()``.
+
+    The value is prepended to the HTTP ``User-Agent`` header that ClickHouse records
+    in ``system.query_log``, so every query is traceable back to its Airflow source.
+
+    Format (no custom label)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version>
+
+    Format (with the ``client_name`` extra field set to ``"my-pipeline"``)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version> (my-pipeline)
+
+    The ``clickhouse_connect`` library appends its own token and OS information, so
+    the full ``User-Agent`` looks like::
+
+        apache-airflow/X.Y.Z apache-airflow-providers-clickhouse/X.Y.Z
+        clickhouse-connect/X.Y.Z (lv:py/X.Y.Z; mode:sync; os:linux)
+    """
+    import airflow
+    from airflow.providers.clickhouse import __version__ as provider_version
+
+    name = f"apache-airflow/{airflow.__version__} apache-airflow-providers-clickhouse/{provider_version}"
+    if custom:
+        name = f"{name} ({custom.strip()})"
+    return name
+
+
+class ClickHouseConnection:
+    """
+    Minimal DB-API 2.0 connection adapter wrapping a ``clickhouse_connect`` Client.
+
+    SQL execution is delegated to ``clickhouse_connect.dbapi.cursor.Cursor``, which
+    routes each statement to ``client.query()`` or ``client.command()`` automatically
+    by inspecting the SQL keyword after stripping comments — the same logic used by
+    the ``clickhouse-connect`` SQLAlchemy dialect.
+
+    ClickHouse has no multi-statement transactions.  Every statement is effectively
+    auto-committed, so ``commit()`` and ``rollback()`` are intentional no-ops and
+    the ``autocommit`` attribute is always ``True``.  ``DbApiHook.run()`` checks
+    ``conn.autocommit`` via :meth:`ClickHouseHook.get_autocommit` and skips the
+    ``conn.commit()`` call when it is truthy.
+    """
+
+    # Signals to DbApiHook.get_autocommit() that no explicit commit is needed.
+    autocommit: bool = True
+
+    def __init__(self, client: clickhouse_connect.driver.client.Client) -> None:
+        self._client = client
+
+    def cursor(self) -> ClickHouseDbApiCursor:
+        from clickhouse_connect.dbapi.cursor import Cursor
+
+        return Cursor(self._client)
+
+    def close(self) -> None:
+        self._client.close()
+
+    def commit(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+    def rollback(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+
+class ClickHouseHook(DbApiHook):
+    """
+    Interact with ClickHouse via the HTTP interface (``clickhouse-connect``).
+
+    This hook wraps ``clickhouse_connect.get_client()`` behind a thin DB-API 2.0
+    adapter (:class:`ClickHouseConnection`), so all standard
+    :class:`~airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`
+    features work out of the box (templating, ``handler``, ``split_statements``,
+    etc.).
+
+    :param database: Optional database name.  Overrides the ``schema`` field of the
+        Airflow connection.  Useful when one connection points to a ClickHouse cluster
+        and individual tasks need to target different databases.
+    :param session_settings: Optional dict of ClickHouse session-level settings
+        (e.g. ``{"max_execution_time": 60, "max_threads": 4}``).  Values supplied
+        here are **merged on top of** any ``session_settings`` dict already present
+        in the connection's ``extra`` JSON field, with the constructor argument
+        taking precedence. For a full list of available session settings visit https://clickhouse.com/docs/operations/settings/settings
+    :param client_kwargs: Optional dict of additional keyword arguments forwarded
+        verbatim to ``clickhouse_connect.get_client()``.  Use this to pass any
+        parameter not otherwise exposed by the hook (e.g. ``http_proxy``,
+        ``pool_mgr_params``).  Values supplied here are **merged on top of** any
+        ``client_kwargs`` dict already present in the connection's ``extra`` JSON
+        field, with the constructor argument taking precedence on conflicting keys.
+        Keys that the hook manages itself (``host``, ``port``, ``username``,
+        ``password``, ``database``, ``secure``, ``verify``, ``client_name``,
+        ``settings``) are silently ignored so that hook-managed values always
+        take precedence.
+
+    .. seealso::
+        - `clickhouse-connect documentation <https://clickhouse.com/docs/en/integrations/python>`_
+        - `clickhouse session settings documentation <https://clickhouse.com/docs/operations/settings/settings>`_
+        - :ref:`howto/connection:clickhouse`
+
+    """
+
+    conn_name_attr = "clickhouse_conn_id"
+    default_conn_name = "clickhouse_default"
+    conn_type = "clickhouse"
+    hook_name = "ClickHouse"
+
+    # ClickHouse has no multi-statement transactions; every statement is auto-committed.
+    supports_autocommit = True
+    _test_connection_sql = "SELECT 1"
+
+    def set_autocommit(self, conn: ClickHouseConnection, autocommit: bool) -> None:
+        """No-op: ClickHouse has no transaction support."""
+
+    def get_autocommit(self, conn: ClickHouseConnection) -> bool:
+        """Return ``True``: ClickHouse auto-commits every statement."""
+        return True
+
+    def __init__(
+        self,
+        *args,
+        database: str | None = None,
+        session_settings: dict[str, Any] | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.database = database
+        self.session_settings: dict[str, Any] = session_settings or {}
+        self.client_kwargs: dict[str, Any] = client_kwargs or {}
+
+    @classmethod
+    def get_connection_form_widgets(cls) -> dict[str, Any]:
+        """Return extra connection form widgets shown in the Airflow UI."""
+        from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+        from flask_babel import lazy_gettext
+        from wtforms import BooleanField, IntegerField, StringField
+
+        return {
+            "secure": BooleanField(
+                label=lazy_gettext("Use TLS (HTTPS)"),
+                description="Enable TLS/HTTPS for the connection (default: False).",
+            ),
+            "verify": BooleanField(
+                label=lazy_gettext("Verify SSL Certificate"),
+                description="Verify the server SSL certificate when TLS is enabled (default: True).",
+            ),
+            "connect_timeout": IntegerField(
+                lazy_gettext("Connection Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "send_receive_timeout": IntegerField(
+                lazy_gettext("Query Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "compress": BooleanField(
+                label=lazy_gettext("Enable LZ4 Compression"),
+                description="Compress query results with LZ4 (default: True).",
+            ),
+            "client_name": StringField(
+                lazy_gettext("Client Name (label)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Optional label appended to the Airflow version identifier in the "
+                    "ClickHouse HTTP User-Agent and system.query_log client_name field. "
+                    "The hook always prepends apache-airflow/<version> "
+                    "apache-airflow-providers-clickhouse/<version>."
+                ),
+            ),
+            "session_settings": StringField(
+                lazy_gettext("Session Settings (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "ClickHouse session-level settings as a JSON object, "
+                    'e.g. {"max_execution_time": 60, "max_threads": 4}.'
+                ),
+            ),
+            "client_kwargs": StringField(
+                lazy_gettext("Client kwargs (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Additional keyword arguments forwarded to clickhouse_connect.get_client() "
+                    'as a JSON object, e.g. {"http_proxy": "http://proxy:8080"}. '
+                    "Hook-managed keys (host, port, username, password, database, secure, "
+                    "verify, client_name, settings) are ignored."
+                ),
+            ),
+        }
+
+    @classmethod
+    def get_ui_field_behaviour(cls) -> dict[str, Any]:
+        """Return custom field behaviour for the connection form."""
+        return {
+            "hidden_fields": [],
+            "relabeling": {
+                "schema": "Database",
+            },
+            "placeholders": {
+                "host": "localhost",
+                "port": "8123",
+                "login": "default",
+                "schema": "default",
+                "extra": json.dumps(
+                    {
+                        "secure": False,
+                        "verify": True,
+                        "connect_timeout": 10,
+                        "send_receive_timeout": 300,
+                        "compress": True,
+                        "client_name": "airflow",
+                        "session_settings": {
+                            "max_execution_time": 300,
+                            "max_memory_usage": 10000000000,
+                        },
+                    },
+                    indent=1,
+                ),
+                "secure": "false",
+                "verify": "true",
+                "connect_timeout": "10",
+                "send_receive_timeout": "300",
+                "compress": "true",
+                "client_name": "airflow",
+                "session_settings": '{"max_execution_time": 300}',
+                "client_kwargs": '{"http_proxy": "http://proxy:8080"}',
+            },
+        }
+
+    def _get_client_kwargs(self) -> dict[str, Any]:
+        """
+        Build the keyword-argument dict passed to ``clickhouse_connect.get_client()``.
+
+        Construction order (last write wins):
+
+        1. ``client_kwargs`` from the connection ``extra`` JSON — connection-level
+           passthrough kwargs.
+        2. ``client_kwargs`` constructor argument — task-level overrides merged on top.
+        3. Hook-managed keys (``host``, ``port``, ``username``, ``password``,
+           ``database``, ``secure``, ``verify``, ``client_name``, ``settings``) —
+           always override anything in ``client_kwargs``.
+
+        Optional tuning parameters (``connect_timeout``, ``send_receive_timeout``,
+        ``compress``) are forwarded only when explicitly set in the ``extra`` JSON
+        field so that the driver's own defaults apply otherwise.
+
+        ``client_name`` is always set to a string that identifies the Airflow
+        installation (see :func:`_build_client_name`).  Any ``client_name`` value in
+        ``extra`` is treated as a human-readable label and appended as a comment so
+        it appears alongside the Airflow version info in ``system.query_log``.
+
+        ``session_settings`` from ``extra`` and from the constructor ``session_settings``
+        argument are **merged**, with the constructor argument taking precedence on
+        conflicting keys.
+        """
+        conn = self.get_connection(self.get_conn_id())
+        extra: dict[str, Any] = conn.extra_dejson
+
+        # Merge client_kwargs: extra values are the base, constructor values override.
+        raw_client_kwargs = extra.get("client_kwargs")

Review Comment:
   I'm not an Airflow expert, but I think this might expose low-level clickhouse-connect client options at the Connection extra level, which may be too broad for an Airflow provider.
   
   From a clickhouse-connect perspective, arbitrary client kwargs are useful when the caller owns the Python code. So I think `ClickHouseHook(client_kwargs=...)` is reasonable at the Dag author level. But for Connection extras, the provider should probably only expose a finite set of reviewed and documented fields like `host`, `port`, `username`, `password`, `database`, `secure`, `verify`, timeouts, compression, etc.
   
   It looks like `_HOOK_MANAGED_KWARGS` prevents overriding hook-owned fields, which is good, but it still allows any other `clickhouse_connect.get_client()` kwarg through. That means a Connection configuration user can configure low-level transport and security behavior on behalf of any Dag that uses the connection.
   
   Long story short, I think we should keep arbitrary `client_kwargs` as a hook constructor argument only, and promote individual kwargs to Connection extras when the provider intentionally supports and documents them. This seems more consistent with Airflow's guidance to allowlist Connection extras rather than forwarding arbitrary kwargs into underlying libraries, but I'll defer to the Airflow maintainers on the provider policy.
   
   Reference on Connection configuration users:
   https://github.com/apache/airflow/blob/cb2aaf41ed286222f44cf5f498db42a730372c72/airflow-core/docs/security/security_model.rst?plain=1#L133



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "BentsiLeviav (via GitHub)" <gi...@apache.org>.
BentsiLeviav commented on code in PR #67080:
URL: https://github.com/apache/airflow/pull/67080#discussion_r3275617470


##########
providers/clickhouse/tests/unit/clickhouse/hooks/test_clickhouse.py:
##########
@@ -0,0 +1,1344 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import json
+import re
+from unittest.mock import ANY, MagicMock, patch
+
+import pytest
+
+from airflow.models import Connection
+from airflow.providers.clickhouse.hooks.clickhouse import (
+    ClickHouseConnection,
+    ClickHouseHook,
+)
+
+# ---------------------------------------------------------------------------
+# Fixtures / shared connection definitions
+# ---------------------------------------------------------------------------
+
+BASE_CONN = Connection(
+    conn_id="clickhouse_test",
+    conn_type="clickhouse",
+    host="clickhouse-host",
+    port=8123,
+    login="user",
+    password="secret",
+    schema="analytics",
+)
+
+BASE_CONN_EXTRA = Connection(
+    conn_id="clickhouse_test_extra",
+    conn_type="clickhouse",
+    host="secure-host",
+    port=8443,
+    login="admin",
+    password="topsecret",
+    schema="prod",
+    extra=json.dumps(
+        {
+            "secure": True,
+            "verify": False,
+            "connect_timeout": 30,
+            "send_receive_timeout": 600,
+            "compress": False,
+            "client_name": "my-airflow",
+        }
+    ),
+)
+
+MINIMAL_CONN = Connection(
+    conn_id="clickhouse_minimal",
+    conn_type="clickhouse",
+)
+
+CONN_WITH_CLIENT_KWARGS = Connection(
+    conn_id="clickhouse_client_kwargs",
+    conn_type="clickhouse",
+    host="ch-host",
+    port=8123,
+    login="user",
+    password="pass",
+    schema="db",
+    extra=json.dumps(
+        {
+            "client_kwargs": {
+                "http_proxy": "http://proxy:8080",
+                "pool_mgr_params": {"num_pools": 4},
+            }
+        }
+    ),
+)
+
+CONN_WITH_CLIENT_KWARGS_STR = Connection(
+    conn_id="clickhouse_client_kwargs_str",
+    conn_type="clickhouse",
+    host="ch-host",
+    port=8123,
+    login="user",
+    password="pass",
+    schema="db",
+    extra=json.dumps(
+        {
+            # client_kwargs stored as a JSON string (as the UI text field stores it)
+            "client_kwargs": '{"http_proxy": "http://proxy:8080"}'
+        }
+    ),
+)
+
+CONN_WITH_SESSION_SETTINGS = Connection(
+    conn_id="clickhouse_session",
+    conn_type="clickhouse",
+    host="ch-host",
+    port=8123,
+    login="user",
+    password="pass",
+    schema="db",
+    extra=json.dumps(
+        {
+            "session_settings": {
+                "max_execution_time": 120,
+                "max_threads": 4,
+            }
+        }
+    ),
+)
+
+CONN_WITH_SESSION_SETTINGS_STR = Connection(
+    conn_id="clickhouse_session_str",
+    conn_type="clickhouse",
+    host="ch-host",
+    port=8123,
+    login="user",
+    password="pass",
+    schema="db",
+    extra=json.dumps(
+        {
+            # session_settings stored as a JSON string (as the UI text field stores it)
+            "session_settings": '{"max_execution_time": 60}'
+        }
+    ),
+)
+
+
+# ---------------------------------------------------------------------------
+# Tests: get_conn
+# ---------------------------------------------------------------------------
+
+
+class TestClickHouseHookGetConn:
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    @patch("clickhouse_connect.get_client")
+    def test_get_conn_basic(self, mock_get_client, mock_get_connection):
+        mock_get_connection.return_value = BASE_CONN
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_test")
+        conn = hook.get_conn()
+
+        mock_get_client.assert_called_once_with(
+            host="clickhouse-host",
+            port=8123,
+            username="user",
+            password="secret",
+            database="analytics",
+            secure=False,
+            verify=True,
+            client_name=ANY,  # always set; format verified in TestClickHouseHookClientName
+        )
+        assert isinstance(conn, ClickHouseConnection)
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    @patch("clickhouse_connect.get_client")
+    def test_get_conn_with_extra_params(self, mock_get_client, mock_get_connection):
+        mock_get_connection.return_value = BASE_CONN_EXTRA
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_test_extra")
+        hook.get_conn()
+
+        mock_get_client.assert_called_once_with(
+            host="secure-host",
+            port=8443,
+            username="admin",
+            password="topsecret",
+            database="prod",
+            secure=True,
+            verify=False,
+            connect_timeout=30,
+            send_receive_timeout=600,
+            compress=False,
+            client_name=ANY,  # contains Airflow version + "(my-airflow)" comment
+        )
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    @patch("clickhouse_connect.get_client")
+    def test_get_conn_minimal_defaults(self, mock_get_client, mock_get_connection):
+        """All connection fields absent → sensible defaults applied."""
+        mock_get_connection.return_value = MINIMAL_CONN
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_minimal")
+        hook.get_conn()
+
+        mock_get_client.assert_called_once_with(
+            host="localhost",
+            port=8123,
+            username="default",
+            password="",
+            database="default",
+            secure=False,
+            verify=True,
+            client_name=ANY,  # always set; format verified in TestClickHouseHookClientName
+        )
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    @patch("clickhouse_connect.get_client")
+    def test_get_conn_database_overrides_connection_schema(self, mock_get_client, mock_get_connection):
+        """database constructor arg must override the schema field from the connection."""
+        mock_get_connection.return_value = BASE_CONN  # schema="analytics"
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_test", database="overridden_db")
+        hook.get_conn()
+
+        call_kwargs = mock_get_client.call_args.kwargs
+        assert call_kwargs["database"] == "overridden_db"
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    @patch("clickhouse_connect.get_client")
+    def test_get_conn_no_database_falls_back_to_connection_schema(self, mock_get_client, mock_get_connection):
+        """Without a database constructor arg the connection schema is used."""
+        mock_get_connection.return_value = BASE_CONN  # schema="analytics"
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_test")
+        hook.get_conn()
+
+        call_kwargs = mock_get_client.call_args.kwargs
+        assert call_kwargs["database"] == "analytics"
+
+
+# ---------------------------------------------------------------------------
+# Tests: get_uri
+# ---------------------------------------------------------------------------
+
+
+class TestClickHouseHookGetUri:
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    def test_get_uri_with_password(self, mock_get_connection):
+        mock_get_connection.return_value = BASE_CONN
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_test")
+        uri = hook.get_uri()
+
+        assert uri == "clickhousedb://user:secret@clickhouse-host:8123/analytics"
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    def test_get_uri_without_password(self, mock_get_connection):
+        conn = Connection(
+            conn_id="clickhouse_nopass",
+            conn_type="clickhouse",
+            host="myhost",
+            port=8123,
+            login="reader",
+            schema="logs",
+        )
+        mock_get_connection.return_value = conn
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_nopass")
+        uri = hook.get_uri()
+
+        assert uri == "clickhousedb://reader@myhost:8123/logs"
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    def test_get_uri_password_url_encoded(self, mock_get_connection):
+        """Special characters in the password must be percent-encoded."""
+        conn = Connection(
+            conn_id="clickhouse_special",
+            conn_type="clickhouse",
+            host="host",
+            port=8123,
+            login="user",
+            password="p@ss/w0rd!",
+            schema="db",
+        )
+        mock_get_connection.return_value = conn
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_special")
+        uri = hook.get_uri()
+
+        assert "@" not in uri.split("://")[1].split("@")[0]  # password part is encoded
+        assert "p%40ss" in uri or "p%40" in uri  # @ is encoded
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    def test_get_uri_defaults(self, mock_get_connection):
+        mock_get_connection.return_value = MINIMAL_CONN
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_minimal")
+        uri = hook.get_uri()
+
+        assert uri == "clickhousedb://default@localhost:8123/default"
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    def test_get_uri_database_overrides_connection_schema(self, mock_get_connection):
+        """database constructor arg must appear in the URI instead of the connection schema."""
+        mock_get_connection.return_value = BASE_CONN  # schema="analytics"
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_test", database="overridden_db")
+        uri = hook.get_uri()
+
+        assert uri.endswith("/overridden_db")
+        assert "analytics" not in uri
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    def test_get_uri_secure_uses_clickhousedbs_scheme(self, mock_get_connection):
+        """secure=True in extra must produce the clickhousedbs:// (HTTPS) scheme."""
+        conn = Connection(
+            conn_id="ch_secure",
+            conn_type="clickhouse",
+            host="secure-host",
+            port=8443,
+            login="user",
+            password="pass",
+            schema="db",
+            extra=json.dumps({"secure": True}),
+        )
+        mock_get_connection.return_value = conn
+        uri = ClickHouseHook(clickhouse_conn_id="ch_secure").get_uri()
+        assert uri == "clickhousedbs://user:pass@secure-host:8443/db"

Review Comment:
   I renamed the test to `test_get_uri_secure_adds_query_param` and updated the assertion to assert `uri ==
     "clickhousedb://user:pass@secure-host:8443/db?secure=true"`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "joe-clickhouse (via GitHub)" <gi...@apache.org>.
joe-clickhouse commented on code in PR #67080:
URL: https://github.com/apache/airflow/pull/67080#discussion_r3262249915


##########
providers/clickhouse/src/airflow/providers/clickhouse/hooks/clickhouse.py:
##########
@@ -0,0 +1,434 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any
+from urllib.parse import quote_plus
+
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+    import clickhouse_connect.driver.client
+    from clickhouse_connect.dbapi.cursor import Cursor as ClickHouseDbApiCursor
+
+# Optional scalar parameters forwarded verbatim to clickhouse_connect.get_client()
+# Note: client_name is handled separately by _build_client_name() so Airflow version
+# info is always embedded in the HTTP User-Agent / system.query_log entry.
+_OPTIONAL_CLIENT_KWARGS = (
+    "connect_timeout",
+    "send_receive_timeout",
+    "compress",
+)
+
+# kwargs that the hook always manages itself.  User-supplied client_kwargs that
+# overlap with these keys are dropped and logged at DEBUG so the hook's own values always win.
+_HOOK_MANAGED_KWARGS: frozenset[str] = frozenset(
+    {
+        "host",
+        "port",
+        "username",
+        "password",
+        "database",
+        "secure",
+        "verify",
+        "client_name",
+        "settings",
+    }
+)
+
+
+def _build_client_name(custom: str | None = None) -> str:
+    """
+    Return the ``client_name`` string passed to ``clickhouse_connect.get_client()``.
+
+    The value is prepended to the HTTP ``User-Agent`` header that ClickHouse records
+    in ``system.query_log``, so every query is traceable back to its Airflow source.
+
+    Format (no custom label)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version>
+
+    Format (with the ``client_name`` extra field set to ``"my-pipeline"``)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version> (my-pipeline)
+
+    The ``clickhouse_connect`` library appends its own token and OS information, so
+    the full ``User-Agent`` looks like::
+
+        apache-airflow/X.Y.Z apache-airflow-providers-clickhouse/X.Y.Z
+        clickhouse-connect/X.Y.Z (lv:py/X.Y.Z; mode:sync; os:linux)
+    """
+    import airflow
+    from airflow.providers.clickhouse import __version__ as provider_version
+
+    name = f"apache-airflow/{airflow.__version__} apache-airflow-providers-clickhouse/{provider_version}"
+    if custom:
+        name = f"{name} ({custom.strip()})"
+    return name
+
+
+class ClickHouseConnection:
+    """
+    Minimal DB-API 2.0 connection adapter wrapping a ``clickhouse_connect`` Client.
+
+    SQL execution is delegated to ``clickhouse_connect.dbapi.cursor.Cursor``, which
+    routes each statement to ``client.query()`` or ``client.command()`` automatically
+    by inspecting the SQL keyword after stripping comments — the same logic used by
+    the ``clickhouse-connect`` SQLAlchemy dialect.
+
+    ClickHouse has no multi-statement transactions.  Every statement is effectively
+    auto-committed, so ``commit()`` and ``rollback()`` are intentional no-ops and
+    the ``autocommit`` attribute is always ``True``.  ``DbApiHook.run()`` checks
+    ``conn.autocommit`` via :meth:`ClickHouseHook.get_autocommit` and skips the
+    ``conn.commit()`` call when it is truthy.
+    """
+
+    # Signals to DbApiHook.get_autocommit() that no explicit commit is needed.
+    autocommit: bool = True
+
+    def __init__(self, client: clickhouse_connect.driver.client.Client) -> None:
+        self._client = client
+
+    def cursor(self) -> ClickHouseDbApiCursor:
+        from clickhouse_connect.dbapi.cursor import Cursor
+
+        return Cursor(self._client)
+
+    def close(self) -> None:
+        self._client.close()
+
+    def commit(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+    def rollback(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+
+class ClickHouseHook(DbApiHook):
+    """
+    Interact with ClickHouse via the HTTP interface (``clickhouse-connect``).
+
+    This hook wraps ``clickhouse_connect.get_client()`` behind a thin DB-API 2.0
+    adapter (:class:`ClickHouseConnection`), so all standard
+    :class:`~airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`
+    features work out of the box (templating, ``handler``, ``split_statements``,
+    etc.).
+
+    :param database: Optional database name.  Overrides the ``schema`` field of the
+        Airflow connection.  Useful when one connection points to a ClickHouse cluster
+        and individual tasks need to target different databases.
+    :param session_settings: Optional dict of ClickHouse session-level settings
+        (e.g. ``{"max_execution_time": 60, "max_threads": 4}``).  Values supplied
+        here are **merged on top of** any ``session_settings`` dict already present
+        in the connection's ``extra`` JSON field, with the constructor argument
+        taking precedence. For a full list of available session settings visit https://clickhouse.com/docs/operations/settings/settings
+    :param client_kwargs: Optional dict of additional keyword arguments forwarded
+        verbatim to ``clickhouse_connect.get_client()``.  Use this to pass any
+        parameter not otherwise exposed by the hook (e.g. ``http_proxy``,
+        ``pool_mgr_params``).  Values supplied here are **merged on top of** any
+        ``client_kwargs`` dict already present in the connection's ``extra`` JSON
+        field, with the constructor argument taking precedence on conflicting keys.
+        Keys that the hook manages itself (``host``, ``port``, ``username``,
+        ``password``, ``database``, ``secure``, ``verify``, ``client_name``,
+        ``settings``) are silently ignored so that hook-managed values always
+        take precedence.
+
+    .. seealso::
+        - `clickhouse-connect documentation <https://clickhouse.com/docs/en/integrations/python>`_
+        - `clickhouse session settings documentation <https://clickhouse.com/docs/operations/settings/settings>`_
+        - :ref:`howto/connection:clickhouse`
+
+    """
+
+    conn_name_attr = "clickhouse_conn_id"
+    default_conn_name = "clickhouse_default"
+    conn_type = "clickhouse"
+    hook_name = "ClickHouse"
+
+    # ClickHouse has no multi-statement transactions; every statement is auto-committed.
+    supports_autocommit = True
+    _test_connection_sql = "SELECT 1"
+
+    def set_autocommit(self, conn: ClickHouseConnection, autocommit: bool) -> None:
+        """No-op: ClickHouse has no transaction support."""
+
+    def get_autocommit(self, conn: ClickHouseConnection) -> bool:
+        """Return ``True``: ClickHouse auto-commits every statement."""
+        return True
+
+    def __init__(
+        self,
+        *args,
+        database: str | None = None,
+        session_settings: dict[str, Any] | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.database = database
+        self.session_settings: dict[str, Any] = session_settings or {}
+        self.client_kwargs: dict[str, Any] = client_kwargs or {}
+
+    @classmethod
+    def get_connection_form_widgets(cls) -> dict[str, Any]:
+        """Return extra connection form widgets shown in the Airflow UI."""
+        from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+        from flask_babel import lazy_gettext
+        from wtforms import BooleanField, IntegerField, StringField
+
+        return {
+            "secure": BooleanField(
+                label=lazy_gettext("Use TLS (HTTPS)"),
+                description="Enable TLS/HTTPS for the connection (default: False).",
+            ),
+            "verify": BooleanField(
+                label=lazy_gettext("Verify SSL Certificate"),
+                description="Verify the server SSL certificate when TLS is enabled (default: True).",
+            ),
+            "connect_timeout": IntegerField(
+                lazy_gettext("Connection Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "send_receive_timeout": IntegerField(
+                lazy_gettext("Query Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "compress": BooleanField(
+                label=lazy_gettext("Enable LZ4 Compression"),
+                description="Compress query results with LZ4 (default: True).",
+            ),
+            "client_name": StringField(
+                lazy_gettext("Client Name (label)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Optional label appended to the Airflow version identifier in the "
+                    "ClickHouse HTTP User-Agent and system.query_log client_name field. "
+                    "The hook always prepends apache-airflow/<version> "
+                    "apache-airflow-providers-clickhouse/<version>."
+                ),
+            ),
+            "session_settings": StringField(
+                lazy_gettext("Session Settings (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "ClickHouse session-level settings as a JSON object, "
+                    'e.g. {"max_execution_time": 60, "max_threads": 4}.'
+                ),
+            ),
+            "client_kwargs": StringField(
+                lazy_gettext("Client kwargs (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Additional keyword arguments forwarded to clickhouse_connect.get_client() "
+                    'as a JSON object, e.g. {"http_proxy": "http://proxy:8080"}. '
+                    "Hook-managed keys (host, port, username, password, database, secure, "
+                    "verify, client_name, settings) are ignored."
+                ),
+            ),
+        }
+
+    @classmethod
+    def get_ui_field_behaviour(cls) -> dict[str, Any]:
+        """Return custom field behaviour for the connection form."""
+        return {
+            "hidden_fields": [],
+            "relabeling": {
+                "schema": "Database",
+            },
+            "placeholders": {
+                "host": "localhost",
+                "port": "8123",
+                "login": "default",
+                "schema": "default",
+                "extra": json.dumps(
+                    {
+                        "secure": False,
+                        "verify": True,
+                        "connect_timeout": 10,
+                        "send_receive_timeout": 300,
+                        "compress": True,
+                        "client_name": "airflow",
+                        "session_settings": {
+                            "max_execution_time": 300,
+                            "max_memory_usage": 10000000000,
+                        },
+                    },
+                    indent=1,
+                ),
+                "secure": "false",
+                "verify": "true",
+                "connect_timeout": "10",
+                "send_receive_timeout": "300",
+                "compress": "true",
+                "client_name": "airflow",
+                "session_settings": '{"max_execution_time": 300}',
+                "client_kwargs": '{"http_proxy": "http://proxy:8080"}',
+            },
+        }
+
+    def _get_client_kwargs(self) -> dict[str, Any]:
+        """
+        Build the keyword-argument dict passed to ``clickhouse_connect.get_client()``.
+
+        Construction order (last write wins):
+
+        1. ``client_kwargs`` from the connection ``extra`` JSON — connection-level
+           passthrough kwargs.
+        2. ``client_kwargs`` constructor argument — task-level overrides merged on top.
+        3. Hook-managed keys (``host``, ``port``, ``username``, ``password``,
+           ``database``, ``secure``, ``verify``, ``client_name``, ``settings``) —
+           always override anything in ``client_kwargs``.
+
+        Optional tuning parameters (``connect_timeout``, ``send_receive_timeout``,
+        ``compress``) are forwarded only when explicitly set in the ``extra`` JSON
+        field so that the driver's own defaults apply otherwise.
+
+        ``client_name`` is always set to a string that identifies the Airflow
+        installation (see :func:`_build_client_name`).  Any ``client_name`` value in
+        ``extra`` is treated as a human-readable label and appended as a comment so
+        it appears alongside the Airflow version info in ``system.query_log``.
+
+        ``session_settings`` from ``extra`` and from the constructor ``session_settings``
+        argument are **merged**, with the constructor argument taking precedence on
+        conflicting keys.
+        """
+        conn = self.get_connection(self.get_conn_id())
+        extra: dict[str, Any] = conn.extra_dejson
+
+        # Merge client_kwargs: extra values are the base, constructor values override.
+        raw_client_kwargs = extra.get("client_kwargs")
+        if isinstance(raw_client_kwargs, str):
+            raw_client_kwargs = json.loads(raw_client_kwargs)
+        merged_client_kwargs: dict[str, Any] = {**(raw_client_kwargs or {}), **self.client_kwargs}
+
+        # Strip hook-managed keys so they cannot overwrite connection credentials
+        # or versioning info.  Log dropped keys at DEBUG to aid troubleshooting.
+        dropped = _HOOK_MANAGED_KWARGS.intersection(merged_client_kwargs)
+        if dropped:
+            self.log.debug(
+                "Ignoring hook-managed keys in client_kwargs: %s",
+                sorted(dropped),
+            )
+        kwargs: dict[str, Any] = {
+            k: v for k, v in merged_client_kwargs.items() if k not in _HOOK_MANAGED_KWARGS
+        }
+
+        # Hook-managed connection parameters always take precedence.
+        kwargs.update(
+            {
+                "host": conn.host or "localhost",
+                "port": int(conn.port) if conn.port else 8123,
+                "username": conn.login or "default",
+                "password": conn.password or "",
+                "database": self.database or conn.schema or "default",
+                "secure": bool(extra.get("secure", False)),
+                "verify": bool(extra.get("verify", True)),
+            }
+        )
+
+        for key in _OPTIONAL_CLIENT_KWARGS:
+            if key in extra:
+                kwargs[key] = extra[key]
+
+        # Always embed Airflow + provider version; user's 'client_name' extra is appended to the User-Agent header.
+        kwargs["client_name"] = _build_client_name(extra.get("client_name"))
+
+        # Merge session_settings: extra values are the base, constructor values override.
+        raw = extra.get("session_settings")
+        if isinstance(raw, str):
+            raw = json.loads(raw)
+        merged: dict[str, Any] = {**(raw or {}), **self.session_settings}
+        if merged:
+            kwargs["settings"] = merged
+
+        return kwargs
+
+    def get_conn(self) -> ClickHouseConnection:
+        """Return a DB-API 2.0 compatible connection backed by ``clickhouse_connect``."""
+        import clickhouse_connect
+
+        client = clickhouse_connect.get_client(**self._get_client_kwargs())
+        return ClickHouseConnection(client)
+
+    def get_client(self) -> clickhouse_connect.driver.client.Client:
+        """
+        Return the raw ``clickhouse_connect`` Client for ClickHouse-specific operations.
+
+        Use this for bulk inserts, streaming queries, or any operation that
+        benefits from the native ``clickhouse_connect`` API rather than DB-API
+        cursors.  The caller is responsible for closing the client.
+        """
+        import clickhouse_connect
+
+        return clickhouse_connect.get_client(**self._get_client_kwargs())
+
+    def bulk_insert_rows(
+        self,
+        table: str,
+        rows: list[tuple],
+        column_names: list[str],
+        commit_every: int = 5000,
+    ) -> None:
+        """
+        Insert rows into a ClickHouse table using the native columnar insert.
+
+        Uses ``clickhouse_connect``'s optimized insert path, which is
+        significantly faster than row-by-row cursor inserts for large datasets.
+
+        :param table: Target table name.
+        :param rows: List of row tuples to insert.
+        :param column_names: Column names matching each position in the row tuples.
+        :param commit_every: Batch size (rows per insert call). Defaults to 5000.
+        """
+        if not rows:
+            self.log.warning(
+                "bulk_insert_rows called with an empty rows list — nothing was inserted into %s.", table
+            )
+            return
+
+        client = self.get_client()
+        try:
+            for i in range(0, len(rows), commit_every):
+                batch = rows[i : i + commit_every]
+                client.insert(table, batch, column_names=column_names)

Review Comment:
   `client.insert()` already streams data to ClickHouse in adaptive ~2MB blocks as a single `Transfer-Encoding: chunked` HTTP request. Batching at the Python layer here doesn't help and could actually even hurt because each `client.insert()` call without a reusable `context` issues a `DESCRIBE TABLE` to resolve column types even when `column_names` is supplied `column_names` is only used to filter/order the describe result.
   
   I'd recommend either:
   1. Just call `client.insert(table, rows, column_names=column_names)` once and let the driver handle block-level streaming internally.
   2. If batching is genuinely needed (e.g. memory pressure on extremely large inputs), build the context once and reuse it like:
     ```python
     ctx = client.create_insert_context(table, column_names=column_names)
     for i in range(0, len(rows), commit_every):
         client.insert(data=rows[i:i+commit_every], context=ctx)
     ````
   
   Side note, I think `commit_every` is a misnomer as inserts are _not_ transactional. `batch_size` might be a more appropriate term.



##########
providers/clickhouse/tests/unit/clickhouse/hooks/test_clickhouse.py:
##########
@@ -0,0 +1,1344 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import json
+import re
+from unittest.mock import ANY, MagicMock, patch
+
+import pytest
+
+from airflow.models import Connection
+from airflow.providers.clickhouse.hooks.clickhouse import (
+    ClickHouseConnection,
+    ClickHouseHook,
+)
+
+# ---------------------------------------------------------------------------
+# Fixtures / shared connection definitions
+# ---------------------------------------------------------------------------
+
+BASE_CONN = Connection(
+    conn_id="clickhouse_test",
+    conn_type="clickhouse",
+    host="clickhouse-host",
+    port=8123,
+    login="user",
+    password="secret",
+    schema="analytics",
+)
+
+BASE_CONN_EXTRA = Connection(
+    conn_id="clickhouse_test_extra",
+    conn_type="clickhouse",
+    host="secure-host",
+    port=8443,
+    login="admin",
+    password="topsecret",
+    schema="prod",
+    extra=json.dumps(
+        {
+            "secure": True,
+            "verify": False,
+            "connect_timeout": 30,
+            "send_receive_timeout": 600,
+            "compress": False,
+            "client_name": "my-airflow",
+        }
+    ),
+)
+
+MINIMAL_CONN = Connection(
+    conn_id="clickhouse_minimal",
+    conn_type="clickhouse",
+)
+
+CONN_WITH_CLIENT_KWARGS = Connection(
+    conn_id="clickhouse_client_kwargs",
+    conn_type="clickhouse",
+    host="ch-host",
+    port=8123,
+    login="user",
+    password="pass",
+    schema="db",
+    extra=json.dumps(
+        {
+            "client_kwargs": {
+                "http_proxy": "http://proxy:8080",
+                "pool_mgr_params": {"num_pools": 4},
+            }
+        }
+    ),
+)
+
+CONN_WITH_CLIENT_KWARGS_STR = Connection(
+    conn_id="clickhouse_client_kwargs_str",
+    conn_type="clickhouse",
+    host="ch-host",
+    port=8123,
+    login="user",
+    password="pass",
+    schema="db",
+    extra=json.dumps(
+        {
+            # client_kwargs stored as a JSON string (as the UI text field stores it)
+            "client_kwargs": '{"http_proxy": "http://proxy:8080"}'
+        }
+    ),
+)
+
+CONN_WITH_SESSION_SETTINGS = Connection(
+    conn_id="clickhouse_session",
+    conn_type="clickhouse",
+    host="ch-host",
+    port=8123,
+    login="user",
+    password="pass",
+    schema="db",
+    extra=json.dumps(
+        {
+            "session_settings": {
+                "max_execution_time": 120,
+                "max_threads": 4,
+            }
+        }
+    ),
+)
+
+CONN_WITH_SESSION_SETTINGS_STR = Connection(
+    conn_id="clickhouse_session_str",
+    conn_type="clickhouse",
+    host="ch-host",
+    port=8123,
+    login="user",
+    password="pass",
+    schema="db",
+    extra=json.dumps(
+        {
+            # session_settings stored as a JSON string (as the UI text field stores it)
+            "session_settings": '{"max_execution_time": 60}'
+        }
+    ),
+)
+
+
+# ---------------------------------------------------------------------------
+# Tests: get_conn
+# ---------------------------------------------------------------------------
+
+
+class TestClickHouseHookGetConn:
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    @patch("clickhouse_connect.get_client")
+    def test_get_conn_basic(self, mock_get_client, mock_get_connection):
+        mock_get_connection.return_value = BASE_CONN
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_test")
+        conn = hook.get_conn()
+
+        mock_get_client.assert_called_once_with(
+            host="clickhouse-host",
+            port=8123,
+            username="user",
+            password="secret",
+            database="analytics",
+            secure=False,
+            verify=True,
+            client_name=ANY,  # always set; format verified in TestClickHouseHookClientName
+        )
+        assert isinstance(conn, ClickHouseConnection)
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    @patch("clickhouse_connect.get_client")
+    def test_get_conn_with_extra_params(self, mock_get_client, mock_get_connection):
+        mock_get_connection.return_value = BASE_CONN_EXTRA
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_test_extra")
+        hook.get_conn()
+
+        mock_get_client.assert_called_once_with(
+            host="secure-host",
+            port=8443,
+            username="admin",
+            password="topsecret",
+            database="prod",
+            secure=True,
+            verify=False,
+            connect_timeout=30,
+            send_receive_timeout=600,
+            compress=False,
+            client_name=ANY,  # contains Airflow version + "(my-airflow)" comment
+        )
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    @patch("clickhouse_connect.get_client")
+    def test_get_conn_minimal_defaults(self, mock_get_client, mock_get_connection):
+        """All connection fields absent → sensible defaults applied."""
+        mock_get_connection.return_value = MINIMAL_CONN
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_minimal")
+        hook.get_conn()
+
+        mock_get_client.assert_called_once_with(
+            host="localhost",
+            port=8123,
+            username="default",
+            password="",
+            database="default",
+            secure=False,
+            verify=True,
+            client_name=ANY,  # always set; format verified in TestClickHouseHookClientName
+        )
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    @patch("clickhouse_connect.get_client")
+    def test_get_conn_database_overrides_connection_schema(self, mock_get_client, mock_get_connection):
+        """database constructor arg must override the schema field from the connection."""
+        mock_get_connection.return_value = BASE_CONN  # schema="analytics"
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_test", database="overridden_db")
+        hook.get_conn()
+
+        call_kwargs = mock_get_client.call_args.kwargs
+        assert call_kwargs["database"] == "overridden_db"
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    @patch("clickhouse_connect.get_client")
+    def test_get_conn_no_database_falls_back_to_connection_schema(self, mock_get_client, mock_get_connection):
+        """Without a database constructor arg the connection schema is used."""
+        mock_get_connection.return_value = BASE_CONN  # schema="analytics"
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_test")
+        hook.get_conn()
+
+        call_kwargs = mock_get_client.call_args.kwargs
+        assert call_kwargs["database"] == "analytics"
+
+
+# ---------------------------------------------------------------------------
+# Tests: get_uri
+# ---------------------------------------------------------------------------
+
+
+class TestClickHouseHookGetUri:
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    def test_get_uri_with_password(self, mock_get_connection):
+        mock_get_connection.return_value = BASE_CONN
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_test")
+        uri = hook.get_uri()
+
+        assert uri == "clickhousedb://user:secret@clickhouse-host:8123/analytics"
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    def test_get_uri_without_password(self, mock_get_connection):
+        conn = Connection(
+            conn_id="clickhouse_nopass",
+            conn_type="clickhouse",
+            host="myhost",
+            port=8123,
+            login="reader",
+            schema="logs",
+        )
+        mock_get_connection.return_value = conn
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_nopass")
+        uri = hook.get_uri()
+
+        assert uri == "clickhousedb://reader@myhost:8123/logs"
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    def test_get_uri_password_url_encoded(self, mock_get_connection):
+        """Special characters in the password must be percent-encoded."""
+        conn = Connection(
+            conn_id="clickhouse_special",
+            conn_type="clickhouse",
+            host="host",
+            port=8123,
+            login="user",
+            password="p@ss/w0rd!",
+            schema="db",
+        )
+        mock_get_connection.return_value = conn
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_special")
+        uri = hook.get_uri()
+
+        assert "@" not in uri.split("://")[1].split("@")[0]  # password part is encoded
+        assert "p%40ss" in uri or "p%40" in uri  # @ is encoded
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    def test_get_uri_defaults(self, mock_get_connection):
+        mock_get_connection.return_value = MINIMAL_CONN
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_minimal")
+        uri = hook.get_uri()
+
+        assert uri == "clickhousedb://default@localhost:8123/default"
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    def test_get_uri_database_overrides_connection_schema(self, mock_get_connection):
+        """database constructor arg must appear in the URI instead of the connection schema."""
+        mock_get_connection.return_value = BASE_CONN  # schema="analytics"
+        hook = ClickHouseHook(clickhouse_conn_id="clickhouse_test", database="overridden_db")
+        uri = hook.get_uri()
+
+        assert uri.endswith("/overridden_db")
+        assert "analytics" not in uri
+
+    @patch("airflow.providers.clickhouse.hooks.clickhouse.ClickHouseHook.get_connection")
+    def test_get_uri_secure_uses_clickhousedbs_scheme(self, mock_get_connection):
+        """secure=True in extra must produce the clickhousedbs:// (HTTPS) scheme."""
+        conn = Connection(
+            conn_id="ch_secure",
+            conn_type="clickhouse",
+            host="secure-host",
+            port=8443,
+            login="user",
+            password="pass",
+            schema="db",
+            extra=json.dumps({"secure": True}),
+        )
+        mock_get_connection.return_value = conn
+        uri = ClickHouseHook(clickhouse_conn_id="ch_secure").get_uri()
+        assert uri == "clickhousedbs://user:pass@secure-host:8443/db"

Review Comment:
   This test can be deleted/reconfigured in light of the other comment explaining how `clickhousedbs` isn't a valid scheme.



##########
providers/clickhouse/src/airflow/providers/clickhouse/hooks/clickhouse.py:
##########
@@ -0,0 +1,434 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any
+from urllib.parse import quote_plus
+
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+    import clickhouse_connect.driver.client
+    from clickhouse_connect.dbapi.cursor import Cursor as ClickHouseDbApiCursor
+
+# Optional scalar parameters forwarded verbatim to clickhouse_connect.get_client()
+# Note: client_name is handled separately by _build_client_name() so Airflow version
+# info is always embedded in the HTTP User-Agent / system.query_log entry.
+_OPTIONAL_CLIENT_KWARGS = (
+    "connect_timeout",
+    "send_receive_timeout",
+    "compress",
+)
+
+# kwargs that the hook always manages itself.  User-supplied client_kwargs that
+# overlap with these keys are dropped and logged at DEBUG so the hook's own values always win.
+_HOOK_MANAGED_KWARGS: frozenset[str] = frozenset(
+    {
+        "host",
+        "port",
+        "username",
+        "password",
+        "database",
+        "secure",
+        "verify",
+        "client_name",
+        "settings",
+    }
+)
+
+
+def _build_client_name(custom: str | None = None) -> str:
+    """
+    Return the ``client_name`` string passed to ``clickhouse_connect.get_client()``.
+
+    The value is prepended to the HTTP ``User-Agent`` header that ClickHouse records
+    in ``system.query_log``, so every query is traceable back to its Airflow source.
+
+    Format (no custom label)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version>
+
+    Format (with the ``client_name`` extra field set to ``"my-pipeline"``)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version> (my-pipeline)
+
+    The ``clickhouse_connect`` library appends its own token and OS information, so
+    the full ``User-Agent`` looks like::
+
+        apache-airflow/X.Y.Z apache-airflow-providers-clickhouse/X.Y.Z
+        clickhouse-connect/X.Y.Z (lv:py/X.Y.Z; mode:sync; os:linux)
+    """
+    import airflow
+    from airflow.providers.clickhouse import __version__ as provider_version
+
+    name = f"apache-airflow/{airflow.__version__} apache-airflow-providers-clickhouse/{provider_version}"
+    if custom:
+        name = f"{name} ({custom.strip()})"
+    return name
+
+
+class ClickHouseConnection:
+    """
+    Minimal DB-API 2.0 connection adapter wrapping a ``clickhouse_connect`` Client.
+
+    SQL execution is delegated to ``clickhouse_connect.dbapi.cursor.Cursor``, which
+    routes each statement to ``client.query()`` or ``client.command()`` automatically
+    by inspecting the SQL keyword after stripping comments — the same logic used by
+    the ``clickhouse-connect`` SQLAlchemy dialect.
+
+    ClickHouse has no multi-statement transactions.  Every statement is effectively
+    auto-committed, so ``commit()`` and ``rollback()`` are intentional no-ops and
+    the ``autocommit`` attribute is always ``True``.  ``DbApiHook.run()`` checks
+    ``conn.autocommit`` via :meth:`ClickHouseHook.get_autocommit` and skips the
+    ``conn.commit()`` call when it is truthy.
+    """
+
+    # Signals to DbApiHook.get_autocommit() that no explicit commit is needed.
+    autocommit: bool = True
+
+    def __init__(self, client: clickhouse_connect.driver.client.Client) -> None:
+        self._client = client
+
+    def cursor(self) -> ClickHouseDbApiCursor:
+        from clickhouse_connect.dbapi.cursor import Cursor
+
+        return Cursor(self._client)
+
+    def close(self) -> None:
+        self._client.close()
+
+    def commit(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+    def rollback(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+
+class ClickHouseHook(DbApiHook):
+    """
+    Interact with ClickHouse via the HTTP interface (``clickhouse-connect``).
+
+    This hook wraps ``clickhouse_connect.get_client()`` behind a thin DB-API 2.0
+    adapter (:class:`ClickHouseConnection`), so all standard
+    :class:`~airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`
+    features work out of the box (templating, ``handler``, ``split_statements``,
+    etc.).
+
+    :param database: Optional database name.  Overrides the ``schema`` field of the
+        Airflow connection.  Useful when one connection points to a ClickHouse cluster
+        and individual tasks need to target different databases.
+    :param session_settings: Optional dict of ClickHouse session-level settings
+        (e.g. ``{"max_execution_time": 60, "max_threads": 4}``).  Values supplied
+        here are **merged on top of** any ``session_settings`` dict already present
+        in the connection's ``extra`` JSON field, with the constructor argument
+        taking precedence. For a full list of available session settings visit https://clickhouse.com/docs/operations/settings/settings
+    :param client_kwargs: Optional dict of additional keyword arguments forwarded
+        verbatim to ``clickhouse_connect.get_client()``.  Use this to pass any
+        parameter not otherwise exposed by the hook (e.g. ``http_proxy``,
+        ``pool_mgr_params``).  Values supplied here are **merged on top of** any
+        ``client_kwargs`` dict already present in the connection's ``extra`` JSON
+        field, with the constructor argument taking precedence on conflicting keys.
+        Keys that the hook manages itself (``host``, ``port``, ``username``,
+        ``password``, ``database``, ``secure``, ``verify``, ``client_name``,
+        ``settings``) are silently ignored so that hook-managed values always
+        take precedence.
+
+    .. seealso::
+        - `clickhouse-connect documentation <https://clickhouse.com/docs/en/integrations/python>`_
+        - `clickhouse session settings documentation <https://clickhouse.com/docs/operations/settings/settings>`_
+        - :ref:`howto/connection:clickhouse`
+
+    """
+
+    conn_name_attr = "clickhouse_conn_id"
+    default_conn_name = "clickhouse_default"
+    conn_type = "clickhouse"
+    hook_name = "ClickHouse"
+
+    # ClickHouse has no multi-statement transactions; every statement is auto-committed.
+    supports_autocommit = True
+    _test_connection_sql = "SELECT 1"
+
+    def set_autocommit(self, conn: ClickHouseConnection, autocommit: bool) -> None:
+        """No-op: ClickHouse has no transaction support."""
+
+    def get_autocommit(self, conn: ClickHouseConnection) -> bool:
+        """Return ``True``: ClickHouse auto-commits every statement."""
+        return True
+
+    def __init__(
+        self,
+        *args,
+        database: str | None = None,
+        session_settings: dict[str, Any] | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.database = database
+        self.session_settings: dict[str, Any] = session_settings or {}
+        self.client_kwargs: dict[str, Any] = client_kwargs or {}
+
+    @classmethod
+    def get_connection_form_widgets(cls) -> dict[str, Any]:
+        """Return extra connection form widgets shown in the Airflow UI."""
+        from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+        from flask_babel import lazy_gettext
+        from wtforms import BooleanField, IntegerField, StringField
+
+        return {
+            "secure": BooleanField(
+                label=lazy_gettext("Use TLS (HTTPS)"),
+                description="Enable TLS/HTTPS for the connection (default: False).",
+            ),
+            "verify": BooleanField(
+                label=lazy_gettext("Verify SSL Certificate"),
+                description="Verify the server SSL certificate when TLS is enabled (default: True).",
+            ),
+            "connect_timeout": IntegerField(
+                lazy_gettext("Connection Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "send_receive_timeout": IntegerField(
+                lazy_gettext("Query Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "compress": BooleanField(
+                label=lazy_gettext("Enable LZ4 Compression"),
+                description="Compress query results with LZ4 (default: True).",
+            ),
+            "client_name": StringField(
+                lazy_gettext("Client Name (label)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Optional label appended to the Airflow version identifier in the "
+                    "ClickHouse HTTP User-Agent and system.query_log client_name field. "
+                    "The hook always prepends apache-airflow/<version> "
+                    "apache-airflow-providers-clickhouse/<version>."
+                ),
+            ),
+            "session_settings": StringField(
+                lazy_gettext("Session Settings (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "ClickHouse session-level settings as a JSON object, "
+                    'e.g. {"max_execution_time": 60, "max_threads": 4}.'
+                ),
+            ),
+            "client_kwargs": StringField(
+                lazy_gettext("Client kwargs (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Additional keyword arguments forwarded to clickhouse_connect.get_client() "
+                    'as a JSON object, e.g. {"http_proxy": "http://proxy:8080"}. '
+                    "Hook-managed keys (host, port, username, password, database, secure, "
+                    "verify, client_name, settings) are ignored."
+                ),
+            ),
+        }
+
+    @classmethod
+    def get_ui_field_behaviour(cls) -> dict[str, Any]:
+        """Return custom field behaviour for the connection form."""
+        return {
+            "hidden_fields": [],
+            "relabeling": {
+                "schema": "Database",
+            },
+            "placeholders": {
+                "host": "localhost",
+                "port": "8123",
+                "login": "default",
+                "schema": "default",
+                "extra": json.dumps(
+                    {
+                        "secure": False,
+                        "verify": True,
+                        "connect_timeout": 10,
+                        "send_receive_timeout": 300,
+                        "compress": True,
+                        "client_name": "airflow",
+                        "session_settings": {
+                            "max_execution_time": 300,
+                            "max_memory_usage": 10000000000,
+                        },
+                    },
+                    indent=1,
+                ),
+                "secure": "false",
+                "verify": "true",
+                "connect_timeout": "10",
+                "send_receive_timeout": "300",
+                "compress": "true",
+                "client_name": "airflow",
+                "session_settings": '{"max_execution_time": 300}',
+                "client_kwargs": '{"http_proxy": "http://proxy:8080"}',
+            },
+        }
+
+    def _get_client_kwargs(self) -> dict[str, Any]:
+        """
+        Build the keyword-argument dict passed to ``clickhouse_connect.get_client()``.
+
+        Construction order (last write wins):
+
+        1. ``client_kwargs`` from the connection ``extra`` JSON — connection-level
+           passthrough kwargs.
+        2. ``client_kwargs`` constructor argument — task-level overrides merged on top.
+        3. Hook-managed keys (``host``, ``port``, ``username``, ``password``,
+           ``database``, ``secure``, ``verify``, ``client_name``, ``settings``) —
+           always override anything in ``client_kwargs``.
+
+        Optional tuning parameters (``connect_timeout``, ``send_receive_timeout``,
+        ``compress``) are forwarded only when explicitly set in the ``extra`` JSON
+        field so that the driver's own defaults apply otherwise.
+
+        ``client_name`` is always set to a string that identifies the Airflow
+        installation (see :func:`_build_client_name`).  Any ``client_name`` value in
+        ``extra`` is treated as a human-readable label and appended as a comment so
+        it appears alongside the Airflow version info in ``system.query_log``.
+
+        ``session_settings`` from ``extra`` and from the constructor ``session_settings``
+        argument are **merged**, with the constructor argument taking precedence on
+        conflicting keys.
+        """
+        conn = self.get_connection(self.get_conn_id())
+        extra: dict[str, Any] = conn.extra_dejson
+
+        # Merge client_kwargs: extra values are the base, constructor values override.
+        raw_client_kwargs = extra.get("client_kwargs")
+        if isinstance(raw_client_kwargs, str):
+            raw_client_kwargs = json.loads(raw_client_kwargs)
+        merged_client_kwargs: dict[str, Any] = {**(raw_client_kwargs or {}), **self.client_kwargs}
+
+        # Strip hook-managed keys so they cannot overwrite connection credentials
+        # or versioning info.  Log dropped keys at DEBUG to aid troubleshooting.
+        dropped = _HOOK_MANAGED_KWARGS.intersection(merged_client_kwargs)
+        if dropped:
+            self.log.debug(
+                "Ignoring hook-managed keys in client_kwargs: %s",
+                sorted(dropped),
+            )
+        kwargs: dict[str, Any] = {
+            k: v for k, v in merged_client_kwargs.items() if k not in _HOOK_MANAGED_KWARGS
+        }
+
+        # Hook-managed connection parameters always take precedence.
+        kwargs.update(
+            {
+                "host": conn.host or "localhost",
+                "port": int(conn.port) if conn.port else 8123,
+                "username": conn.login or "default",
+                "password": conn.password or "",
+                "database": self.database or conn.schema or "default",
+                "secure": bool(extra.get("secure", False)),
+                "verify": bool(extra.get("verify", True)),
+            }
+        )
+
+        for key in _OPTIONAL_CLIENT_KWARGS:
+            if key in extra:
+                kwargs[key] = extra[key]
+
+        # Always embed Airflow + provider version; user's 'client_name' extra is appended to the User-Agent header.
+        kwargs["client_name"] = _build_client_name(extra.get("client_name"))
+
+        # Merge session_settings: extra values are the base, constructor values override.
+        raw = extra.get("session_settings")
+        if isinstance(raw, str):
+            raw = json.loads(raw)
+        merged: dict[str, Any] = {**(raw or {}), **self.session_settings}
+        if merged:
+            kwargs["settings"] = merged
+
+        return kwargs
+
+    def get_conn(self) -> ClickHouseConnection:
+        """Return a DB-API 2.0 compatible connection backed by ``clickhouse_connect``."""
+        import clickhouse_connect
+
+        client = clickhouse_connect.get_client(**self._get_client_kwargs())
+        return ClickHouseConnection(client)
+
+    def get_client(self) -> clickhouse_connect.driver.client.Client:
+        """
+        Return the raw ``clickhouse_connect`` Client for ClickHouse-specific operations.
+
+        Use this for bulk inserts, streaming queries, or any operation that
+        benefits from the native ``clickhouse_connect`` API rather than DB-API
+        cursors.  The caller is responsible for closing the client.
+        """
+        import clickhouse_connect
+
+        return clickhouse_connect.get_client(**self._get_client_kwargs())
+
+    def bulk_insert_rows(
+        self,
+        table: str,
+        rows: list[tuple],
+        column_names: list[str],
+        commit_every: int = 5000,
+    ) -> None:
+        """
+        Insert rows into a ClickHouse table using the native columnar insert.
+
+        Uses ``clickhouse_connect``'s optimized insert path, which is
+        significantly faster than row-by-row cursor inserts for large datasets.
+
+        :param table: Target table name.
+        :param rows: List of row tuples to insert.
+        :param column_names: Column names matching each position in the row tuples.
+        :param commit_every: Batch size (rows per insert call). Defaults to 5000.
+        """
+        if not rows:
+            self.log.warning(
+                "bulk_insert_rows called with an empty rows list — nothing was inserted into %s.", table
+            )
+            return
+
+        client = self.get_client()
+        try:
+            for i in range(0, len(rows), commit_every):
+                batch = rows[i : i + commit_every]
+                client.insert(table, batch, column_names=column_names)
+            self.log.info("Inserted %d rows into %s", len(rows), table)
+        finally:
+            client.close()
+
+    def get_uri(self) -> str:
+        """
+        Return a SQLAlchemy-compatible URI for ``get_sqlalchemy_engine()``.
+
+        Uses the ``clickhousedb://`` scheme (HTTP) or ``clickhousedbs://`` scheme
+        (HTTPS/TLS) provided by ``clickhouse-connect``, chosen based on the
+        ``secure`` field in the connection's ``extra`` JSON.
+        """
+        conn = self.get_connection(self.get_conn_id())
+        extra: dict[str, Any] = conn.extra_dejson
+        username = conn.login or "default"
+        password = quote_plus(conn.password) if conn.password else ""
+        host = conn.host or "localhost"
+        port = int(conn.port) if conn.port else 8123
+        database = self.database or conn.schema or "default"
+        scheme = "clickhousedbs" if bool(extra.get("secure", False)) else "clickhousedb"

Review Comment:
   `clickhousedbs` isn't a registered scheme. clickhouse-connect only registers `clickhousedb` (and `clickhousedb+connect`) as a SQLAlchemy dialect. The way you'll want to use with a TLS connection is a single scheme with secure as a query parameter, which the dbapi `Connection.__init__` forwards via `generic_args` -> `create_client`. e.g. `clickhousedb://user:pw@host:port/db?secure=true&verify=true`.
   
   It's probably worth wiring the other tuning params like `connect_timeout`, `send_receive_timeout`, `compress`, etc. through the query string the same way, otherwise SQLAlchemy-path users silently lose the settings change ability that DB-API-path users get.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "BentsiLeviav (via GitHub)" <gi...@apache.org>.
BentsiLeviav commented on code in PR #67080:
URL: https://github.com/apache/airflow/pull/67080#discussion_r3275740156


##########
providers/clickhouse/src/airflow/providers/clickhouse/hooks/clickhouse.py:
##########
@@ -0,0 +1,434 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any
+from urllib.parse import quote_plus
+
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+    import clickhouse_connect.driver.client
+    from clickhouse_connect.dbapi.cursor import Cursor as ClickHouseDbApiCursor
+
+# Optional scalar parameters forwarded verbatim to clickhouse_connect.get_client()
+# Note: client_name is handled separately by _build_client_name() so Airflow version
+# info is always embedded in the HTTP User-Agent / system.query_log entry.
+_OPTIONAL_CLIENT_KWARGS = (
+    "connect_timeout",
+    "send_receive_timeout",
+    "compress",
+)
+
+# kwargs that the hook always manages itself.  User-supplied client_kwargs that
+# overlap with these keys are dropped and logged at DEBUG so the hook's own values always win.
+_HOOK_MANAGED_KWARGS: frozenset[str] = frozenset(
+    {
+        "host",
+        "port",
+        "username",
+        "password",
+        "database",
+        "secure",
+        "verify",
+        "client_name",
+        "settings",
+    }
+)
+
+
+def _build_client_name(custom: str | None = None) -> str:
+    """
+    Return the ``client_name`` string passed to ``clickhouse_connect.get_client()``.
+
+    The value is prepended to the HTTP ``User-Agent`` header that ClickHouse records
+    in ``system.query_log``, so every query is traceable back to its Airflow source.
+
+    Format (no custom label)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version>
+
+    Format (with the ``client_name`` extra field set to ``"my-pipeline"``)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version> (my-pipeline)
+
+    The ``clickhouse_connect`` library appends its own token and OS information, so
+    the full ``User-Agent`` looks like::
+
+        apache-airflow/X.Y.Z apache-airflow-providers-clickhouse/X.Y.Z
+        clickhouse-connect/X.Y.Z (lv:py/X.Y.Z; mode:sync; os:linux)
+    """
+    import airflow
+    from airflow.providers.clickhouse import __version__ as provider_version
+
+    name = f"apache-airflow/{airflow.__version__} apache-airflow-providers-clickhouse/{provider_version}"
+    if custom:
+        name = f"{name} ({custom.strip()})"
+    return name
+
+
+class ClickHouseConnection:
+    """
+    Minimal DB-API 2.0 connection adapter wrapping a ``clickhouse_connect`` Client.
+
+    SQL execution is delegated to ``clickhouse_connect.dbapi.cursor.Cursor``, which
+    routes each statement to ``client.query()`` or ``client.command()`` automatically
+    by inspecting the SQL keyword after stripping comments — the same logic used by
+    the ``clickhouse-connect`` SQLAlchemy dialect.
+
+    ClickHouse has no multi-statement transactions.  Every statement is effectively
+    auto-committed, so ``commit()`` and ``rollback()`` are intentional no-ops and
+    the ``autocommit`` attribute is always ``True``.  ``DbApiHook.run()`` checks
+    ``conn.autocommit`` via :meth:`ClickHouseHook.get_autocommit` and skips the
+    ``conn.commit()`` call when it is truthy.
+    """
+
+    # Signals to DbApiHook.get_autocommit() that no explicit commit is needed.
+    autocommit: bool = True
+
+    def __init__(self, client: clickhouse_connect.driver.client.Client) -> None:
+        self._client = client
+
+    def cursor(self) -> ClickHouseDbApiCursor:
+        from clickhouse_connect.dbapi.cursor import Cursor
+
+        return Cursor(self._client)
+
+    def close(self) -> None:
+        self._client.close()
+
+    def commit(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+    def rollback(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+
+class ClickHouseHook(DbApiHook):
+    """
+    Interact with ClickHouse via the HTTP interface (``clickhouse-connect``).
+
+    This hook wraps ``clickhouse_connect.get_client()`` behind a thin DB-API 2.0
+    adapter (:class:`ClickHouseConnection`), so all standard
+    :class:`~airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`
+    features work out of the box (templating, ``handler``, ``split_statements``,
+    etc.).
+
+    :param database: Optional database name.  Overrides the ``schema`` field of the
+        Airflow connection.  Useful when one connection points to a ClickHouse cluster
+        and individual tasks need to target different databases.
+    :param session_settings: Optional dict of ClickHouse session-level settings
+        (e.g. ``{"max_execution_time": 60, "max_threads": 4}``).  Values supplied
+        here are **merged on top of** any ``session_settings`` dict already present
+        in the connection's ``extra`` JSON field, with the constructor argument
+        taking precedence. For a full list of available session settings visit https://clickhouse.com/docs/operations/settings/settings
+    :param client_kwargs: Optional dict of additional keyword arguments forwarded
+        verbatim to ``clickhouse_connect.get_client()``.  Use this to pass any
+        parameter not otherwise exposed by the hook (e.g. ``http_proxy``,
+        ``pool_mgr_params``).  Values supplied here are **merged on top of** any
+        ``client_kwargs`` dict already present in the connection's ``extra`` JSON
+        field, with the constructor argument taking precedence on conflicting keys.
+        Keys that the hook manages itself (``host``, ``port``, ``username``,
+        ``password``, ``database``, ``secure``, ``verify``, ``client_name``,
+        ``settings``) are silently ignored so that hook-managed values always
+        take precedence.
+
+    .. seealso::
+        - `clickhouse-connect documentation <https://clickhouse.com/docs/en/integrations/python>`_
+        - `clickhouse session settings documentation <https://clickhouse.com/docs/operations/settings/settings>`_
+        - :ref:`howto/connection:clickhouse`
+
+    """
+
+    conn_name_attr = "clickhouse_conn_id"
+    default_conn_name = "clickhouse_default"
+    conn_type = "clickhouse"
+    hook_name = "ClickHouse"
+
+    # ClickHouse has no multi-statement transactions; every statement is auto-committed.
+    supports_autocommit = True
+    _test_connection_sql = "SELECT 1"
+
+    def set_autocommit(self, conn: ClickHouseConnection, autocommit: bool) -> None:
+        """No-op: ClickHouse has no transaction support."""
+
+    def get_autocommit(self, conn: ClickHouseConnection) -> bool:
+        """Return ``True``: ClickHouse auto-commits every statement."""
+        return True
+
+    def __init__(
+        self,
+        *args,
+        database: str | None = None,
+        session_settings: dict[str, Any] | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.database = database
+        self.session_settings: dict[str, Any] = session_settings or {}
+        self.client_kwargs: dict[str, Any] = client_kwargs or {}
+
+    @classmethod
+    def get_connection_form_widgets(cls) -> dict[str, Any]:
+        """Return extra connection form widgets shown in the Airflow UI."""
+        from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+        from flask_babel import lazy_gettext
+        from wtforms import BooleanField, IntegerField, StringField
+
+        return {
+            "secure": BooleanField(
+                label=lazy_gettext("Use TLS (HTTPS)"),
+                description="Enable TLS/HTTPS for the connection (default: False).",
+            ),
+            "verify": BooleanField(
+                label=lazy_gettext("Verify SSL Certificate"),
+                description="Verify the server SSL certificate when TLS is enabled (default: True).",
+            ),
+            "connect_timeout": IntegerField(
+                lazy_gettext("Connection Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "send_receive_timeout": IntegerField(
+                lazy_gettext("Query Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "compress": BooleanField(
+                label=lazy_gettext("Enable LZ4 Compression"),
+                description="Compress query results with LZ4 (default: True).",
+            ),
+            "client_name": StringField(
+                lazy_gettext("Client Name (label)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Optional label appended to the Airflow version identifier in the "
+                    "ClickHouse HTTP User-Agent and system.query_log client_name field. "
+                    "The hook always prepends apache-airflow/<version> "
+                    "apache-airflow-providers-clickhouse/<version>."
+                ),
+            ),
+            "session_settings": StringField(
+                lazy_gettext("Session Settings (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "ClickHouse session-level settings as a JSON object, "
+                    'e.g. {"max_execution_time": 60, "max_threads": 4}.'
+                ),
+            ),
+            "client_kwargs": StringField(
+                lazy_gettext("Client kwargs (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Additional keyword arguments forwarded to clickhouse_connect.get_client() "
+                    'as a JSON object, e.g. {"http_proxy": "http://proxy:8080"}. '
+                    "Hook-managed keys (host, port, username, password, database, secure, "
+                    "verify, client_name, settings) are ignored."
+                ),
+            ),
+        }
+
+    @classmethod
+    def get_ui_field_behaviour(cls) -> dict[str, Any]:
+        """Return custom field behaviour for the connection form."""
+        return {
+            "hidden_fields": [],
+            "relabeling": {
+                "schema": "Database",
+            },
+            "placeholders": {
+                "host": "localhost",
+                "port": "8123",
+                "login": "default",
+                "schema": "default",
+                "extra": json.dumps(
+                    {
+                        "secure": False,
+                        "verify": True,
+                        "connect_timeout": 10,
+                        "send_receive_timeout": 300,
+                        "compress": True,
+                        "client_name": "airflow",
+                        "session_settings": {
+                            "max_execution_time": 300,
+                            "max_memory_usage": 10000000000,
+                        },
+                    },
+                    indent=1,
+                ),
+                "secure": "false",
+                "verify": "true",
+                "connect_timeout": "10",
+                "send_receive_timeout": "300",
+                "compress": "true",
+                "client_name": "airflow",
+                "session_settings": '{"max_execution_time": 300}',
+                "client_kwargs": '{"http_proxy": "http://proxy:8080"}',
+            },
+        }
+
+    def _get_client_kwargs(self) -> dict[str, Any]:
+        """
+        Build the keyword-argument dict passed to ``clickhouse_connect.get_client()``.
+
+        Construction order (last write wins):
+
+        1. ``client_kwargs`` from the connection ``extra`` JSON — connection-level
+           passthrough kwargs.
+        2. ``client_kwargs`` constructor argument — task-level overrides merged on top.
+        3. Hook-managed keys (``host``, ``port``, ``username``, ``password``,
+           ``database``, ``secure``, ``verify``, ``client_name``, ``settings``) —
+           always override anything in ``client_kwargs``.
+
+        Optional tuning parameters (``connect_timeout``, ``send_receive_timeout``,
+        ``compress``) are forwarded only when explicitly set in the ``extra`` JSON
+        field so that the driver's own defaults apply otherwise.
+
+        ``client_name`` is always set to a string that identifies the Airflow
+        installation (see :func:`_build_client_name`).  Any ``client_name`` value in
+        ``extra`` is treated as a human-readable label and appended as a comment so
+        it appears alongside the Airflow version info in ``system.query_log``.
+
+        ``session_settings`` from ``extra`` and from the constructor ``session_settings``
+        argument are **merged**, with the constructor argument taking precedence on
+        conflicting keys.
+        """
+        conn = self.get_connection(self.get_conn_id())
+        extra: dict[str, Any] = conn.extra_dejson
+
+        # Merge client_kwargs: extra values are the base, constructor values override.
+        raw_client_kwargs = extra.get("client_kwargs")
+        if isinstance(raw_client_kwargs, str):
+            raw_client_kwargs = json.loads(raw_client_kwargs)
+        merged_client_kwargs: dict[str, Any] = {**(raw_client_kwargs or {}), **self.client_kwargs}
+
+        # Strip hook-managed keys so they cannot overwrite connection credentials
+        # or versioning info.  Log dropped keys at DEBUG to aid troubleshooting.
+        dropped = _HOOK_MANAGED_KWARGS.intersection(merged_client_kwargs)
+        if dropped:
+            self.log.debug(
+                "Ignoring hook-managed keys in client_kwargs: %s",
+                sorted(dropped),
+            )
+        kwargs: dict[str, Any] = {
+            k: v for k, v in merged_client_kwargs.items() if k not in _HOOK_MANAGED_KWARGS
+        }
+
+        # Hook-managed connection parameters always take precedence.
+        kwargs.update(
+            {
+                "host": conn.host or "localhost",
+                "port": int(conn.port) if conn.port else 8123,
+                "username": conn.login or "default",
+                "password": conn.password or "",
+                "database": self.database or conn.schema or "default",
+                "secure": bool(extra.get("secure", False)),
+                "verify": bool(extra.get("verify", True)),
+            }
+        )
+
+        for key in _OPTIONAL_CLIENT_KWARGS:
+            if key in extra:
+                kwargs[key] = extra[key]
+
+        # Always embed Airflow + provider version; user's 'client_name' extra is appended to the User-Agent header.
+        kwargs["client_name"] = _build_client_name(extra.get("client_name"))
+
+        # Merge session_settings: extra values are the base, constructor values override.
+        raw = extra.get("session_settings")
+        if isinstance(raw, str):
+            raw = json.loads(raw)
+        merged: dict[str, Any] = {**(raw or {}), **self.session_settings}
+        if merged:
+            kwargs["settings"] = merged
+
+        return kwargs
+
+    def get_conn(self) -> ClickHouseConnection:
+        """Return a DB-API 2.0 compatible connection backed by ``clickhouse_connect``."""
+        import clickhouse_connect
+
+        client = clickhouse_connect.get_client(**self._get_client_kwargs())
+        return ClickHouseConnection(client)
+
+    def get_client(self) -> clickhouse_connect.driver.client.Client:
+        """
+        Return the raw ``clickhouse_connect`` Client for ClickHouse-specific operations.
+
+        Use this for bulk inserts, streaming queries, or any operation that
+        benefits from the native ``clickhouse_connect`` API rather than DB-API
+        cursors.  The caller is responsible for closing the client.
+        """
+        import clickhouse_connect
+
+        return clickhouse_connect.get_client(**self._get_client_kwargs())
+
+    def bulk_insert_rows(
+        self,
+        table: str,
+        rows: list[tuple],
+        column_names: list[str],
+        commit_every: int = 5000,
+    ) -> None:
+        """
+        Insert rows into a ClickHouse table using the native columnar insert.
+
+        Uses ``clickhouse_connect``'s optimized insert path, which is
+        significantly faster than row-by-row cursor inserts for large datasets.
+
+        :param table: Target table name.
+        :param rows: List of row tuples to insert.
+        :param column_names: Column names matching each position in the row tuples.
+        :param commit_every: Batch size (rows per insert call). Defaults to 5000.
+        """
+        if not rows:
+            self.log.warning(
+                "bulk_insert_rows called with an empty rows list — nothing was inserted into %s.", table
+            )
+            return
+
+        client = self.get_client()
+        try:
+            for i in range(0, len(rows), commit_every):
+                batch = rows[i : i + commit_every]
+                client.insert(table, batch, column_names=column_names)

Review Comment:
   Thanks for the input.
   Took up your recommendation and did the following:
   - renamed `commit_every` to `batch_size`
   - Change the default value of it to None, so if it is not provided, we will have a single insert.
   - In case it is provided, create the context once before the insertion loop.
   - updated the tests to verify all these 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #67080:
URL: https://github.com/apache/airflow/pull/67080


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "BentsiLeviav (via GitHub)" <gi...@apache.org>.
BentsiLeviav commented on PR #67080:
URL: https://github.com/apache/airflow/pull/67080#issuecomment-4531887261

   @potiuk @eladkal can one of you re-run the tests?
   
   I fixed most of them; there are some failures related to an Amazon provider (doesn't seem to be related to our changes at all)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #67080:
URL: https://github.com/apache/airflow/pull/67080#issuecomment-4609080571

   Doc build is failing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #67080:
URL: https://github.com/apache/airflow/pull/67080#issuecomment-4566489424

   <!-- uv-lock-rebase-notice -->
   `uv.lock` on `main` just moved via [#62790](https://github.com/apache/airflow/pull/62790) ("Add IBM MQ provider"), commit [`23b334f`](https://github.com/apache/airflow/commit/23b334f29eba4f3e3ebfed46540f2ea2d59d1e68) and this PR currently conflicts.
   
   Quickest fix:
   ```bash
   git fetch upstream main && git rebase upstream/main
   rm uv.lock && uv lock
   git add uv.lock && git rebase --continue
   git push --force-with-lease
   ```
   
   _Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future `uv.lock` bumps._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #67080:
URL: https://github.com/apache/airflow/pull/67080#issuecomment-4473810275

   Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our [Contributors' Guide](https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
   Here are some useful points:
   - Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example Dag that shows how users should use it.
   - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
   - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
   - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
   - Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
   Apache Airflow is a community-driven project and together we are making it better 🚀.
   In case of doubts contact the developers at:
   Mailing List: dev@airflow.apache.org
   Slack: https://s.apache.org/airflow-slack
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #67080:
URL: https://github.com/apache/airflow/pull/67080#issuecomment-4529322531

   @BentsiLeviav A few things need addressing before review — see our [Pull Request quality criteria](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-quality-criteria).
   
   - :x: **Static checks, tests, and docs build all fail** — 22 failing checks across `Basic tests / Breeze unit tests`, `CI image checks / Static checks`, `CI image checks / Build documentation (--docs-only)`, and provider distribution test matrices. See the [Checks tab](https://github.com/apache/airflow/pull/67080/checks) for the full list and individual job logs.
   
   Adding a new provider is a substantial change; please get a clean CI run before requesting maintainer review. No rush.
   
   ---
   
   _Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._
   
   ---
   Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "BentsiLeviav (via GitHub)" <gi...@apache.org>.
BentsiLeviav commented on code in PR #67080:
URL: https://github.com/apache/airflow/pull/67080#discussion_r3275608385


##########
providers/clickhouse/src/airflow/providers/clickhouse/hooks/clickhouse.py:
##########
@@ -0,0 +1,434 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any
+from urllib.parse import quote_plus
+
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+
+if TYPE_CHECKING:
+    import clickhouse_connect.driver.client
+    from clickhouse_connect.dbapi.cursor import Cursor as ClickHouseDbApiCursor
+
+# Optional scalar parameters forwarded verbatim to clickhouse_connect.get_client()
+# Note: client_name is handled separately by _build_client_name() so Airflow version
+# info is always embedded in the HTTP User-Agent / system.query_log entry.
+_OPTIONAL_CLIENT_KWARGS = (
+    "connect_timeout",
+    "send_receive_timeout",
+    "compress",
+)
+
+# kwargs that the hook always manages itself.  User-supplied client_kwargs that
+# overlap with these keys are dropped and logged at DEBUG so the hook's own values always win.
+_HOOK_MANAGED_KWARGS: frozenset[str] = frozenset(
+    {
+        "host",
+        "port",
+        "username",
+        "password",
+        "database",
+        "secure",
+        "verify",
+        "client_name",
+        "settings",
+    }
+)
+
+
+def _build_client_name(custom: str | None = None) -> str:
+    """
+    Return the ``client_name`` string passed to ``clickhouse_connect.get_client()``.
+
+    The value is prepended to the HTTP ``User-Agent`` header that ClickHouse records
+    in ``system.query_log``, so every query is traceable back to its Airflow source.
+
+    Format (no custom label)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version>
+
+    Format (with the ``client_name`` extra field set to ``"my-pipeline"``)::
+
+        apache-airflow/<airflow_version> apache-airflow-providers-clickhouse/<provider_version> (my-pipeline)
+
+    The ``clickhouse_connect`` library appends its own token and OS information, so
+    the full ``User-Agent`` looks like::
+
+        apache-airflow/X.Y.Z apache-airflow-providers-clickhouse/X.Y.Z
+        clickhouse-connect/X.Y.Z (lv:py/X.Y.Z; mode:sync; os:linux)
+    """
+    import airflow
+    from airflow.providers.clickhouse import __version__ as provider_version
+
+    name = f"apache-airflow/{airflow.__version__} apache-airflow-providers-clickhouse/{provider_version}"
+    if custom:
+        name = f"{name} ({custom.strip()})"
+    return name
+
+
+class ClickHouseConnection:
+    """
+    Minimal DB-API 2.0 connection adapter wrapping a ``clickhouse_connect`` Client.
+
+    SQL execution is delegated to ``clickhouse_connect.dbapi.cursor.Cursor``, which
+    routes each statement to ``client.query()`` or ``client.command()`` automatically
+    by inspecting the SQL keyword after stripping comments — the same logic used by
+    the ``clickhouse-connect`` SQLAlchemy dialect.
+
+    ClickHouse has no multi-statement transactions.  Every statement is effectively
+    auto-committed, so ``commit()`` and ``rollback()`` are intentional no-ops and
+    the ``autocommit`` attribute is always ``True``.  ``DbApiHook.run()`` checks
+    ``conn.autocommit`` via :meth:`ClickHouseHook.get_autocommit` and skips the
+    ``conn.commit()`` call when it is truthy.
+    """
+
+    # Signals to DbApiHook.get_autocommit() that no explicit commit is needed.
+    autocommit: bool = True
+
+    def __init__(self, client: clickhouse_connect.driver.client.Client) -> None:
+        self._client = client
+
+    def cursor(self) -> ClickHouseDbApiCursor:
+        from clickhouse_connect.dbapi.cursor import Cursor
+
+        return Cursor(self._client)
+
+    def close(self) -> None:
+        self._client.close()
+
+    def commit(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+    def rollback(self) -> None:
+        pass  # ClickHouse has no multi-statement transactions
+
+
+class ClickHouseHook(DbApiHook):
+    """
+    Interact with ClickHouse via the HTTP interface (``clickhouse-connect``).
+
+    This hook wraps ``clickhouse_connect.get_client()`` behind a thin DB-API 2.0
+    adapter (:class:`ClickHouseConnection`), so all standard
+    :class:`~airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`
+    features work out of the box (templating, ``handler``, ``split_statements``,
+    etc.).
+
+    :param database: Optional database name.  Overrides the ``schema`` field of the
+        Airflow connection.  Useful when one connection points to a ClickHouse cluster
+        and individual tasks need to target different databases.
+    :param session_settings: Optional dict of ClickHouse session-level settings
+        (e.g. ``{"max_execution_time": 60, "max_threads": 4}``).  Values supplied
+        here are **merged on top of** any ``session_settings`` dict already present
+        in the connection's ``extra`` JSON field, with the constructor argument
+        taking precedence. For a full list of available session settings visit https://clickhouse.com/docs/operations/settings/settings
+    :param client_kwargs: Optional dict of additional keyword arguments forwarded
+        verbatim to ``clickhouse_connect.get_client()``.  Use this to pass any
+        parameter not otherwise exposed by the hook (e.g. ``http_proxy``,
+        ``pool_mgr_params``).  Values supplied here are **merged on top of** any
+        ``client_kwargs`` dict already present in the connection's ``extra`` JSON
+        field, with the constructor argument taking precedence on conflicting keys.
+        Keys that the hook manages itself (``host``, ``port``, ``username``,
+        ``password``, ``database``, ``secure``, ``verify``, ``client_name``,
+        ``settings``) are silently ignored so that hook-managed values always
+        take precedence.
+
+    .. seealso::
+        - `clickhouse-connect documentation <https://clickhouse.com/docs/en/integrations/python>`_
+        - `clickhouse session settings documentation <https://clickhouse.com/docs/operations/settings/settings>`_
+        - :ref:`howto/connection:clickhouse`
+
+    """
+
+    conn_name_attr = "clickhouse_conn_id"
+    default_conn_name = "clickhouse_default"
+    conn_type = "clickhouse"
+    hook_name = "ClickHouse"
+
+    # ClickHouse has no multi-statement transactions; every statement is auto-committed.
+    supports_autocommit = True
+    _test_connection_sql = "SELECT 1"
+
+    def set_autocommit(self, conn: ClickHouseConnection, autocommit: bool) -> None:
+        """No-op: ClickHouse has no transaction support."""
+
+    def get_autocommit(self, conn: ClickHouseConnection) -> bool:
+        """Return ``True``: ClickHouse auto-commits every statement."""
+        return True
+
+    def __init__(
+        self,
+        *args,
+        database: str | None = None,
+        session_settings: dict[str, Any] | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.database = database
+        self.session_settings: dict[str, Any] = session_settings or {}
+        self.client_kwargs: dict[str, Any] = client_kwargs or {}
+
+    @classmethod
+    def get_connection_form_widgets(cls) -> dict[str, Any]:
+        """Return extra connection form widgets shown in the Airflow UI."""
+        from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+        from flask_babel import lazy_gettext
+        from wtforms import BooleanField, IntegerField, StringField
+
+        return {
+            "secure": BooleanField(
+                label=lazy_gettext("Use TLS (HTTPS)"),
+                description="Enable TLS/HTTPS for the connection (default: False).",
+            ),
+            "verify": BooleanField(
+                label=lazy_gettext("Verify SSL Certificate"),
+                description="Verify the server SSL certificate when TLS is enabled (default: True).",
+            ),
+            "connect_timeout": IntegerField(
+                lazy_gettext("Connection Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "send_receive_timeout": IntegerField(
+                lazy_gettext("Query Timeout (seconds)"),
+                widget=BS3TextFieldWidget(),
+            ),
+            "compress": BooleanField(
+                label=lazy_gettext("Enable LZ4 Compression"),
+                description="Compress query results with LZ4 (default: True).",
+            ),
+            "client_name": StringField(
+                lazy_gettext("Client Name (label)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Optional label appended to the Airflow version identifier in the "
+                    "ClickHouse HTTP User-Agent and system.query_log client_name field. "
+                    "The hook always prepends apache-airflow/<version> "
+                    "apache-airflow-providers-clickhouse/<version>."
+                ),
+            ),
+            "session_settings": StringField(
+                lazy_gettext("Session Settings (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "ClickHouse session-level settings as a JSON object, "
+                    'e.g. {"max_execution_time": 60, "max_threads": 4}.'
+                ),
+            ),
+            "client_kwargs": StringField(
+                lazy_gettext("Client kwargs (JSON)"),
+                widget=BS3TextFieldWidget(),
+                description=(
+                    "Additional keyword arguments forwarded to clickhouse_connect.get_client() "
+                    'as a JSON object, e.g. {"http_proxy": "http://proxy:8080"}. '
+                    "Hook-managed keys (host, port, username, password, database, secure, "
+                    "verify, client_name, settings) are ignored."
+                ),
+            ),
+        }
+
+    @classmethod
+    def get_ui_field_behaviour(cls) -> dict[str, Any]:
+        """Return custom field behaviour for the connection form."""
+        return {
+            "hidden_fields": [],
+            "relabeling": {
+                "schema": "Database",
+            },
+            "placeholders": {
+                "host": "localhost",
+                "port": "8123",
+                "login": "default",
+                "schema": "default",
+                "extra": json.dumps(
+                    {
+                        "secure": False,
+                        "verify": True,
+                        "connect_timeout": 10,
+                        "send_receive_timeout": 300,
+                        "compress": True,
+                        "client_name": "airflow",
+                        "session_settings": {
+                            "max_execution_time": 300,
+                            "max_memory_usage": 10000000000,
+                        },
+                    },
+                    indent=1,
+                ),
+                "secure": "false",
+                "verify": "true",
+                "connect_timeout": "10",
+                "send_receive_timeout": "300",
+                "compress": "true",
+                "client_name": "airflow",
+                "session_settings": '{"max_execution_time": 300}',
+                "client_kwargs": '{"http_proxy": "http://proxy:8080"}',
+            },
+        }
+
+    def _get_client_kwargs(self) -> dict[str, Any]:
+        """
+        Build the keyword-argument dict passed to ``clickhouse_connect.get_client()``.
+
+        Construction order (last write wins):
+
+        1. ``client_kwargs`` from the connection ``extra`` JSON — connection-level
+           passthrough kwargs.
+        2. ``client_kwargs`` constructor argument — task-level overrides merged on top.
+        3. Hook-managed keys (``host``, ``port``, ``username``, ``password``,
+           ``database``, ``secure``, ``verify``, ``client_name``, ``settings``) —
+           always override anything in ``client_kwargs``.
+
+        Optional tuning parameters (``connect_timeout``, ``send_receive_timeout``,
+        ``compress``) are forwarded only when explicitly set in the ``extra`` JSON
+        field so that the driver's own defaults apply otherwise.
+
+        ``client_name`` is always set to a string that identifies the Airflow
+        installation (see :func:`_build_client_name`).  Any ``client_name`` value in
+        ``extra`` is treated as a human-readable label and appended as a comment so
+        it appears alongside the Airflow version info in ``system.query_log``.
+
+        ``session_settings`` from ``extra`` and from the constructor ``session_settings``
+        argument are **merged**, with the constructor argument taking precedence on
+        conflicting keys.
+        """
+        conn = self.get_connection(self.get_conn_id())
+        extra: dict[str, Any] = conn.extra_dejson
+
+        # Merge client_kwargs: extra values are the base, constructor values override.
+        raw_client_kwargs = extra.get("client_kwargs")
+        if isinstance(raw_client_kwargs, str):
+            raw_client_kwargs = json.loads(raw_client_kwargs)
+        merged_client_kwargs: dict[str, Any] = {**(raw_client_kwargs or {}), **self.client_kwargs}
+
+        # Strip hook-managed keys so they cannot overwrite connection credentials
+        # or versioning info.  Log dropped keys at DEBUG to aid troubleshooting.
+        dropped = _HOOK_MANAGED_KWARGS.intersection(merged_client_kwargs)
+        if dropped:
+            self.log.debug(
+                "Ignoring hook-managed keys in client_kwargs: %s",
+                sorted(dropped),
+            )
+        kwargs: dict[str, Any] = {
+            k: v for k, v in merged_client_kwargs.items() if k not in _HOOK_MANAGED_KWARGS
+        }
+
+        # Hook-managed connection parameters always take precedence.
+        kwargs.update(
+            {
+                "host": conn.host or "localhost",
+                "port": int(conn.port) if conn.port else 8123,
+                "username": conn.login or "default",
+                "password": conn.password or "",
+                "database": self.database or conn.schema or "default",
+                "secure": bool(extra.get("secure", False)),
+                "verify": bool(extra.get("verify", True)),
+            }
+        )
+
+        for key in _OPTIONAL_CLIENT_KWARGS:
+            if key in extra:
+                kwargs[key] = extra[key]
+
+        # Always embed Airflow + provider version; user's 'client_name' extra is appended to the User-Agent header.
+        kwargs["client_name"] = _build_client_name(extra.get("client_name"))
+
+        # Merge session_settings: extra values are the base, constructor values override.
+        raw = extra.get("session_settings")
+        if isinstance(raw, str):
+            raw = json.loads(raw)
+        merged: dict[str, Any] = {**(raw or {}), **self.session_settings}
+        if merged:
+            kwargs["settings"] = merged
+
+        return kwargs
+
+    def get_conn(self) -> ClickHouseConnection:
+        """Return a DB-API 2.0 compatible connection backed by ``clickhouse_connect``."""
+        import clickhouse_connect
+
+        client = clickhouse_connect.get_client(**self._get_client_kwargs())
+        return ClickHouseConnection(client)
+
+    def get_client(self) -> clickhouse_connect.driver.client.Client:
+        """
+        Return the raw ``clickhouse_connect`` Client for ClickHouse-specific operations.
+
+        Use this for bulk inserts, streaming queries, or any operation that
+        benefits from the native ``clickhouse_connect`` API rather than DB-API
+        cursors.  The caller is responsible for closing the client.
+        """
+        import clickhouse_connect
+
+        return clickhouse_connect.get_client(**self._get_client_kwargs())
+
+    def bulk_insert_rows(
+        self,
+        table: str,
+        rows: list[tuple],
+        column_names: list[str],
+        commit_every: int = 5000,
+    ) -> None:
+        """
+        Insert rows into a ClickHouse table using the native columnar insert.
+
+        Uses ``clickhouse_connect``'s optimized insert path, which is
+        significantly faster than row-by-row cursor inserts for large datasets.
+
+        :param table: Target table name.
+        :param rows: List of row tuples to insert.
+        :param column_names: Column names matching each position in the row tuples.
+        :param commit_every: Batch size (rows per insert call). Defaults to 5000.
+        """
+        if not rows:
+            self.log.warning(
+                "bulk_insert_rows called with an empty rows list — nothing was inserted into %s.", table
+            )
+            return
+
+        client = self.get_client()
+        try:
+            for i in range(0, len(rows), commit_every):
+                batch = rows[i : i + commit_every]
+                client.insert(table, batch, column_names=column_names)
+            self.log.info("Inserted %d rows into %s", len(rows), table)
+        finally:
+            client.close()
+
+    def get_uri(self) -> str:
+        """
+        Return a SQLAlchemy-compatible URI for ``get_sqlalchemy_engine()``.
+
+        Uses the ``clickhousedb://`` scheme (HTTP) or ``clickhousedbs://`` scheme
+        (HTTPS/TLS) provided by ``clickhouse-connect``, chosen based on the
+        ``secure`` field in the connection's ``extra`` JSON.
+        """
+        conn = self.get_connection(self.get_conn_id())
+        extra: dict[str, Any] = conn.extra_dejson
+        username = conn.login or "default"
+        password = quote_plus(conn.password) if conn.password else ""
+        host = conn.host or "localhost"
+        port = int(conn.port) if conn.port else 8123
+        database = self.database or conn.schema or "default"
+        scheme = "clickhousedbs" if bool(extra.get("secure", False)) else "clickhousedb"

Review Comment:
   Thanks! pushed a fix. LMK if it is ok now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #67080:
URL: https://github.com/apache/airflow/pull/67080#issuecomment-4475359546

   thanks for adding Clickhouse provider @BentsiLeviav 
   I am happy to sponsor the provider. Please follow the procedure listed in https://github.com/apache/airflow/blob/main/providers/ACCEPTING_PROVIDERS.rst#discussion-thread-template we simplified the process of accepting new providers and now it requires just one mailing list thread. This is something that should be in done in parallel to code review of the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add ClickHouse Provider [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #67080:
URL: https://github.com/apache/airflow/pull/67080#issuecomment-4589281616

   Need rebase and resolve conflicts


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org