You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "uzhastik (via GitHub)" <gi...@apache.org> on 2024/02/15 16:51:39 UTC

[PR] Add Yandex Query support from Yandex.Cloud [airflow]

uzhastik opened a new pull request, #37458:
URL: https://github.com/apache/airflow/pull/37458

   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   Add support for Yandex Query service in Yandex Cloud. Ability to run federative queries in Yandex Cloud.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1525091341


##########
airflow/providers/yandex/provider.yaml:
##########


Review Comment:
   Contributions are welcome. What exactly test did you have in mind to detect it @uzhastik ? How logically it would look ? We are in a process of designing a new way of testing our dependencies (with --resolution lowest)  - but I think you had other test in mind here. could you elaborate what test you had in mind?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1522956524


##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Sequence
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.providers.yandex.hooks.yq import YQHook
+from airflow.providers.yandex.links.yq import YQLink
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):

Review Comment:
   Is it required to extend `SQLExecuteQueryOperator`? asked because this generic class for iterate around Hooks which implements (or thought that it implements) DBApi2.
   
   If not just better to use BaseOperator instead



##########
airflow/providers/yandex/provider.yaml:
##########


Review Comment:
   need also add selected packages as dependencies, this one also a core dependency, however it can't be guarantied that it would be in the future
   
   ```yaml
   - python-dateutil>=2.3
   # Requests 3 if it will be released, will be heavily breaking.
   - requests>=2.27.0,<3
   ```
   
   `python-dateutil` to be honest could be >=2.8.0, which released 3 years ago



##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Sequence
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.providers.yandex.hooks.yq import YQHook
+from airflow.providers.yandex.links.yq import YQLink
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):
+    """
+    Executes sql code using Yandex Query service.
+
+    :param sql: the SQL code to be executed as a single string
+    :param name: name of the query in YandexQuery
+    :param folder_id: cloud folder id where to create query
+    :param connection_id: Airflow connection ID to get parameters from
+    :param folder_id: cloud folder id where to create query
+    """
+
+    operator_extra_links = (YQLink(),)
+    template_fields: Sequence[str] = ("sql",)
+    template_fields_renderers = {"sql": "sql"}
+    template_ext: Sequence[str] = (".sql",)
+    ui_color = "#ededed"
+
+    def __init__(
+        self,
+        *,
+        name: str | None = None,
+        folder_id: str | None = None,
+        connection_id: str | None = None,
+        public_ssh_key: str | None = None,
+        service_account_id: str | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.name = name
+        self.folder_id = folder_id
+        self.connection_id = connection_id
+        self.public_ssh_key = public_ssh_key
+        self.service_account_id = service_account_id
+
+        self.hook: YQHook | None = None

Review Comment:
   Could be a cached property, e.g
   
   ```python
   @cached_property
   def hook() -> YQHook:
       return YQHook(
               yandex_conn_id=self.connection_id,
               default_folder_id=self.folder_id,
               default_public_ssh_key=self.public_ssh_key,
               default_service_account_id=self.service_account_id,
           )
   ```
   
   It would be evaluated first time when it accessed



##########
tests/providers/yandex/operators/test_yandexcloud_yq.py:
##########


Review Comment:
   `tests/providers/yandex/operators/test_yandexcloud_yq.py` -> `tests/providers/yandex/operators/test_yq.py`



##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Sequence
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.providers.yandex.hooks.yq import YQHook
+from airflow.providers.yandex.links.yq import YQLink
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):
+    """
+    Executes sql code using Yandex Query service.
+
+    :param sql: the SQL code to be executed as a single string
+    :param name: name of the query in YandexQuery
+    :param folder_id: cloud folder id where to create query
+    :param connection_id: Airflow connection ID to get parameters from
+    :param folder_id: cloud folder id where to create query
+    """
+
+    operator_extra_links = (YQLink(),)
+    template_fields: Sequence[str] = ("sql",)
+    template_fields_renderers = {"sql": "sql"}
+    template_ext: Sequence[str] = (".sql",)
+    ui_color = "#ededed"
+
+    def __init__(
+        self,
+        *,
+        name: str | None = None,
+        folder_id: str | None = None,
+        connection_id: str | None = None,

Review Comment:
   ```suggestion
           yandex_conn_id: str | None = None,
   ```
   
   I guess better to use same naming elsewhere, do not forget to change in docstring and into the other places



##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########


Review Comment:
   `airflow/providers/yandex/operators/yandexcloud_yq.py` -> `airflow/providers/yandex/operators/yq.py` the same naming as for Hook



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "uzhastik (via GitHub)" <gi...@apache.org>.
uzhastik commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1524713996


##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Sequence
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.providers.yandex.hooks.yq import YQHook
+from airflow.providers.yandex.links.yq import YQLink
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):
+    """
+    Executes sql code using Yandex Query service.
+
+    :param sql: the SQL code to be executed as a single string
+    :param name: name of the query in YandexQuery
+    :param folder_id: cloud folder id where to create query
+    :param connection_id: Airflow connection ID to get parameters from
+    :param folder_id: cloud folder id where to create query
+    """
+
+    operator_extra_links = (YQLink(),)
+    template_fields: Sequence[str] = ("sql",)
+    template_fields_renderers = {"sql": "sql"}
+    template_ext: Sequence[str] = (".sql",)
+    ui_color = "#ededed"
+
+    def __init__(
+        self,
+        *,
+        name: str | None = None,
+        folder_id: str | None = None,
+        connection_id: str | None = None,
+        public_ssh_key: str | None = None,
+        service_account_id: str | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.name = name
+        self.folder_id = folder_id
+        self.connection_id = connection_id
+        self.public_ssh_key = public_ssh_key
+        self.service_account_id = service_account_id
+
+        self.hook: YQHook | None = None

Review Comment:
   okey, as I can see this is very common approach across airflow codebase to have cached property for hook. I will follow common approach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1525095700


##########
airflow/providers/yandex/provider.yaml:
##########


Review Comment:
   But maybe that's what you indeed had in mind -> this is the description of what we plan to do https://github.com/apache/airflow/issues/35549 -> if you have other ideas than that, I am all ears what you could propose.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "uzhastik (via GitHub)" <gi...@apache.org>.
uzhastik commented on PR #37458:
URL: https://github.com/apache/airflow/pull/37458#issuecomment-1988008811

   @Taragolis @potiuk @eladkal @rozza @ashb @jedcunningham hello, team! finally PR is green. please check it out! =))


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1524615923


##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Sequence
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.providers.yandex.hooks.yq import YQHook
+from airflow.providers.yandex.links.yq import YQLink
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):

Review Comment:
   > We use it for query text templating and for syntax highlighting (sql dialect). Can I achieve this with BaseOperator
   
   1. `template_fields: Sequence[str] = ("sql",)` - uses for template operator's sql attribute
   2. `template_ext: Sequence[str] = (".sql",)` - uses for template fields from the `.sql` files
   3. `template_fields_renderers = {"sql": "sql"}` - uses for highlight syntax



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1524619767


##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Sequence
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.providers.yandex.hooks.yq import YQHook
+from airflow.providers.yandex.links.yq import YQLink
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):

Review Comment:
   pp 1 and 2 even testable
   
   some samples from the other providers
   
   https://github.com/apache/airflow/blob/1d3010c4ab738b4fc8e31e2ef0fb880dcf06bec6/tests/providers/cncf/kubernetes/operators/test_spark_kubernetes.py#L511-L535
   
   https://github.com/apache/airflow/blob/1d3010c4ab738b4fc8e31e2ef0fb880dcf06bec6/tests/providers/cncf/kubernetes/operators/test_spark_kubernetes.py#L570-L595



##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Sequence
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.providers.yandex.hooks.yq import YQHook
+from airflow.providers.yandex.links.yq import YQLink
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):

Review Comment:
   > We use it for query text templating and for syntax highlighting (sql dialect). Can I achieve this with BaseOperator
   
   1. `template_fields: Sequence[str] = ("sql",)` - uses for template operator's sql attribute
   2. `template_ext: Sequence[str] = (".sql",)` - uses for template fields from the `.sql` files
   3. `template_fields_renderers = {"sql": "sql"}` - uses for highlight syntax into the UI



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #37458:
URL: https://github.com/apache/airflow/pull/37458#issuecomment-1947076850

   And static checks and mypy should be fixed:
   - https://github.com/apache/airflow/actions/runs/7919541474/job/21623012828?pr=37458
   - https://github.com/apache/airflow/actions/runs/7919541474/job/21623011765?pr=37458
   
   See [Static Code Checks](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst) for the reference 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "uzhastik (via GitHub)" <gi...@apache.org>.
uzhastik commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1492377851


##########
airflow/providers/yandex/hooks/http_client.py:
##########
@@ -0,0 +1,321 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This file is a copy of https://github.com/ydb-platform/ydb/tree/284b7efb67edcdade0b12c849b7fad40739ad62b/ydb/core/fq/libs/http_api_client
+# It is highly recommended to modify original file first in YDB project and merge it here afterwards

Review Comment:
   I'm not sure the current vendoring workflow is suitable for my situation. I'm the author of these two files, but they are hosted in two places: ydb github repo and airflow github repo. In ydb repo we have integration tests where we ensure protocol between client and YandexQuery (YQ) backend. These tests cannot be moved to airflow repo. So the source of truth is ydb repo. But this client logic is required for airflow operator and there is no pypi package which contains this code. So I decided just to copy it and add some tests with hardcoded YQ backend reponses. I would suggest to fix original code first and then backport it to airflow repo. I can rework these files as if they where created just inside airflow without link to ydb repo, but I wouldn't recommend to maintain them separately from ydb repo.
   YDB repo has Apache 2 licence, therefore it is safe to copy files from ydb repo completely or partially to another repository.
   We have amazing plan to improve our YQ backend so that it would pretend to be a YDB cluster. That means that we can use existing YDB python SDK ( https://github.com/ydb-platform/ydb-python-sdk ) to access YQ backend and these two files with http client logic could be removed afterwards. But improving YQ backend is not easy^ we would spent several months to make it.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #37458:
URL: https://github.com/apache/airflow/pull/37458#issuecomment-1946570818

   Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
   Here are some useful points:
   - Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-pre-commit-hooks) will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
   - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
   - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
   - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
   - Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
   Apache Airflow is a community-driven project and together we are making it better 🚀.
   In case of doubts contact the developers at:
   Mailing List: dev@airflow.apache.org
   Slack: https://s.apache.org/airflow-slack
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1525091341


##########
airflow/providers/yandex/provider.yaml:
##########


Review Comment:
   Contributions are welcome. What exactly test did you have in mind to detect it @uzhastik ? How logically it would look ? We are in a process of designing a new way of testing our dependencies (with `--resolution lowest` of `uv` - per provider)  - but I think you had other test in mind here. could you elaborate what test you had in mind?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "uzhastik (via GitHub)" <gi...@apache.org>.
uzhastik commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1523224271


##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Sequence
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.providers.yandex.hooks.yq import YQHook
+from airflow.providers.yandex.links.yq import YQLink
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):

Review Comment:
   We use it for query text templating and for syntax highlighting (sql dialect). Can I achieve this with BaseOperator? I doubt.
   We use it for backfill as well when Airflow starts our query and provides time intervals



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1491369765


##########
airflow/providers/yandex/provider.yaml:
##########
@@ -62,25 +61,41 @@ integrations:
     logo: /integration-logos/yandex/Yandex-Cloud.png
     tags: [service]
 
+  - integration-name: Yandex.Cloud YQ
+    external-doc-url: https://cloud.yandex.com/en/services/query
+    how-to-guide:
+      - /docs/apache-airflow-providers-yandex/operators.rst
+    logo: /integration-logos/yandex/Yandex-Cloud.png
+    tags: [service]
+
 operators:
   - integration-name: Yandex.Cloud Dataproc
     python-modules:
       - airflow.providers.yandex.operators.yandexcloud_dataproc
 
+  - integration-name: Yandex.Cloud YQ
+    python-modules:
+      - airflow.providers.yandex.operators.yandexcloud_yq
+
 hooks:
   - integration-name: Yandex.Cloud
     python-modules:
       - airflow.providers.yandex.hooks.yandex
   - integration-name: Yandex.Cloud Dataproc
     python-modules:
       - airflow.providers.yandex.hooks.yandexcloud_dataproc
+  - integration-name: Yandex.Cloud YQ
+    python-modules:
+      - airflow.providers.yandex.hooks.yandexcloud_yq
+      - airflow.providers.yandex.hooks.http_client
+      - airflow.providers.yandex.hooks.query_results
 
 connection-types:
   - hook-class-name: airflow.providers.yandex.hooks.yandex.YandexCloudBaseHook
     connection-type: yandexcloud
 
-secrets-backends:
-  - airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend

Review Comment:
   I guess you accidentally delete support of Secrets Backend



##########
airflow/providers/yandex/hooks/http_client.py:
##########


Review Comment:
   This doesn't look like Airflow Hook to me



##########
airflow/providers/yandex/hooks/query_results.py:
##########


Review Comment:
   This doesn't look as Airflow Hook to me



##########
airflow/providers/yandex/hooks/http_client.py:
##########
@@ -0,0 +1,321 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This file is a copy of https://github.com/ydb-platform/ydb/tree/284b7efb67edcdade0b12c849b7fad40739ad62b/ydb/core/fq/libs/http_api_client
+# It is highly recommended to modify original file first in YDB project and merge it here afterwards

Review Comment:
   And are you check that original licence allow to copy the code to another project?



##########
airflow/providers/yandex/hooks/http_client.py:
##########
@@ -0,0 +1,321 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This file is a copy of https://github.com/ydb-platform/ydb/tree/284b7efb67edcdade0b12c849b7fad40739ad62b/ydb/core/fq/libs/http_api_client
+# It is highly recommended to modify original file first in YDB project and merge it here afterwards

Review Comment:
   Vendored parts should be stored into the `_vendor`, may I know the reason why we need vendor this code?



##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,100 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Sequence, Any
+from airflow.configuration import conf
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.models import BaseOperator, BaseOperatorLink, XCom
+from airflow.models.taskinstancekey import TaskInstanceKey
+from airflow.providers.yandex.hooks.yandexcloud_yq import YQHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+XCOM_WEBLINK_KEY = "web_link"
+
+
+class YQLink(BaseOperatorLink):
+    name = "Yandex Query"
+
+    def get_link(self, operator: BaseOperator, *, ti_key: TaskInstanceKey):
+        return XCom.get_value(key=XCOM_WEBLINK_KEY, ti_key=ti_key) or "https://yq.cloud.yandex.ru"
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):
+    """
+    Executes sql code using Yandex Query service.
+
+    :param sql: the SQL code to be executed as a single string
+    :param name: name of the query in YandexQuery
+    :param folder_id: cloud folder id where to create query
+    :param connection_id: Airflow connection ID to get parameters from
+    :param folder_id: cloud folder id where to create query
+    """
+
+    operator_extra_links = (YQLink(),)
+    template_fields: Sequence[str] = ("sql",)
+    template_fields_renderers = {"sql": "sql"}
+    template_ext: Sequence[str] = (".sql",)
+    ui_color = "#ededed"
+
+    def __init__(
+        self,
+        *,
+        name: str | None = None,
+        folder_id: str | None = None,
+        connection_id: str | None = None,
+        public_ssh_key: str | None = None,
+        service_account_id: str | None = None,
+        deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False),

Review Comment:
   I can't find any deferrable implementations



##########
airflow/providers/yandex/hooks/http_client.py:
##########
@@ -0,0 +1,321 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This file is a copy of https://github.com/ydb-platform/ydb/tree/284b7efb67edcdade0b12c849b7fad40739ad62b/ydb/core/fq/libs/http_api_client
+# It is highly recommended to modify original file first in YDB project and merge it here afterwards
+
+from __future__ import annotations
+
+import logging
+import time
+from datetime import datetime
+import requests
+
+from requests.adapters import HTTPAdapter
+from typing import Any
+from urllib3.util.retry import Retry
+
+from .query_results import YQResults

Review Comment:
   We do not use relative imports into the Airflow repo



##########
airflow/providers/yandex/hooks/yandexcloud_yq.py:
##########
@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+from requests.packages.urllib3.util.retry import Retry
+
+from datetime import timedelta
+import logging
+import requests
+import time
+from typing import Any
+import jwt
+
+# These two lines enable debugging at httplib level (requests->urllib3->http.client)
+# You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
+# The only thing missing will be the response.body which is not logged.
+
+import http.client
+
+http.client.HTTPConnection.debuglevel = 1
+
+# You must initialize logging, otherwise you'll not see debug output.
+logging.basicConfig()
+logging.getLogger().setLevel(logging.DEBUG)
+requests_log = logging.getLogger("requests.packages.urllib3")
+requests_log.setLevel(logging.DEBUG)
+requests_log.propagate = True

Review Comment:
   ```suggestion
   ```
   
   Looks like some debug code



##########
airflow/providers/yandex/hooks/http_client.py:
##########
@@ -0,0 +1,321 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This file is a copy of https://github.com/ydb-platform/ydb/tree/284b7efb67edcdade0b12c849b7fad40739ad62b/ydb/core/fq/libs/http_api_client
+# It is highly recommended to modify original file first in YDB project and merge it here afterwards
+
+from __future__ import annotations
+
+import logging
+import time
+from datetime import datetime
+import requests
+
+from requests.adapters import HTTPAdapter
+from typing import Any
+from urllib3.util.retry import Retry
+
+from .query_results import YQResults
+
+MAX_RETRY_FOR_SESSION = 4
+BACK_OFF_FACTOR = 0.3
+TIME_BETWEEN_RETRIES = 1000
+ERROR_CODES = (500, 502, 504)
+
+
+def requests_retry_session(
+    session, retries=MAX_RETRY_FOR_SESSION, back_off_factor=BACK_OFF_FACTOR, status_force_list=ERROR_CODES
+):
+    retry = Retry(
+        total=retries,
+        read=retries,
+        connect=retries,
+        backoff_factor=back_off_factor,
+        status_forcelist=status_force_list,
+        allowed_methods=frozenset(["GET", "POST"]),
+    )
+    adapter = HTTPAdapter(max_retries=retry)
+    session.mount("http://", adapter)
+    session.mount("https://", adapter)
+    return session
+
+
+class YQHttpClientConfig(object):
+    def __init__(
+        self,
+        token: str | None = None,
+        project: str | None = None,
+        user_agent: str | None = "Python YQ HTTP SDK",
+    ) -> None:
+        assert len(token) > 0, "empty token"
+        self.token = token
+        self.project = project
+        self.user_agent = user_agent
+
+        # urls should not contain trailing /
+        self.endpoint: str = "https://api.yandex-query.cloud.yandex.net"
+        self.web_base_url: str = "https://yq.cloud.yandex.ru"
+        self.token_prefix = "Bearer "
+
+
+class YQHttpClientException(Exception):
+    def __init__(self, message: str, status: str = None, msg: str = None, details: Any = None) -> None:
+        super().__init__(message)
+        self.status = status
+        self.msg = msg
+        self.details = details
+
+
+class YQHttpClient(object):

Review Comment:
   ```suggestion
   class YQHttpClient:
   ```



##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,100 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Sequence, Any
+from airflow.configuration import conf
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.models import BaseOperator, BaseOperatorLink, XCom
+from airflow.models.taskinstancekey import TaskInstanceKey
+from airflow.providers.yandex.hooks.yandexcloud_yq import YQHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+XCOM_WEBLINK_KEY = "web_link"
+
+
+class YQLink(BaseOperatorLink):
+    name = "Yandex Query"
+
+    def get_link(self, operator: BaseOperator, *, ti_key: TaskInstanceKey):
+        return XCom.get_value(key=XCOM_WEBLINK_KEY, ti_key=ti_key) or "https://yq.cloud.yandex.ru"

Review Comment:
   Better to move links into the separate sub-package
   
   - https://github.com/apache/airflow/tree/main/airflow/providers/amazon/aws/links
   - https://github.com/apache/airflow/tree/main/airflow/providers/google/cloud/links



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "uzhastik (via GitHub)" <gi...@apache.org>.
uzhastik commented on PR #37458:
URL: https://github.com/apache/airflow/pull/37458#issuecomment-1948914923

   > I'm not familiar with Yandex cloud and their API, but is any chance that Yandex Query supported by [`yandexcloud`](https://pypi.org/project/yandexcloud/)
   > 
   > cc: @Piatachock @ImpressionableRaccoon
   
   yes, It's possible, but not now. we are trying different approach, described here: https://github.com/apache/airflow/pull/37458#discussion_r1491375103


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "uzhastik (via GitHub)" <gi...@apache.org>.
uzhastik commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1528362006


##########
airflow/providers/yandex/provider.yaml:
##########


Review Comment:
   @potiuk I'm working in minimal context - single provider. And I have problem with describing dependencies. It would be nice to have ability to run provider tests in venv of described dependencies. If I forgot to add dependency in yaml, my tests should fail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis merged PR #37458:
URL: https://github.com/apache/airflow/pull/37458


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "uzhastik (via GitHub)" <gi...@apache.org>.
uzhastik commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1523224271


##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Sequence
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.providers.yandex.hooks.yq import YQHook
+from airflow.providers.yandex.links.yq import YQLink
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):

Review Comment:
   We use it for query text templating and for syntax highlighting (sql dialect). Can I achieve this with BaseOperator? I doubt
   We use it for backfill as well when Airflow starts our query and provides time intervals



##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Sequence
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.providers.yandex.hooks.yq import YQHook
+from airflow.providers.yandex.links.yq import YQLink
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):
+    """
+    Executes sql code using Yandex Query service.
+
+    :param sql: the SQL code to be executed as a single string
+    :param name: name of the query in YandexQuery
+    :param folder_id: cloud folder id where to create query
+    :param connection_id: Airflow connection ID to get parameters from
+    :param folder_id: cloud folder id where to create query
+    """
+
+    operator_extra_links = (YQLink(),)
+    template_fields: Sequence[str] = ("sql",)
+    template_fields_renderers = {"sql": "sql"}
+    template_ext: Sequence[str] = (".sql",)
+    ui_color = "#ededed"
+
+    def __init__(
+        self,
+        *,
+        name: str | None = None,
+        folder_id: str | None = None,
+        connection_id: str | None = None,
+        public_ssh_key: str | None = None,
+        service_account_id: str | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.name = name
+        self.folder_id = folder_id
+        self.connection_id = connection_id
+        self.public_ssh_key = public_ssh_key
+        self.service_account_id = service_account_id
+
+        self.hook: YQHook | None = None

Review Comment:
   I know exact moment when to create and destroy hook. Any situations when caching is preferable? More ver it would be nice to call "hook.close()" when it is not needed anymore.
   Current code is quite straightforward: creates hook in execute and destroys it in on_kill.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "uzhastik (via GitHub)" <gi...@apache.org>.
uzhastik commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1525050181


##########
airflow/providers/yandex/provider.yaml:
##########


Review Comment:
   it would be nice to have some tests to detect problems in dependencies configuration. for instance run provider tests in venv



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "uzhastik (via GitHub)" <gi...@apache.org>.
uzhastik commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1509972280


##########
airflow/providers/yandex/hooks/http_client.py:
##########
@@ -0,0 +1,321 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This file is a copy of https://github.com/ydb-platform/ydb/tree/284b7efb67edcdade0b12c849b7fad40739ad62b/ydb/core/fq/libs/http_api_client
+# It is highly recommended to modify original file first in YDB project and merge it here afterwards

Review Comment:
   added test with mock http client



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1498408740


##########
airflow/providers/yandex/hooks/http_client.py:
##########
@@ -0,0 +1,321 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This file is a copy of https://github.com/ydb-platform/ydb/tree/284b7efb67edcdade0b12c849b7fad40739ad62b/ydb/core/fq/libs/http_api_client
+# It is highly recommended to modify original file first in YDB project and merge it here afterwards

Review Comment:
   If this is just the one file, I am not too concerned for licence and copying. These are merely wrappers around a few calls. 
   
   However,  that calls even more for a set of tests there. It would be enough to have a set of test cases where the client is mocked (as we do in many other hooks. This has the nice feature as well that it protects against accidental regressions. if this code is copied from elsewhere, having independent set of hook tests that a) mock client calls b) simulate what the operator does - is ratther benefitial to prevent problems, imagine for example the code that got copied will have a method renamed but you forget to update the operator. If you have a set of Hook unit tests that call all the methods - it wil be cautght there (of course MyPy and linters might catch it as well but this is not always possible). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1524627620


##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Sequence
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.providers.yandex.hooks.yq import YQHook
+from airflow.providers.yandex.links.yq import YQLink
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):
+    """
+    Executes sql code using Yandex Query service.
+
+    :param sql: the SQL code to be executed as a single string
+    :param name: name of the query in YandexQuery
+    :param folder_id: cloud folder id where to create query
+    :param connection_id: Airflow connection ID to get parameters from
+    :param folder_id: cloud folder id where to create query
+    """
+
+    operator_extra_links = (YQLink(),)
+    template_fields: Sequence[str] = ("sql",)
+    template_fields_renderers = {"sql": "sql"}
+    template_ext: Sequence[str] = (".sql",)
+    ui_color = "#ededed"
+
+    def __init__(
+        self,
+        *,
+        name: str | None = None,
+        folder_id: str | None = None,
+        connection_id: str | None = None,
+        public_ssh_key: str | None = None,
+        service_account_id: str | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.name = name
+        self.folder_id = folder_id
+        self.connection_id = connection_id
+        self.public_ssh_key = public_ssh_key
+        self.service_account_id = service_account_id
+
+        self.hook: YQHook | None = None

Review Comment:
   In most cases it preferable:
   - Type Checking, time-to time mypy might complain about `YQHook | None` because for it not every-time clear is it  already assigned or not,
   - Changes in the code, we not assume that code will frozen in time, which might accidentally run before self.hook assigned
   
   So properties help in this cases, and also make mocking easier in complex situation.
   However property itself (getter) will create hook every time, so better to use `cached_propery` which assigned exactly one time during first invocation.
   
   For now that is fine, but it is a big chance that it moved in `cached_propery` in the future when something goes wrong.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #37458:
URL: https://github.com/apache/airflow/pull/37458#issuecomment-2010010788

   Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "uzhastik (via GitHub)" <gi...@apache.org>.
uzhastik commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1524882892


##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Sequence
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.providers.yandex.hooks.yq import YQHook
+from airflow.providers.yandex.links.yq import YQLink
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):

Review Comment:
   BaseOperator is much better!=) Thank you for help!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

Posted by "uzhastik (via GitHub)" <gi...@apache.org>.
uzhastik commented on PR #37458:
URL: https://github.com/apache/airflow/pull/37458#issuecomment-1993994426

   @Taragolis @potiuk @eladkal @rozza @ashb @jedcunningham ping again =)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org