You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "eladkal (via GitHub)" <gi...@apache.org> on 2023/11/17 16:45:31 UTC

[PR] update elasticsearch doc for `use_ssl` parameter [airflow]

eladkal opened a new pull request, #35707:
URL: https://github.com/apache/airflow/pull/35707

   Starting `elasticsearch>=8` there is no more `use_ssl`
   see https://github.com/apache/airflow/pull/33135/files#r1285347163
   
   mentioning also open question about supporting `elasticsearch<8`
   https://github.com/apache/airflow/pull/33281#issuecomment-1816741321
   
   cc @sunank200 @Owen-CH-Leung 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] update elasticsearch doc for `use_ssl` parameter [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #35707:
URL: https://github.com/apache/airflow/pull/35707#discussion_r1397602563


##########
docs/apache-airflow-providers-elasticsearch/logging/index.rst:
##########
@@ -61,7 +61,7 @@ cert, etc.) use the ``elasticsearch_configs`` setting in your ``airflow.cfg``
     remote_logging = True
 
     [elasticsearch_configs]
-    use_ssl=True
+    use_ssl=True # (This setting is valid only for ``elasticsearch<8``)

Review Comment:
   Maybe we just need to remove it from the documentation?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Cleanup code for elasticsearch<8 [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #35707:
URL: https://github.com/apache/airflow/pull/35707#discussion_r1407281095


##########
airflow/providers/elasticsearch/log/es_task_handler.py:
##########
@@ -70,17 +70,6 @@ def get_es_kwargs_from_config() -> dict[str, Any]:
         if elastic_search_config
         else {}
     )
-    # For elasticsearch>8 retry_timeout have changed for elasticsearch to retry_on_timeout
-    # in Elasticsearch() compared to previous versions.
-    # Read more at: https://elasticsearch-py.readthedocs.io/en/v8.8.2/api.html#module-elasticsearch
-    if (
-        elastic_search_config
-        and "retry_timeout" in elastic_search_config
-        and not kwargs_dict.get("retry_on_timeout")
-    ):
-        retry_timeout = elastic_search_config.get("retry_timeout")
-        if retry_timeout is not None:
-            kwargs_dict["retry_on_timeout"] = retry_timeout

Review Comment:
   I think this for backcompat with previous config.
   
   The problem here that we never raise Depreciation Warning here and just silently replace `retry_timeout` to `retry_on_timeout`
   In theory it could be breaking changes in the other hand note to 5.0.0 release could cover this change https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/stable/changelog.html#id9



##########
tests/providers/elasticsearch/log/test_es_task_handler.py:
##########
@@ -671,14 +671,11 @@ def test_retrieve_config_keys():
     """
     with conf_vars(
         {
-            ("elasticsearch_configs", "use_ssl"): "True",

Review Comment:
   I think we need to keep it in the tests for check that we do not use non-existed parameters



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Cleanup code for elasticsearch<8 [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #35707:
URL: https://github.com/apache/airflow/pull/35707


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Cleanup code for elasticsearch<8 [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #35707:
URL: https://github.com/apache/airflow/pull/35707#discussion_r1407283542


##########
tests/providers/elasticsearch/log/test_es_task_handler.py:
##########
@@ -694,12 +691,10 @@ def test_retrieve_retry_on_timeout():
     """
     with conf_vars(
         {
-            ("elasticsearch_configs", "retry_timeout"): "True",
+            ("elasticsearch_configs", "retry_on_timeout"): "True",

Review Comment:
   From test docstring: ```Test if retrieve timeout is converted to retry_on_timeout.```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Cleanup code for elasticsearch<8 [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #35707:
URL: https://github.com/apache/airflow/pull/35707#discussion_r1411831635


##########
tests/providers/elasticsearch/log/test_es_task_handler.py:
##########
@@ -671,14 +671,11 @@ def test_retrieve_config_keys():
     """
     with conf_vars(
         {
-            ("elasticsearch_configs", "use_ssl"): "True",

Review Comment:
   yes but we no longer need it because we are on `elasticsearch>7`
   the only reference to this parameter was in the docs which was wrong



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] update elasticsearch doc for `use_ssl` parameter [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #35707:
URL: https://github.com/apache/airflow/pull/35707#discussion_r1397609413


##########
docs/apache-airflow-providers-elasticsearch/logging/index.rst:
##########
@@ -61,7 +61,7 @@ cert, etc.) use the ``elasticsearch_configs`` setting in your ``airflow.cfg``
     remote_logging = True
 
     [elasticsearch_configs]
-    use_ssl=True
+    use_ssl=True # (This setting is valid only for ``elasticsearch<8``)

Review Comment:
   maybe but that depends on what I asked in https://github.com/apache/airflow/pull/33281#issuecomment-1816741321



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Cleanup code for elasticsearch<8 [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #35707:
URL: https://github.com/apache/airflow/pull/35707#discussion_r1411827713


##########
tests/providers/elasticsearch/log/test_es_task_handler.py:
##########
@@ -671,14 +671,11 @@ def test_retrieve_config_keys():
     """
     with conf_vars(
         {
-            ("elasticsearch_configs", "use_ssl"): "True",

Review Comment:
   `("elasticsearch_configs", "use_ssl"): "True",` This mostly for check that we remove it from resulting arguments here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Cleanup code for elasticsearch<8 [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #35707:
URL: https://github.com/apache/airflow/pull/35707#discussion_r1407386990


##########
tests/providers/elasticsearch/log/test_es_task_handler.py:
##########
@@ -671,14 +671,11 @@ def test_retrieve_config_keys():
     """
     with conf_vars(
         {
-            ("elasticsearch_configs", "use_ssl"): "True",

Review Comment:
   Which parameter are you worried about?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] update elasticsearch doc for `use_ssl` parameter [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #35707:
URL: https://github.com/apache/airflow/pull/35707#discussion_r1397613990


##########
docs/apache-airflow-providers-elasticsearch/logging/index.rst:
##########
@@ -61,7 +61,7 @@ cert, etc.) use the ``elasticsearch_configs`` setting in your ``airflow.cfg``
     remote_logging = True
 
     [elasticsearch_configs]
-    use_ssl=True
+    use_ssl=True # (This setting is valid only for ``elasticsearch<8``)

Review Comment:
   As far as I remember it is only related to compatible parameters names and should be fixed in: 
   - https://github.com/apache/airflow/pull/34119
   - https://github.com/apache/airflow/issues/34099



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org