You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "SameerMesiah97 (via GitHub)" <gi...@apache.org> on 2026/06/11 22:09:55 UTC

[PR] ElasticsearchSQLHook: add chunked Polars DataFrame support via custom SQL reader [airflow]

SameerMesiah97 opened a new pull request, #68411:
URL: https://github.com/apache/airflow/pull/68411

   **Description**
   
   This PR is a follow-up to #66220, which added support for returning Elasticsearch SQL query results as a Polars DataFrame via a custom cursor-based reader.
   
   This change extends that implementation to support `get_df_by_chunks(df_type="polars")` by introducing a chunked reader that uses Elasticsearch SQL cursor-based pagination to yield Polars DataFrames incrementally.
   
   **Rationale**
   
   The Common SQL implementation of `get_df_by_chunks(df_type="polars")` relies on `polars.read_database`, which is not compatible with Elasticsearch's SQL API and custom cursor implementation.
   
   To provide feature parity with the existing Polars DataFrame support added in #66220, this change introduces an Elasticsearch-specific chunked reader that interacts directly with the Elasticsearch SQL API and yields Polars DataFrames of the requested chunk size.
   
   **Tests**
   
   Added unit tests verifying that:
   
   * Elasticsearch SQL responses are correctly yielded as Polars DataFrames when the result set fits within a single chunk.
   * Result sets are split correctly across multiple chunks.
   * Chunk boundaries are handled correctly when they span multiple Elasticsearch cursor pages.
   * Cursor cleanup is performed using the last non-null cursor and is skipped when no cursor is returned by Elasticsearch.
   * The hook correctly delegates to the custom chunked reader when `get_df_by_chunks(df_type="polars")` is requested.
   
   **Documentation**
   
   Added docstrings for the chunked reader describing the use of Elasticsearch SQL cursor-based pagination and chunked Polars DataFrame generation.
   
   **Backwards Compatibility**
   
   This change adds support for `get_df_by_chunks(df_type="polars")` in ElasticsearchSQLHook and does not modify existing behavior. No breaking changes are introduced.
   
   Closes: ##66290
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: [GPT 5.5] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ElasticsearchSQLHook: add chunked Polars DataFrame support via custom SQL reader [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #68411:
URL: https://github.com/apache/airflow/pull/68411#issuecomment-4832679030

   > @shahar1 
   > 
   > Can this be merged?
   
   Done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ElasticsearchSQLHook: add chunked Polars DataFrame support via custom SQL reader [airflow]

Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #68411:
URL: https://github.com/apache/airflow/pull/68411#discussion_r3476608467


##########
providers/elasticsearch/tests/unit/elasticsearch/utils/test_sql.py:
##########
@@ -179,3 +177,170 @@ def test_read_sql_to_polars_no_cursor_cleanup():
     read_sql_to_polars(es, "SELECT *")
 
     es.sql.clear_cursor.assert_not_called()
+
+
+def test_read_sql_to_polars_by_chunks_single_chunk():
+    es = _mock_es(
+        [
+            {
+                "columns": COLUMNS,
+                "rows": [[1, "a"], [2, "b"]],
+            }
+        ]
+    )
+
+    chunks = list(
+        read_sql_to_polars_by_chunks(
+            es,
+            "SELECT *",
+            chunksize=10,
+        )
+    )
+
+    assert len(chunks) == 1
+
+    df = chunks[0]
+    assert df.shape == (2, 2)
+    assert df.to_dict(as_series=False) == {
+        "id": [1, 2],
+        "name": ["a", "b"],
+    }
+
+
+def test_read_sql_to_polars_by_chunks_single_page_multiple_chunks():
+    es = _mock_es(
+        [
+            {
+                "columns": COLUMNS,
+                "rows": [
+                    [1, "a"],
+                    [2, "b"],
+                    [3, "c"],
+                ],
+            }
+        ]
+    )
+
+    chunks = list(
+        read_sql_to_polars_by_chunks(
+            es,
+            "SELECT *",
+            chunksize=2,
+        )
+    )
+
+    assert [chunk.shape for chunk in chunks] == [
+        (2, 2),
+        (1, 2),
+    ]
+
+    assert chunks[0].to_dict(as_series=False) == {
+        "id": [1, 2],
+        "name": ["a", "b"],
+    }
+
+    assert chunks[1].to_dict(as_series=False) == {
+        "id": [3],
+        "name": ["c"],
+    }
+
+
+def test_read_sql_to_polars_by_chunks_across_cursor_pages():
+    es = _mock_es(
+        [
+            {
+                "columns": COLUMNS,
+                "rows": [[1, "a"], [2, "b"]],
+                "cursor": "cursor_1",
+            },
+            {
+                "rows": [[3, "c"], [4, "d"]],
+                "cursor": None,
+            },
+        ]
+    )
+
+    chunks = list(
+        read_sql_to_polars_by_chunks(
+            es,
+            "SELECT *",
+            chunksize=3,
+        )
+    )
+
+    assert len(chunks) == 2
+
+    assert chunks[0].to_dict(as_series=False) == {
+        "id": [1, 2, 3],
+        "name": ["a", "b", "c"],
+    }
+
+    assert chunks[1].to_dict(as_series=False) == {
+        "id": [4],
+        "name": ["d"],
+    }
+
+
+def test_read_sql_to_polars_by_chunks_clears_cursor():
+    es = _mock_es(
+        [
+            {
+                "columns": COLUMNS,
+                "rows": [[1, "a"]],
+                "cursor": "cursor_1",
+            },
+            {
+                "rows": [[2, "b"]],
+                "cursor": None,
+            },
+        ]
+    )
+
+    list(
+        read_sql_to_polars_by_chunks(
+            es,
+            "SELECT *",
+            chunksize=1,
+        )
+    )
+
+    es.sql.clear_cursor.assert_called_once()
+
+
+def test_read_sql_to_polars_by_chunks_no_cursor_cleanup():
+    es = _mock_es(
+        [
+            {
+                "columns": COLUMNS,
+                "rows": [[1, "a"]],
+            }
+        ]
+    )
+
+    list(
+        read_sql_to_polars_by_chunks(
+            es,
+            "SELECT *",
+            chunksize=1,
+        )
+    )
+
+    es.sql.clear_cursor.assert_not_called()
+
+
+def test_read_sql_to_polars_clears_cursor_on_pagination_error():
+    es = MagicMock()
+
+    es.sql.query.side_effect = [
+        {
+            "columns": COLUMNS,
+            "rows": [[1, "a"]],
+            "cursor": "cursor_1",
+        },
+        RuntimeError("boom"),
+    ]
+
+    with pytest.raises(RuntimeError, match="boom"):
+        read_sql_to_polars(es, "SELECT *")

Review Comment:
   Thanks for pointing that out. It should be testing `read_sql_to_polars_by_chunks` so I have changed it to that in my latest commit. And it is now being wrapped in a `list` as this is a generator.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ElasticsearchSQLHook: add chunked Polars DataFrame support via custom SQL reader [airflow]

Posted by "justinpakzad (via GitHub)" <gi...@apache.org>.
justinpakzad commented on code in PR #68411:
URL: https://github.com/apache/airflow/pull/68411#discussion_r3408712295


##########
providers/elasticsearch/tests/unit/elasticsearch/utils/test_sql.py:
##########
@@ -179,3 +177,152 @@ def test_read_sql_to_polars_no_cursor_cleanup():
     read_sql_to_polars(es, "SELECT *")
 
     es.sql.clear_cursor.assert_not_called()
+
+
+def test_read_sql_to_polars_by_chunks_single_chunk():
+    es = _mock_es(
+        [
+            {
+                "columns": COLUMNS,
+                "rows": [[1, "a"], [2, "b"]],
+            }
+        ]
+    )
+
+    chunks = list(
+        read_sql_to_polars_by_chunks(
+            es,
+            "SELECT *",
+            chunksize=10,
+        )
+    )
+
+    assert len(chunks) == 1
+
+    df = chunks[0]
+    assert df.shape == (2, 2)
+    assert df.to_dict(as_series=False) == {
+        "id": [1, 2],
+        "name": ["a", "b"],
+    }
+
+
+def test_read_sql_to_polars_by_chunks_single_page_multiple_chunks():
+    es = _mock_es(
+        [
+            {
+                "columns": COLUMNS,
+                "rows": [
+                    [1, "a"],
+                    [2, "b"],
+                    [3, "c"],
+                ],
+            }
+        ]
+    )
+
+    chunks = list(
+        read_sql_to_polars_by_chunks(
+            es,
+            "SELECT *",
+            chunksize=2,
+        )
+    )
+
+    assert [chunk.shape for chunk in chunks] == [
+        (2, 2),
+        (1, 2),
+    ]
+
+    assert chunks[0].to_dict(as_series=False) == {
+        "id": [1, 2],
+        "name": ["a", "b"],
+    }
+
+    assert chunks[1].to_dict(as_series=False) == {
+        "id": [3],
+        "name": ["c"],
+    }
+
+
+def test_read_sql_to_polars_by_chunks_across_cursor_pages():
+    es = _mock_es(
+        [
+            {
+                "columns": COLUMNS,
+                "rows": [[1, "a"], [2, "b"]],
+                "cursor": "cursor_1",
+            },
+            {
+                "rows": [[3, "c"], [4, "d"]],
+                "cursor": None,
+            },
+        ]
+    )
+
+    chunks = list(
+        read_sql_to_polars_by_chunks(
+            es,
+            "SELECT *",
+            chunksize=3,
+        )
+    )
+
+    assert len(chunks) == 2
+
+    assert chunks[0].to_dict(as_series=False) == {
+        "id": [1, 2, 3],
+        "name": ["a", "b", "c"],
+    }
+
+    assert chunks[1].to_dict(as_series=False) == {
+        "id": [4],
+        "name": ["d"],
+    }
+
+
+def test_read_sql_to_polars_by_chunks_clears_cursor():

Review Comment:
   This test and the one below both test the happy paths, last_cursor exists cleanup, no last_cursor exists, cleanup is skipped. Is it worth adding a test for the case where an error occurs mid-iteration but last_cursor exists, and cleanup still happens?



##########
providers/elasticsearch/tests/unit/elasticsearch/hooks/test_elasticsearch.py:
##########
@@ -276,6 +276,32 @@ def test_get_df_polars(self, mock_reader):
             max_rows=10,
         )
 
+    @mock.patch("airflow.providers.elasticsearch.hooks.elasticsearch.read_sql_to_polars_by_chunks")
+    def test_get_df_polars_df_by_chunks(self, mock_reader):
+        mock_reader.return_value = "df"

Review Comment:
   Is this needed here? It gets overridden a couple lines below with `mock_reader.return_value = generator`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ElasticsearchSQLHook: add chunked Polars DataFrame support via custom SQL reader [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 merged PR #68411:
URL: https://github.com/apache/airflow/pull/68411


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ElasticsearchSQLHook: add chunked Polars DataFrame support via custom SQL reader [airflow]

Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on PR #68411:
URL: https://github.com/apache/airflow/pull/68411#issuecomment-4832615709

   @shahar1 
   
   Can this be merged?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ElasticsearchSQLHook: add chunked Polars DataFrame support via custom SQL reader [airflow]

Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on PR #68411:
URL: https://github.com/apache/airflow/pull/68411#issuecomment-4703128286

   @justinpakzad 
   
   I have addressed your feedback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ElasticsearchSQLHook: add chunked Polars DataFrame support via custom SQL reader [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #68411:
URL: https://github.com/apache/airflow/pull/68411#discussion_r3476412066


##########
providers/elasticsearch/tests/unit/elasticsearch/utils/test_sql.py:
##########
@@ -179,3 +177,170 @@ def test_read_sql_to_polars_no_cursor_cleanup():
     read_sql_to_polars(es, "SELECT *")
 
     es.sql.clear_cursor.assert_not_called()
+
+
+def test_read_sql_to_polars_by_chunks_single_chunk():
+    es = _mock_es(
+        [
+            {
+                "columns": COLUMNS,
+                "rows": [[1, "a"], [2, "b"]],
+            }
+        ]
+    )
+
+    chunks = list(
+        read_sql_to_polars_by_chunks(
+            es,
+            "SELECT *",
+            chunksize=10,
+        )
+    )
+
+    assert len(chunks) == 1
+
+    df = chunks[0]
+    assert df.shape == (2, 2)
+    assert df.to_dict(as_series=False) == {
+        "id": [1, 2],
+        "name": ["a", "b"],
+    }
+
+
+def test_read_sql_to_polars_by_chunks_single_page_multiple_chunks():
+    es = _mock_es(
+        [
+            {
+                "columns": COLUMNS,
+                "rows": [
+                    [1, "a"],
+                    [2, "b"],
+                    [3, "c"],
+                ],
+            }
+        ]
+    )
+
+    chunks = list(
+        read_sql_to_polars_by_chunks(
+            es,
+            "SELECT *",
+            chunksize=2,
+        )
+    )
+
+    assert [chunk.shape for chunk in chunks] == [
+        (2, 2),
+        (1, 2),
+    ]
+
+    assert chunks[0].to_dict(as_series=False) == {
+        "id": [1, 2],
+        "name": ["a", "b"],
+    }
+
+    assert chunks[1].to_dict(as_series=False) == {
+        "id": [3],
+        "name": ["c"],
+    }
+
+
+def test_read_sql_to_polars_by_chunks_across_cursor_pages():
+    es = _mock_es(
+        [
+            {
+                "columns": COLUMNS,
+                "rows": [[1, "a"], [2, "b"]],
+                "cursor": "cursor_1",
+            },
+            {
+                "rows": [[3, "c"], [4, "d"]],
+                "cursor": None,
+            },
+        ]
+    )
+
+    chunks = list(
+        read_sql_to_polars_by_chunks(
+            es,
+            "SELECT *",
+            chunksize=3,
+        )
+    )
+
+    assert len(chunks) == 2
+
+    assert chunks[0].to_dict(as_series=False) == {
+        "id": [1, 2, 3],
+        "name": ["a", "b", "c"],
+    }
+
+    assert chunks[1].to_dict(as_series=False) == {
+        "id": [4],
+        "name": ["d"],
+    }
+
+
+def test_read_sql_to_polars_by_chunks_clears_cursor():
+    es = _mock_es(
+        [
+            {
+                "columns": COLUMNS,
+                "rows": [[1, "a"]],
+                "cursor": "cursor_1",
+            },
+            {
+                "rows": [[2, "b"]],
+                "cursor": None,
+            },
+        ]
+    )
+
+    list(
+        read_sql_to_polars_by_chunks(
+            es,
+            "SELECT *",
+            chunksize=1,
+        )
+    )
+
+    es.sql.clear_cursor.assert_called_once()
+
+
+def test_read_sql_to_polars_by_chunks_no_cursor_cleanup():
+    es = _mock_es(
+        [
+            {
+                "columns": COLUMNS,
+                "rows": [[1, "a"]],
+            }
+        ]
+    )
+
+    list(
+        read_sql_to_polars_by_chunks(
+            es,
+            "SELECT *",
+            chunksize=1,
+        )
+    )
+
+    es.sql.clear_cursor.assert_not_called()
+
+
+def test_read_sql_to_polars_clears_cursor_on_pagination_error():
+    es = MagicMock()
+
+    es.sql.query.side_effect = [
+        {
+            "columns": COLUMNS,
+            "rows": [[1, "a"]],
+            "cursor": "cursor_1",
+        },
+        RuntimeError("boom"),
+    ]
+
+    with pytest.raises(RuntimeError, match="boom"):
+        read_sql_to_polars(es, "SELECT *")

Review Comment:
   Doesn't seem to test the intended function (I assume that it should have tested `finally...if last_cursor` in the new one)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org