[I] SQLInsertRowsOperator crashes when it tries to log length on non materialized collection [airflow]

["dabla (via GitHub)" <gi...@apache.org>]

dabla opened a new issue, #63344:
URL: https://github.com/apache/airflow/issues/63344

   ### Apache Airflow version
   
   3.1.7
   
   ### If "Other Airflow 3 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   When passing a non-materialized iterable (e.g., a generator) to the rows parameter of the SQLInsertRowsOperator, the operator crashes during execution.
   
   This happens because the operator attempts to log the length of the rows parameter using len(rows). Non-materialized iterables do not implement __len__, which results in a TypeError.
   
   ### What you think should happen instead?
   
   The operator should not assume that rows has a defined length and therefore should avoid calling len(rows).
   
   Additionally, the existing tests for SQLInsertRowsOperator currently pass a list to the rows parameter. These tests should be updated to use a non-materialized iterable (such as a generator) to ensure this behavior is properly supported and to prevent regressions in the future.
   
   ### How to reproduce
   
   ```
       def rows_gen():
           yield {"index": 1, "name": "Stallone", "firstname": "Sylvester", "age": 78}
           yield {"index": 2, "name": "Statham", "firstname": "Jason", "age": 57}
           yield {"index": 3, "name": "Li", "firstname": "Jet", "age": 61}
           yield {"index": 4, "name": "Lundgren", "firstname": "Dolph", "age": 66}
           yield {"index": 5, "name": "Norris", "firstname": "Chuck", "age": 84}
   
       operator = SQLInsertRowsOperator(
           task_id="test_task",
           conn_id="default_conn",
           schema="hollywood",
           table_name="actors",
           rows=rows_gen(),  # generator (no __len__)
           rows_processor=lambda rows, **context: [tuple(row.values()) for row in rows],
       )
   ```
   
   ### Operating System
   
   Linux
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-common-sql 1.32.0
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] SQLInsertRowsOperator crashes when it tries to log length on non materialized collection [airflow]

["dabla (via GitHub)" <gi...@apache.org>]

dabla closed issue #63344: SQLInsertRowsOperator crashes when it tries to log length on non materialized collection
URL: https://github.com/apache/airflow/issues/63344


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] SQLInsertRowsOperator crashes when it tries to log length on non materialized collection [airflow]

["dabla (via GitHub)" <gi...@apache.org>]

dabla commented on issue #63344:
URL: https://github.com/apache/airflow/issues/63344#issuecomment-4045229795

   Fixed in [PR #63346](https://github.com/apache/airflow/pull/63346)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org