You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/11 14:43:46 UTC
[GitHub] [airflow] hewerthomn opened a new issue, #25676: OracleToOracleOperator fails when source_sql returns more rows than rows_chunk value
hewerthomn opened a new issue, #25676:
URL: https://github.com/apache/airflow/issues/25676
### Apache Airflow Provider(s)
oracle
### Versions of Apache Airflow Providers
apache-airflow-providers-oracle == 3.2.0
### Apache Airflow version
2.3.3
### Operating System
Debian GNU/Linux 11 (bullseye)
### Deployment
Docker-Compose
### Deployment details
_No response_
### What happened
When `source_sql` query returns more or equal rows than `rows_chunk` value, an exception is raised and the transfer process stops, but the first chunk part is inserted.
In some cases it worked fine because the `source_sql` returned a few rows and the default `rows_chuck` value is `5000`.
### What you think should happen instead
Log of an limited query under default `rows_chuck` size:
```
[2022-08-11, 10:11:39 -04] {oracle.py:273} INFO - [TABLE_NAME] inserted 4999 rows
[2022-08-11, 10:11:39 -04] {oracle_to_oracle.py:81} INFO - Total inserted: 4999 rows
[2022-08-11, 10:11:39 -04] {oracle_to_oracle.py:83} INFO - Finished data transfer.
[2022-08-11, 10:11:39 -04] {taskinstance.py:1420} INFO - Marking task as SUCCESS.
```
Log without limit query results and need to chuck:
```
[2022-08-11, 10:07:01 -04] {oracle.py:266} INFO - [TABLE_NAME] inserted 5000 rows
[2022-08-11, 10:07:01 -04] {taskinstance.py:1909} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/oracle/transfers/oracle_to_oracle.py", line 89, in execute
self._execute(src_hook, dest_hook, context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/oracle/transfers/oracle_to_oracle.py", line 78, in _execute
self.destination_table, rows, target_fields=target_fields, commit_every=self.rows_chunk
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/oracle/hooks/oracle.py", line 271, in bulk_insert_rows
cursor.executemany(None, row_chunk)
File "/home/airflow/.local/lib/python3.7/site-packages/oracledb/cursor.py", line 440, in executemany
bool(arraydmlrowcounts))
File "src/oracledb/impl/thin/cursor.pyx", line 132, in oracledb.thin_impl.ThinCursorImpl.executemany
File "src/oracledb/impl/thin/cursor.pyx", line 113, in oracledb.thin_impl.ThinCursorImpl._preprocess_execute
File "/home/airflow/.local/lib/python3.7/site-packages/oracledb/errors.py", line 103, in _raise_err
raise exc_type(_Error(message)) from cause
oracledb.exceptions.DatabaseError: DPY-4010: a bind variable replacement value for placeholder ":1" was not provided
[2022-08-11, 10:07:01 -04] {taskinstance.py:1420} INFO - Marking task as UP_FOR_RETRY.
```
When I changed the `rows_chunk` value to a higher value, it worked:
```
[2022-08-11, 10:12:11 -04] {oracle.py:273} INFO - [TABLE_NAME] inserted 106849 rows
[2022-08-11, 10:12:11 -04] {oracle_to_oracle.py:81} INFO - Total inserted: 106849 rows
[2022-08-11, 10:12:11 -04] {oracle_to_oracle.py:83} INFO - Finished data transfer.
[2022-08-11, 10:12:11 -04] {taskinstance.py:1420} INFO - Marking task as SUCCESS.
```
### How to reproduce
To reproduce this bug the `source_sql` needs to returns more or equal rows than `rows_chunk` value.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #25676: OracleToOracleOperator fails when source_sql returns more rows than rows_chunk value
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #25676:
URL: https://github.com/apache/airflow/issues/25676#issuecomment-1221489130
Does the Sql execute successfully when you use oracledb directly (without Airflow)?
If not then you should report it in https://github.com/oracle/python-oracledb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #25676: OracleToOracleOperator fails when source_sql returns more rows than rows_chunk value
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25676:
URL: https://github.com/apache/airflow/issues/25676#issuecomment-1212085288
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25676: OracleToOracleOperator fails when source_sql returns more rows than rows_chunk value
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25676:
URL: https://github.com/apache/airflow/issues/25676#issuecomment-1235405463
> hi all,
>
> im facing this error DPY-4010: a bind variable replacement value for placeholder ":1" was not provided my job running for oracle to oracler transfer data . currently using apache airflow oracle providers 3.3.0
>
> please advise, currently using airflow 2.3.4 version. it latest version
You can downgrade the oracle provider to fix it. Information on updating providers separately from core are here: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installing-upgrading-downgrading-providers-separately-from-airflow-core
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25676: OracleToOracleOperator fails when source_sql returns more rows than rows_chunk value
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25676:
URL: https://github.com/apache/airflow/issues/25676#issuecomment-1241444728
the common.sql provider has been released today - you can upgrade both oracle and common and it should work.
https://pypi.org/project/apache-airflow-providers-common-sql/1.2.0/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #25676: OracleToOracleOperator fails when source_sql returns more rows than rows_chunk value
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #25676:
URL: https://github.com/apache/airflow/issues/25676#issuecomment-1212324952
This is exception raised from oracledb.
Can you please explain why do you consider this to be a bug in the operator? Could it be bug in the oracledb?
Please add more information so we can be confident the issue is in the operator code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] deen992 commented on issue #25676: OracleToOracleOperator fails when source_sql returns more rows than rows_chunk value
Posted by GitBox <gi...@apache.org>.
deen992 commented on issue #25676:
URL: https://github.com/apache/airflow/issues/25676#issuecomment-1235351642
hi all,
im facing this error DPY-4010: a bind variable replacement value for placeholder ":1" was not provided
my job running for oracle to oracler transfer data .
currently using apache airflow oracle providers 3.3.0
please advise, currently using airflow 2.3.4 version. it latest version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hewerthomn commented on issue #25676: OracleToOracleOperator fails when source_sql returns more rows than rows_chunk value
Posted by GitBox <gi...@apache.org>.
hewerthomn commented on issue #25676:
URL: https://github.com/apache/airflow/issues/25676#issuecomment-1220595646
I think this can be a bug in oracledb because when I used the version 3.0.0, before the change to oracledb, this bug won't happen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hewerthomn commented on issue #25676: OracleToOracleOperator fails when source_sql returns more rows than rows_chunk value
Posted by GitBox <gi...@apache.org>.
hewerthomn commented on issue #25676:
URL: https://github.com/apache/airflow/issues/25676#issuecomment-1222405860
Yes, i created a simple script with the query and executed without errors.
The error happens only chunk is necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #25676: OracleToOracleOperator fails when source_sql returns more rows than rows_chunk value
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #25676:
URL: https://github.com/apache/airflow/issues/25676#issuecomment-1212329441
@lmeerwood maybe you have some input on this as you recently worked on replacing the oracle package in https://github.com/apache/airflow/pull/24311
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hewerthomn closed issue #25676: OracleToOracleOperator fails when source_sql returns more rows than rows_chunk value
Posted by GitBox <gi...@apache.org>.
hewerthomn closed issue #25676: OracleToOracleOperator fails when source_sql returns more rows than rows_chunk value
URL: https://github.com/apache/airflow/issues/25676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org