You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/30 17:43:56 UTC
[GitHub] [airflow] dudutch opened a new issue #19900: BigQueryToMsSqlOperator uses deprecated method and doesn't use keyword arguments
dudutch opened a new issue #19900:
URL: https://github.com/apache/airflow/issues/19900
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
This looks to be the same issue as was addressed in [17198](https://github.com/apache/airflow/issues/17198) except for the MsSqlOperator instead of the MySql Operator.
I'm attaching a file that I used to fix the issue
[bigquery_to_mssql.py.txt](https://github.com/apache/airflow/files/7627607/bigquery_to_mssql.py.txt)
.
### Apache Airflow version
2.2.2 (latest released)
### Operating System
Ubuntu 20.04
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### What happened
Tried calling BigQueryToMsSqlOperater using the following
bigquery_to_mssql_survey_response = BigQueryToMsSqlOperator(
task_id="bigquery_to_mssql_response",
gcp_conn_id=GCP_CONNECTION,
mssql_conn_id=MSSQL_CONNECTION,
source_project_dataset_table=f'{TARGET_PROJECT_ID}.{TARGET_DATASET_NAME}.XXXXXXX',
database=MSSQL_DATABASE,
mssql_table=f"{MSSQL_DATABASE}.{MSSQL_LOAD_SCHEMA}.XXXXXXX",
selected_fields="survey_id, response_id, element_name, element_value",
replace=False,
batch_size=100000,
)
### What you expected to happen
Data to transfer form GCP to SQL Server instance
[2021-11-30, 16:46:22 UTC] {taskinstance.py:1241} INFO -
--------------------------------------------------------------------------------
[2021-11-30, 16:46:22 UTC] {taskinstance.py:1242} INFO - Starting attempt 1 of 1
[2021-11-30, 16:46:22 UTC] {taskinstance.py:1243} INFO -
--------------------------------------------------------------------------------
[2021-11-30, 16:46:22 UTC] {taskinstance.py:1262} INFO - Executing <Task(BigQueryToMsSqlOperator): bigquery_to_mssql_survey> on 2021-11-30 16:45:13.181436+00:00
[2021-11-30, 16:46:22 UTC] {standard_task_runner.py:52} INFO - Started process 394851 to run task
[2021-11-30, 16:46:22 UTC] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'XXXXXXXX', 'bigquery_to_mssql_survey', 'manual__2021-11-30T16:45:13.181436+00:00', '--job-id', '38723', '--raw', '--subdir', 'DAGS_FOLDER/whl_XXXXXXXXX_master.py', '--cfg-path', '/tmp/tmpwhre2np1', '--error-file', '/tmp/tmpa9vqlsn2']
[2021-11-30, 16:46:22 UTC] {standard_task_runner.py:77} INFO - Job 38723: Subtask bigquery_to_mssql_survey
[2021-11-30, 16:46:25 UTC] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/models/xcom.py:323: SAWarning: TypeDecorator UtcDateTime(timezone=True) will not produce a cache key because the ``cache_ok`` flag is not set to True. Set this flag to True if this type object's state is safe to use in a cache key, or False to disable this warning.
return query.delete()
[2021-11-30, 16:46:26 UTC] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/models/renderedtifields.py:126: SAWarning: TypeDecorator UtcDateTime(timezone=True) will not produce a cache key because the ``cache_ok`` flag is not set to True. Set this flag to True if this type object's state is safe to use in a cache key, or False to disable this warning.
session.merge(self)
[2021-11-30, 16:46:26 UTC] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/models/renderedtifields.py:162: SAWarning: Coercing Subquery object into a select() for use in IN(); please pass a select() construct explicitly
tuple_(cls.dag_id, cls.task_id, cls.execution_date).notin_(subq1),
[2021-11-30, 16:46:27 UTC] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/models/renderedtifields.py:159: SAWarning: TypeDecorator UtcDateTime(timezone=True) will not produce a cache key because the ``cache_ok`` flag is not set to True. Set this flag to True if this type object's state is safe to use in a cache key, or False to disable this warning.
session.query(cls).filter(
[2021-11-30, 16:46:27 UTC] {taskinstance.py:1427} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=XXXXXXXXXX
AIRFLOW_CTX_TASK_ID=bigquery_to_mssql_survey
AIRFLOW_CTX_EXECUTION_DATE=2021-11-30T16:45:13.181436+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-11-30T16:45:13.181436+00:00
[2021-11-30, 16:46:27 UTC] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/providers/google/cloud/transfers/bigquery_to_mssql.py:118: DeprecationWarning: The bigquery_conn_id parameter has been deprecated. You should pass the gcp_conn_id parameter.
hook = BigQueryHook(
[2021-11-30, 16:46:27 UTC] {crypto.py:82} WARNING - empty cryptography key - values will not be stored encrypted.
[2021-11-30, 16:46:27 UTC] {bigquery_to_mssql.py:125} INFO - Fetching Data from:
[2021-11-30, 16:46:27 UTC] {bigquery_to_mssql.py:126} INFO - Dataset: XXXXXXXXXXX, Table: qualtrics_surveys_full
[2021-11-30, 16:46:27 UTC] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/providers/google/cloud/hooks/bigquery.py:140: DeprecationWarning: This method will be deprecated. Please use `BigQueryHook.get_client` method
warnings.warn(
[2021-11-30, 16:46:27 UTC] {credentials_provider.py:295} INFO - Getting connection using `google.auth.default()` since no key file is defined for hook.
[2021-11-30, 16:46:27 UTC] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/providers/google/cloud/transfers/bigquery_to_mssql.py:156: DeprecationWarning: This method is deprecated. Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_tabledata`
for rows in self._bq_get_data():
[2021-11-30, 16:46:27 UTC] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/providers/google/cloud/hooks/bigquery.py:1301: DeprecationWarning: This method is deprecated. Please use `list_rows`.
warnings.warn("This method is deprecated. Please use `list_rows`.", DeprecationWarning)
[2021-11-30, 16:46:28 UTC] {taskinstance.py:1703} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
result = execute_callable(context=context)
File "/usr/local/lib/python3.8/dist-packages/airflow/providers/google/cloud/transfers/bigquery_to_mssql.py", line 156, in execute
for rows in self._bq_get_data():
File "/usr/local/lib/python3.8/dist-packages/airflow/providers/google/cloud/transfers/bigquery_to_mssql.py", line 132, in _bq_get_data
response = cursor.get_tabledata(
File "/usr/local/lib/python3.8/dist-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 2617, in get_tabledata
return self.hook.get_tabledata(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 1302, in get_tabledata
rows = self.list_rows(dataset_id, table_id, max_results, selected_fields, page_token, start_index)
File "/usr/local/lib/python3.8/dist-packages/airflow/providers/google/common/hooks/base_google.py", line 412, in inner_wrapper
raise AirflowException(
airflow.exceptions.AirflowException: You must use keyword arguments in this methods rather than positional
[2021-11-30, 16:46:28 UTC] {taskinstance.py:1270} INFO - Marking task as FAILED. dag_id=XXXXXXXX, task_id=bigquery_to_mssql_survey, execution_date=20211130T164513, start_date=20211130T164622, end_date=20211130T164628
[2021-11-30, 16:46:28 UTC] {standard_task_runner.py:88} ERROR - Failed to execute job 38723 for task bigquery_to_mssql_survey
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/airflow/task/task_runner/standard_task_runner.py", line 85, in _start_by_fork
args.func(args, dag=self.dag)
File "/usr/local/lib/python3.8/dist-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/cli/commands/task_command.py", line 292, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/usr/local/lib/python3.8/dist-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
_run_raw_task(args, ti)
File "/usr/local/lib/python3.8/dist-packages/airflow/cli/commands/task_command.py", line 180, in _run_raw_task
ti._run_raw_task(
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
result = execute_callable(context=context)
File "/usr/local/lib/python3.8/dist-packages/airflow/providers/google/cloud/transfers/bigquery_to_mssql.py", line 156, in execute
for rows in self._bq_get_data():
File "/usr/local/lib/python3.8/dist-packages/airflow/providers/google/cloud/transfers/bigquery_to_mssql.py", line 132, in _bq_get_data
response = cursor.get_tabledata(
File "/usr/local/lib/python3.8/dist-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 2617, in get_tabledata
return self.hook.get_tabledata(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 1302, in get_tabledata
rows = self.list_rows(dataset_id, table_id, max_results, selected_fields, page_token, start_index)
File "/usr/local/lib/python3.8/dist-packages/airflow/providers/google/common/hooks/base_google.py", line 412, in inner_wrapper
raise AirflowException(
airflow.exceptions.AirflowException: You must use keyword arguments in this methods rather than positional
[2021-11-30, 16:46:29 UTC] {local_task_job.py:154} INFO - Task exited with return code 1
[2021-11-30, 16:46:31 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2021-11-30, 16:46:31 UTC] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/models/dag.py:1076: SADeprecationWarning: Query.value() is deprecated and will be removed in a future release. Please use Query.with_entities() in combination with Query.scalar() (deprecated since: 1.4)
return qry.value(DagModel.is_paused)
[2021-11-30, 16:46:31 UTC] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/models/dag.py:1076: SADeprecationWarning: Query.values() is deprecated and will be removed in a future release. Please use Query.with_entities() (deprecated since: 1.4)
return qry.value(DagModel.is_paused)
### How to reproduce
Try and copy a table from GCP BigQuery to a MSSql Server instance using BigQueryToMsSqlOperator
### Anything else
Appears to be the same issue as [17198](https://github.com/apache/airflow/issues/17198) but for SQL Server not My SQL
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #19900: BigQueryToMsSqlOperator uses deprecated method and doesn't use keyword arguments
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #19900:
URL: https://github.com/apache/airflow/issues/19900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19900: BigQueryToMsSqlOperator uses deprecated method and doesn't use keyword arguments
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19900:
URL: https://github.com/apache/airflow/issues/19900#issuecomment-982869525
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org