kk9087 commented on issue #28751:
URL: https://github.com/apache/airflow/issues/28751#issuecomment-1628732197
Hi,
i am also getting the same issue. Whenever a task is successful, during the teardown of the pod by airflow, the same error deepcopy and pickling comes up.
Airflow Version: 2.6.2
Airflow GCP Connection: Workload Identity
Google Cloud Connection setting: I already had google_cloud_default settings and later also added AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT='google-cloud-platform://' as per https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html. Error still exists
```
[2023-07-07T17:06:13.836+0000] {credentials_provider.py:354} INFO - Getting connection using `google.auth.default()` since no explicit credentials are provided. │
│ Traceback (most recent call last): │
│ File "/home/airflow/.local/bin/airflow", line 8, in <module> │
│ sys.exit(main()) │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 48, in main │
│ args.func(args) │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_config.py", line 52, in command
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_config.py", line 52, in command │
│ return func(*args, **kwargs) │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 112, in wrapper │
│ return f(*args, **kwargs) │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 425, in task_run │
│ task_return_code = _run_task_by_selected_method(args, _dag, ti) │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 203, in _run_task_by_selected_method │
│ return _run_task_by_local_task_job(args, ti) │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 265, in _run_task_by_local_task_job │
│ ret = run_job(job=job_runner.job, execute_callable=job_runner._execute) │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 76, in wrapper │
│ return func(*args, session=session, **kwargs) │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/job.py", line 284, in run_job │
│ return execute_job(job, execute_callable=execute_callable) │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/job.py", line 313, in execute_job │
│ ret = execute_callable() │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job_runner.py", line 189, in _execute │
│ self.handle_task_exit(return_code) │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job_runner.py", line 229, in handle_task_exit │
│ self.task_instance.schedule_downstream_tasks() │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 76, in wrapper │
│ return func(*args, session=session, **kwargs) │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2623, in schedule_downstream_tasks │
│ partial_dag = task.dag.partial_subset( │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2224, in partial_subset │
│ dag.task_dict = { │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2225, in <dictcomp> │
│ t.task_id: _deepcopy_task(t) │
│ File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2222, in _deepcopy_task │
│ return copy.deepcopy(t, memo) │
│ File "/usr/local/lib/python3.8/copy.py", line 153, in deepcopy │
│ y = copier(memo)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1224, in __deepcopy__ │
│ setattr(result, k, copy.deepcopy(v, memo)) │
│ File "/usr/local/lib/python3.8/copy.py", line 161, in deepcopy │
│ rv = reductor(4) │
│ File "/home/airflow/.local/lib/python3.8/site-packages/google/cloud/client/__init__.py", line 194, in __getstate__ │
│ raise PicklingError( │
│ _pickle.PicklingError: Pickling client objects is explicitly not supported.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org