You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "hejnal (via GitHub)" <gi...@apache.org> on 2024/02/20 14:06:33 UTC
[I] GCSObjectsWithPrefixExistenceSensor in deferrable mode does not return matched objects in the xcom in the first poke (before entering deferrable mode) [airflow]
hejnal opened a new issue, #37561:
URL: https://github.com/apache/airflow/issues/37561
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
apache-airflow-providers-google==10.14.0
### Apache Airflow version
2.6.3
### Operating System
MacOS
### Deployment
Other
### Deployment details
Local deployment of Airflow (with RabbitMQ queue). Connecting to GCS through custom connection with a SA.
### What happened
On the first execution - before entering the deferrable mode, the sensor (in deferrable=True mode) the sensor picks up the files on GCS, turn green, but it does not return the object in the XCom interface.
It works fine if the file has arrived after the sensor entered in the deferrable mode and it is picked up by the triggerer.
The line that might cause this issue is this one:
self.poke(context=context)
../site-packages/airflow/providers/google/cloud/sensors/gcs.py
if not self.poke(context=context):
self.defer(
timeout=timedelta(seconds=self.timeout),
trigger=GCSUploadSessionTrigger(
bucket=self.bucket,
prefix=self.prefix,
poke_interval=self.poke_interval,
google_cloud_conn_id=self.google_cloud_conn_id,
inactivity_period=self.inactivity_period,
min_objects=self.min_objects,
previous_objects=self.previous_objects,
allow_delete=self.allow_delete,
hook_params=hook_params,
),
method_name="execute_complete",
)
It might be related to:
https://github.com/apache/airflow/commit/f89d7b98487d993387ebea4af526fe20204ce02a#diff-03d898c7c92dfd56502de1a64ff072fb8e88b57b3eac7cec7fb5ee14eb4b47d2
### What you think should happen instead
_No response_
### How to reproduce
Put the file in the bucket first.
Run the DAG code, on the first execution the sensor picks up the file, but does not return it in the Xcom.
wait_for_data_op = GCSObjectsWithPrefixExistenceSensor(
task_id="00.wait_for_data_files",
google_cloud_conn_id=GOOGLE_CLOUD_LANDING_BUCKET_CONN_ID,
bucket=landing_bucket,
prefix=f"{source_objects_prefix}{schema_version_suffix}" +
("" if not source_config.enforce_logical_date_detection else
"_{{ ds_nodash }}"),
deferrable=True,
)
The downstream tasks fail because of that:
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GCSObjectsWithPrefixExistenceSensor in deferrable mode does not return matched objects in the xcom in the first poke (before entering deferrable mode) [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #37561:
URL: https://github.com/apache/airflow/issues/37561#issuecomment-1954288628
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GCSObjectsWithPrefixExistenceSensor in deferrable mode does not return matched objects in the xcom in the first poke (before entering deferrable mode) [airflow]
Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro closed issue #37561: GCSObjectsWithPrefixExistenceSensor in deferrable mode does not return matched objects in the xcom in the first poke (before entering deferrable mode)
URL: https://github.com/apache/airflow/issues/37561
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org