You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dominikhei (via GitHub)" <gi...@apache.org> on 2025/07/05 19:54:32 UTC
[I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
dominikhei opened a new issue, #52920:
URL: https://github.com/apache/airflow/issues/52920
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
_No response_
### Apache Airflow version
3.0.2
### Operating System
linux
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
I was trying to replicate the behavior in #52869 and noticed something interesting.
When executed with `deferable = True` no files are passed to the S3KeySensor by a check_fn, even though they exist in S3. If `deferable = False`, the correct files in the bucket will be passed. I used the following function:
```
def check_key_not_present(files: list, **kwargs) -> bool:
print(f"Files received by check_fn: {files}")
return len(files) == 0
```
These are the two tasks:
```
deferrable_sensor = S3KeySensor(
task_id='deferrable_sensor',
bucket_name='#########################',
bucket_key='scripts/extract-raw-data-job.py',
poke_interval=10,
timeout=60,
check_fn=check_key_not_present,
deferrable=True,
mode="poke",
aws_conn_id='aws_default',
dag=dag,
trigger_rule='none_skipped'
)
non_deferrable_sensor = S3KeySensor(
task_id='non_deferrable_sensor',
bucket_name='######################',
bucket_key='scripts/extract-raw-data-job.py',
poke_interval=10,
timeout=60,
check_fn=check_key_not_present,
deferrable=False,
mode="poke",
aws_conn_id='aws_default',
dag=dag,
trigger_rule='none_skipped'
)
```
Logs of non_deferrable_sensor:
`[2025-07-04, 20:24:09] INFO - Files received by check_fn: [{'Size': 363, 'Key': None}]: chan="stdout": source="task"`
Logs of the deferrable_sensor:
`[2025-07-04, 20:23:53] INFO - Files received by check_fn: []: chan="stdout": source="task"`
After my adjustments:
```INFO - Files received by check_fn: [{'Key': 'xxxxxxxxxxx', 'LastModified': DateTime(2025, 7, 5, 8, 59, 14, tzinfo=Timezone('UTC')), 'ETag': '"9fe1b02003ad4a0a062ff7df14b36d54"', 'ChecksumAlgorithm': ['CRC64NVME'], 'ChecksumType': 'FULL_OBJECT', 'Size': 209, 'StorageClass': 'STANDARD'}]
```
The problem lies here:
```
async def get_files_async(
self,
client: AioBaseClient,
bucket: str,
bucket_keys: str | list[str],
wildcard_match: bool,
delimiter: str | None = "/",
) -> list[Any]:
"""Get a list of files in the bucket."""
keys: list[Any] = []
for key in bucket_keys:
prefix = key
if wildcard_match:
prefix = re.split(r"[\[*?]", key, 1)[0]
paginator = client.get_paginator("list_objects_v2")
params = {
"Bucket": bucket,
"Prefix": prefix,
"Delimiter": delimiter,
}
```
Why dont we check whether `bucket_keys` is a single string or a list? Is this somehow intentional? If `bucket_keys` is a single string: `for key in bucket_keys` will split it into individual chars.
I am opening this issue instead of a direct PR, to validate that this is not actually intended and I am missing something. Because [this test](https://github.com/apache/airflow/blob/main/providers/amazon/tests/unit/amazon/aws/hooks/test_s3.py#L718) makes me question whether this is intended behavior?
### What you think should happen instead
_No response_
### How to reproduce
Create a Dag with two S3 KeySensors, one deferrable and one not and pass a custom check_fn.
### Anything else
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #52920:
URL: https://github.com/apache/airflow/issues/52920#issuecomment-3045719465
@dominikhei, would you like this assigned?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #52920:
URL: https://github.com/apache/airflow/issues/52920#issuecomment-3045877938
I've created a PR for this issue: https://github.com/apache/airflow/pull/52983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #52920:
URL: https://github.com/apache/airflow/issues/52920#issuecomment-3039810302
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #52920: S3KeySensor in deferrable mode splits key into individual chars
URL: https://github.com/apache/airflow/issues/52920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on issue #52920:
URL: https://github.com/apache/airflow/issues/52920#issuecomment-3045394274
As far as I know this is not intentional and most likely a bug, please feel free to create a PR :) Thanks for catching this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
Posted by "dominikhei (via GitHub)" <gi...@apache.org>.
dominikhei commented on issue #52920:
URL: https://github.com/apache/airflow/issues/52920#issuecomment-3045358491
@vincbeck is the splitting of the key in deferrable mode somehow intentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org