You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/25 03:27:39 UTC
[GitHub] [airflow] AdamPaslawski opened a new pull request #21811: add newer_than parameter to SFTP sensor
AdamPaslawski opened a new pull request #21811:
URL: https://github.com/apache/airflow/pull/21811
closes: #21655
Add a parameter newer_than to the sftp sensor to allow for specifying a datetime for which the file in the SFTP site should be newer than
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21811: add newer_than parameter to SFTP sensor
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21811:
URL: https://github.com/apache/airflow/pull/21811#discussion_r815481666
##########
File path: airflow/providers/sftp/sensors/sftp.py
##########
@@ -54,4 +63,9 @@ def poke(self, context: 'Context') -> bool:
raise e
return False
self.hook.close_conn()
- return True
+ if self.newer_than:
+ _mod_time = datetime.strptime(mod_time, '%Y%m%d%H%M%S')
+ _newer_than = make_naive(self.newer_than) if not is_naive(self.newer_than) else self.newer_than
+ return _newer_than <= _mod_time
Review comment:
I think you shoudl convert all dates to UTC. It's surprisingly hard to find what is the timezone of st_mtime but all signs point at UTC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] AdamPaslawski commented on a change in pull request #21811: add newer_than parameter to SFTP sensor
Posted by GitBox <gi...@apache.org>.
AdamPaslawski commented on a change in pull request #21811:
URL: https://github.com/apache/airflow/pull/21811#discussion_r815491599
##########
File path: airflow/providers/sftp/sensors/sftp.py
##########
@@ -54,4 +63,9 @@ def poke(self, context: 'Context') -> bool:
raise e
return False
self.hook.close_conn()
- return True
+ if self.newer_than:
+ _mod_time = datetime.strptime(mod_time, '%Y%m%d%H%M%S')
+ _newer_than = make_naive(self.newer_than) if not is_naive(self.newer_than) else self.newer_than
+ return _newer_than <= _mod_time
Review comment:
I agree. Assuming the SFTP site is always on the same time-zone as the airflow instance is weaker than assuming UTC.
I'll make that change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] AdamPaslawski closed pull request #21811: add newer_than parameter to SFTP sensor
Posted by GitBox <gi...@apache.org>.
AdamPaslawski closed pull request #21811:
URL: https://github.com/apache/airflow/pull/21811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] AdamPaslawski commented on a change in pull request #21811: add newer_than parameter to SFTP sensor
Posted by GitBox <gi...@apache.org>.
AdamPaslawski commented on a change in pull request #21811:
URL: https://github.com/apache/airflow/pull/21811#discussion_r814745208
##########
File path: airflow/providers/sftp/sensors/sftp.py
##########
@@ -54,4 +63,9 @@ def poke(self, context: 'Context') -> bool:
raise e
return False
self.hook.close_conn()
- return True
+ if self.newer_than:
+ _mod_time = datetime.strptime(mod_time, '%Y%m%d%H%M%S')
+ _newer_than = make_naive(self.newer_than) if not is_naive(self.newer_than) else self.newer_than
+ return _newer_than <= _mod_time
Review comment:
I decided to assume a naive comparison and allow the sensor to be configured with an offset aware or unaware newer_than parameter, thoughts on this @eladkal ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org