You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jroachgolf84 (via GitHub)" <gi...@apache.org> on 2024/02/07 02:47:35 UTC
[I] Inconsistency in S3 Transfer Operators [airflow]
jroachgolf84 opened a new issue, #37213:
URL: https://github.com/apache/airflow/issues/37213
### Description
Standardize similar AWS transfer operators to have similar functionality, as well as similar naming conventions.
This would start with updating transfer operators using S3 as a source or destination to have similar behavior and naming.
### Use case/motivation
There seems to be substantial discrepancies between transfer operators, especially those using S3 as a source or destination. For example, in the `S3ToSFTPOperator`, only a single file can be transferred from the SFTP to the S3 bucket. However, in the `S3ToFTPOperator` an arbitrary number of files can be transferred from S3 to an FTP site. To add to this, the `FTPToS3Operator` only supports a single file being transferred.
These sort of discrepancies exist for a number of transfer operators. In addition to that, parameter names are quite different from operator to operator, which makes using them quite difficult.
This will allow for a more seamless experience for Airflow users, and ensure common functionality across similar tools.
### Related issues
No related issues.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistency in S3 Transfer Operators [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37213:
URL: https://github.com/apache/airflow/issues/37213#issuecomment-3986489854
@bricefotzo We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistency in S3 Transfer Operators [airflow]
Posted by "bricefotzo (via GitHub)" <gi...@apache.org>.
bricefotzo commented on issue #37213:
URL: https://github.com/apache/airflow/issues/37213#issuecomment-1937020179
Hi, I would like to bé assigned on this as my first issue, if possible. @jroachgolf84 would it be ok for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistency in S3 Transfer Operators [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #37213:
URL: https://github.com/apache/airflow/issues/37213#issuecomment-2174229556
@bricefotzo, I'm continuing to work on this issue, I'll let you know if anything changes. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistency in S3 Transfer Operators [airflow]
Posted by "bricefotzo (via GitHub)" <gi...@apache.org>.
bricefotzo commented on issue #37213:
URL: https://github.com/apache/airflow/issues/37213#issuecomment-2175553520
@jroachgolf84 I have added those changes https://github.com/apache/airflow/commit/3da0dcff0779079b3347b685c53e4e7c153ccbb4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistency in S3 Transfer Operators [airflow]
Posted by "bricefotzo (via GitHub)" <gi...@apache.org>.
bricefotzo commented on issue #37213:
URL: https://github.com/apache/airflow/issues/37213#issuecomment-2254506660
Yes, I'll create a PR this week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistency in S3 Transfer Operators [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #37213:
URL: https://github.com/apache/airflow/issues/37213#issuecomment-2174233886
@bricefotzo, were you able to complete this issue? If not, I'm looking forward to working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistency in S3 Transfer Operators [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #37213:
URL: https://github.com/apache/airflow/issues/37213#issuecomment-1931171719
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistency in S3 Transfer Operators [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #37213:
URL: https://github.com/apache/airflow/issues/37213#issuecomment-2254468986
@bricefotzo do you plan to raise a PR for this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistency in S3 Transfer Operators [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck closed issue #37213: Inconsistency in S3 Transfer Operators
URL: https://github.com/apache/airflow/issues/37213
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org