You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/20 19:22:53 UTC
[GitHub] [airflow] JavierLopezT opened a new pull request #13796: Fix S3ToFTPOperator
JavierLopezT opened a new pull request #13796:
URL: https://github.com/apache/airflow/pull/13796
Recently this operator was merged, and even though the tests was passed, the file that the operator creates in the FTP server is empty. This solves the problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#issuecomment-763901332
[The Workflow run](https://github.com/apache/airflow/actions/runs/499453737) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RosterIn commented on a change in pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
RosterIn commented on a change in pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#discussion_r561293394
##########
File path: airflow/providers/amazon/aws/transfers/s3_to_ftp.py
##########
@@ -27,12 +27,11 @@
class S3ToFTPOperator(BaseOperator):
"""
This operator enables the transferring of files from S3 to a FTP server.
-
:param ftp_conn_id: The ftp connection id. The name or identifier for
establishing a connection to the FTP server.
:type ftp_conn_id: str
- :param ftp_path: The ftp remote path. This is the specified file path for
- uploading file to the FTP server.
+ :param ftp_path: The ftp remote path in where the file will be stored.
+ The desired filename must be specified here.
Review comment:
```suggestion
The desired file name must be specified here.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#issuecomment-763901332
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RosterIn commented on a change in pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
RosterIn commented on a change in pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#discussion_r561293394
##########
File path: airflow/providers/amazon/aws/transfers/s3_to_ftp.py
##########
@@ -27,12 +27,11 @@
class S3ToFTPOperator(BaseOperator):
"""
This operator enables the transferring of files from S3 to a FTP server.
-
:param ftp_conn_id: The ftp connection id. The name or identifier for
establishing a connection to the FTP server.
:type ftp_conn_id: str
- :param ftp_path: The ftp remote path. This is the specified file path for
- uploading file to the FTP server.
+ :param ftp_path: The ftp remote path in where the file will be stored.
+ The desired filename must be specified here.
Review comment:
```suggestion
The desired file name must be specified here.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#issuecomment-773944647
[The Workflow run](https://github.com/apache/airflow/actions/runs/539893935) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#discussion_r561218792
##########
File path: airflow/providers/amazon/aws/transfers/s3_to_ftp.py
##########
@@ -27,12 +27,11 @@
class S3ToFTPOperator(BaseOperator):
"""
This operator enables the transferring of files from S3 to a FTP server.
-
:param ftp_conn_id: The ftp connection id. The name or identifier for
establishing a connection to the FTP server.
:type ftp_conn_id: str
- :param ftp_path: The ftp remote path. This is the specified file path for
- uploading file to the FTP server.
+ :param ftp_path: The ftp remote path in where the file will be stored.
+ The desired filename bust be specified here.
Review comment:
```suggestion
The desired filename must be specified here.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#issuecomment-764874033
[The Workflow run](https://github.com/apache/airflow/actions/runs/501762799) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#issuecomment-833501075
> An easier fix/smaller change:
>
> Add `os.seek(local_tmp_file, os.SEEK_SET, 0)` and keep the rest of the code the same..
With your suggestion I get the error:
`AttributeError: module 'os' has no attribute 'seek'`
I have tried also with
`s3_obj.seek(local_tmp_file, os.SEEK_SET, 0)`
and then I get
AttributeError: 's3.Object' object has no attribute 'seek'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#issuecomment-811813030
[The Workflow run](https://github.com/apache/airflow/actions/runs/708164569) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xinbinhuang commented on pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#issuecomment-821279946
@JavierLopezT can you fix the pylint error thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#issuecomment-859725722
> Please use `local_tmp_file.seek(0)` instead of changing how we download the object from S3, as it makes the change much smaller!
I think I've just uploaded the change you requested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#issuecomment-763901802
[The Workflow run](https://github.com/apache/airflow/actions/runs/499380958) is cancelling this PR. Building image for the PR has been cancelled
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jhtimmins commented on pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
jhtimmins commented on pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#issuecomment-858771132
@ashb can you take another look to see if your requested changes have been addressed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#issuecomment-775255106
[The Workflow run](https://github.com/apache/airflow/actions/runs/548257832) is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#discussion_r625630877
##########
File path: airflow/providers/amazon/aws/transfers/s3_to_ftp.py
##########
@@ -51,8 +56,8 @@ def __init__(
s3_bucket,
s3_key,
ftp_path,
- aws_conn_id='aws_default',
ftp_conn_id='ftp_default',
+ aws_conn_id='aws_default',
Review comment:
Please don't change the order of the parameters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#issuecomment-797003790
[The Workflow run](https://github.com/apache/airflow/actions/runs/643952967) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#discussion_r570702241
##########
File path: airflow/providers/amazon/aws/transfers/s3_to_ftp.py
##########
@@ -27,15 +27,14 @@
class S3ToFTPOperator(BaseOperator):
"""
This operator enables the transferring of files from S3 to a FTP server.
-
:param ftp_conn_id: The ftp connection id. The name or identifier for
establishing a connection to the FTP server.
:type ftp_conn_id: str
- :param ftp_path: The ftp remote path. This is the specified file path for
- uploading file to the FTP server.
+ :param ftp_path: The ftp remote path in where the file will be stored.
+ The desired file name must be specified here.
Review comment:
```suggestion
:param ftp_path: The ftp remote path where the file will be stored, inclusive of filename.
```
##########
File path: airflow/providers/amazon/aws/transfers/s3_to_ftp.py
##########
@@ -27,15 +27,14 @@
class S3ToFTPOperator(BaseOperator):
"""
This operator enables the transferring of files from S3 to a FTP server.
-
:param ftp_conn_id: The ftp connection id. The name or identifier for
establishing a connection to the FTP server.
:type ftp_conn_id: str
- :param ftp_path: The ftp remote path. This is the specified file path for
- uploading file to the FTP server.
+ :param ftp_path: The ftp remote path in where the file will be stored.
+ The desired file name must be specified here.
:type ftp_path: str
:param s3_conn_id: The s3 connection id. The name or identifier for
- establishing a connection to S3.
+ establish a connection to S3.
Review comment:
was correct before
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#discussion_r571460674
##########
File path: airflow/providers/amazon/aws/transfers/s3_to_ftp.py
##########
@@ -27,12 +27,10 @@
class S3ToFTPOperator(BaseOperator):
"""
This operator enables the transferring of files from S3 to a FTP server.
Review comment:
```suggestion
This operator enables the transferring of files from S3 to a FTP server.
```
It's needed to build docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#discussion_r570702241
##########
File path: airflow/providers/amazon/aws/transfers/s3_to_ftp.py
##########
@@ -27,15 +27,14 @@
class S3ToFTPOperator(BaseOperator):
"""
This operator enables the transferring of files from S3 to a FTP server.
-
:param ftp_conn_id: The ftp connection id. The name or identifier for
establishing a connection to the FTP server.
:type ftp_conn_id: str
- :param ftp_path: The ftp remote path. This is the specified file path for
- uploading file to the FTP server.
+ :param ftp_path: The ftp remote path in where the file will be stored.
+ The desired file name must be specified here.
Review comment:
```suggestion
:param ftp_path: The ftp remote path where the file will be stored, inclusive of filename.
```
##########
File path: airflow/providers/amazon/aws/transfers/s3_to_ftp.py
##########
@@ -27,15 +27,14 @@
class S3ToFTPOperator(BaseOperator):
"""
This operator enables the transferring of files from S3 to a FTP server.
-
:param ftp_conn_id: The ftp connection id. The name or identifier for
establishing a connection to the FTP server.
:type ftp_conn_id: str
- :param ftp_path: The ftp remote path. This is the specified file path for
- uploading file to the FTP server.
+ :param ftp_path: The ftp remote path in where the file will be stored.
+ The desired file name must be specified here.
:type ftp_path: str
:param s3_conn_id: The s3 connection id. The name or identifier for
- establishing a connection to S3.
+ establish a connection to S3.
Review comment:
was correct before
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13796: Fix S3ToFTPOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13796:
URL: https://github.com/apache/airflow/pull/13796#issuecomment-763925013
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest master or amend the last commit of the PR, and push it with --force-with-lease.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org