You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/10/10 16:48:46 UTC
[GitHub] [airflow] RachitSharma2001 opened a new pull request, #26974: Add FTP Operator
RachitSharma2001 opened a new pull request, #26974:
URL: https://github.com/apache/airflow/pull/26974
The corresponding issue: #26531
I have added the `FTPOperator`, which allows for writing to files in FTP and downloading them to the local directory.
I have also added tests in the tests/providers/ftp/operators directory. In addition, I modified the scripts/docker/entrypoint_ci.sh file to pull up a local FTP server in ci in order to perform these tests. I have made sure to follow the suggested format for adding new airflow operators from [here](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst).
After this PR is approved and merged, I will add the `FTPSOperator` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1020949454
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
Review Comment:
I think we want the example dag to be in AIP-47 format and under system test folder?
@potiuk or are we converting them seperatly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
o-nikolas commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1339730791
> NICE!. I know @eladkal had previously a doubt whether those shoudl not be local_to or _to_local set of transfer operators, but I personaly do not see the need for consistency (and I consider the `local_*` a bit strange now when I think of it.
>
> WDYT Everyone.?
I don't feel too strongly either way, I'm personally happy with the current name :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1009835749
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
my comment is on the whole operator
check for example https://github.com/apache/airflow/blob/219df18414185e426aab9bb8f297785d6f54f6a5/airflow/providers/microsoft/azure/transfers/local_to_wasb.py#L29
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1009812933
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
Do you mean that the `local_filepath` should not be set to `/tmp/filepath`? I was looking at the example given for SFTP put [here](https://airflow.apache.org/docs/apache-airflow-providers-sftp/stable/_api/airflow/providers/sftp/operators/sftp/index.html), and it seems that they put a similar path. Do you mean that the `local_filepath` should instead be something like "relative/path/to/file" like from [here](https://airflow.apache.org/docs/apache-airflow-providers-amazon/3.3.0/operators/transfer/local_to_s3.html)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1020802540
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
I have updated the operator to reflect this name change to `FTPFileTransmitOperator`. Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
o-nikolas commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r996459193
##########
airflow/providers/ftp/operators/ftp.py:
##########
@@ -0,0 +1,135 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains FTP operator."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Any, Sequence
+
+from airflow.compat.functools import cached_property
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.ftp.hooks.ftp import FTPHook
+
+
+class FTPOperation:
+ """Operation that can be used with FTP"""
+
+ PUT = 'put'
+ GET = 'get'
+
+
+class FTPOperator(BaseOperator):
+ """
+ FTPOperator for transferring files from remote host to local or vice a versa.
+ This operator uses an FTPHook to open ftp transport channel that serve as basis
+ for file transfer.
+
+ :param ftp_conn_id: :ref:`ftp connection id<howto/connection:ftp>`
+ from airflow Connections.
+ :param local_filepath: local file path to get or put. (templated)
+ :param remote_filepath: remote file path to get or put. (templated)
+ :param operation: specify operation 'get' or 'put', defaults to put
+ :param create_intermediate_dirs: create missing intermediate directories when
+ copying from remote to local and vice-versa. Default is False.
+
+ Example: The following task would copy ``file.txt`` to the remote host
+ at ``/tmp/tmp1/tmp2/`` while creating ``tmp``,``tmp1`` and ``tmp2`` if they
+ don't exist. If the parameter is not passed it would error as the directory
+ does not exist. ::
+
+ put_file = FTPOperator(
+ task_id="test_ftp",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/file.txt",
+ remote_filepath="/tmp/tmp1/tmp2/file.txt",
+ operation="put",
+ create_intermediate_dirs=True,
+ dag=dag
+ )
+
+ """
+
+ template_fields: Sequence[str] = ('local_filepath', 'remote_filepath')
+
+ def __init__(
+ self,
+ *,
+ ftp_conn_id: str,
Review Comment:
Use your new `default_ftp` conn as a default here?
##########
docs/apache-airflow-providers-ftp/operators/index.rst:
##########
@@ -0,0 +1,64 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/operator:FTPOperator:
+
+FTPOperator
+===============
+
+
+Use the FTPOperator to get or
+pull files to/from an FTP server.
+
+Using the Operator
+^^^^^^^^^^^^^^^^^^
+
+| **ftp_conn_id**: id referencing an FTP Connection **str**
+| **local_filepath**: local file path to get or put. **str**
+| **remote_filepath**: remote file path to get or put. **str**
+| **operation**: specify operation 'get' or 'put' **str** - Default: **PUT**
+| **create_intermediate_dirs**: create missing intermediate directories when copying from remote to local and vice-versa. **bool** - Default: **False**
+|
+
+Example: The following task would copy ``file.txt`` to the remote host
+at ``/tmp/tmp1/tmp2/`` while creating ``tmp``, ``tmp1`` and ``tmp2`` if they
+don't exist. If the ``create_immediate_dirs`` parameter is not passed it would error as the directory
+does not exist. ::
+
+ put_file = FTPOperator(
Review Comment:
You could consider adding an example dag. It's something users can use to get started using your operator and you can link to the example usage of the task from the example, which again helps to keep the docs up to date. But not all provider packages use them.
[Here is an example example-dag](https://github.com/apache/airflow/blob/providers-amazon/3.3.0/airflow/providers/amazon/aws/example_dags/example_ftp_to_s3.py)
[And here is the doc linking to a code snippets from that example](https://airflow.apache.org/docs/apache-airflow-providers-amazon/3.3.0/operators/transfer/ftp_to_s3.html#ftp-to-amazon-s3)
##########
airflow/providers/ftp/operators/ftp.py:
##########
@@ -0,0 +1,135 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains FTP operator."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Any, Sequence
+
+from airflow.compat.functools import cached_property
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.ftp.hooks.ftp import FTPHook
+
+
+class FTPOperation:
+ """Operation that can be used with FTP"""
+
+ PUT = 'put'
+ GET = 'get'
+
+
+class FTPOperator(BaseOperator):
+ """
+ FTPOperator for transferring files from remote host to local or vice a versa.
+ This operator uses an FTPHook to open ftp transport channel that serve as basis
+ for file transfer.
+
+ :param ftp_conn_id: :ref:`ftp connection id<howto/connection:ftp>`
+ from airflow Connections.
+ :param local_filepath: local file path to get or put. (templated)
+ :param remote_filepath: remote file path to get or put. (templated)
+ :param operation: specify operation 'get' or 'put', defaults to put
+ :param create_intermediate_dirs: create missing intermediate directories when
+ copying from remote to local and vice-versa. Default is False.
+
+ Example: The following task would copy ``file.txt`` to the remote host
+ at ``/tmp/tmp1/tmp2/`` while creating ``tmp``,``tmp1`` and ``tmp2`` if they
+ don't exist. If the parameter is not passed it would error as the directory
Review Comment:
```suggestion
at ``/tmp/tmp1/tmp2/`` while creating ``tmp``,``tmp1`` and ``tmp2`` if they
don't exist. If the ``create_intermediate_dirs`` parameter is not passed it would error as the directory
```
##########
airflow/providers/ftp/operators/ftp.py:
##########
@@ -0,0 +1,135 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains FTP operator."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Any, Sequence
+
+from airflow.compat.functools import cached_property
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.ftp.hooks.ftp import FTPHook
+
+
+class FTPOperation:
+ """Operation that can be used with FTP"""
+
+ PUT = 'put'
+ GET = 'get'
+
+
+class FTPOperator(BaseOperator):
+ """
+ FTPOperator for transferring files from remote host to local or vice a versa.
+ This operator uses an FTPHook to open ftp transport channel that serve as basis
+ for file transfer.
+
+ :param ftp_conn_id: :ref:`ftp connection id<howto/connection:ftp>`
+ from airflow Connections.
+ :param local_filepath: local file path to get or put. (templated)
+ :param remote_filepath: remote file path to get or put. (templated)
+ :param operation: specify operation 'get' or 'put', defaults to put
+ :param create_intermediate_dirs: create missing intermediate directories when
+ copying from remote to local and vice-versa. Default is False.
+
+ Example: The following task would copy ``file.txt`` to the remote host
+ at ``/tmp/tmp1/tmp2/`` while creating ``tmp``,``tmp1`` and ``tmp2`` if they
+ don't exist. If the parameter is not passed it would error as the directory
+ does not exist. ::
+
+ put_file = FTPOperator(
+ task_id="test_ftp",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/file.txt",
+ remote_filepath="/tmp/tmp1/tmp2/file.txt",
+ operation="put",
+ create_intermediate_dirs=True,
+ dag=dag
+ )
+
+ """
+
+ template_fields: Sequence[str] = ('local_filepath', 'remote_filepath')
+
+ def __init__(
+ self,
+ *,
+ ftp_conn_id: str,
+ local_filepath: str | list[str],
+ remote_filepath: str | list[str],
+ operation: str = FTPOperation.PUT,
+ create_intermediate_dirs: bool = False,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.ftp_conn_id = ftp_conn_id
+ self.operation = operation
+ self.create_intermediate_dirs = create_intermediate_dirs
+
+ self.local_filepath_was_str = False
+ if isinstance(local_filepath, str):
+ self.local_filepath = [local_filepath]
+ else:
+ self.local_filepath = local_filepath
+
+ if isinstance(remote_filepath, str):
+ self.remote_filepath = [remote_filepath]
+ else:
+ self.remote_filepath = remote_filepath
+
+ if len(self.local_filepath) != len(self.remote_filepath):
+ raise ValueError(
+ f'{len(self.local_filepath)} paths in local_filepath '
+ f'!= {len(self.remote_filepath)} paths in remote_filepath'
+ )
+
+ if self.operation.lower() not in [FTPOperation.GET, FTPOperation.PUT]:
+ raise TypeError(
+ f"Unsupported operation value {self.operation}, "
+ f"expected {FTPOperation.GET} or {FTPOperation.PUT}."
+ )
+
+ @cached_property
+ def hook(self) -> FTPHook:
+ """Create and return an FTPHook."""
+ return FTPHook(ftp_conn_id=self.ftp_conn_id)
+
+ def execute(self, context: Any) -> str | list[str] | None:
+ file_msg = None
+ try:
+ for local_filepath, remote_filepath in zip(self.local_filepath, self.remote_filepath):
+ if self.operation.lower() == FTPOperation.GET:
+ local_folder = os.path.dirname(local_filepath)
+ if self.create_intermediate_dirs:
+ Path(local_folder).mkdir(parents=True, exist_ok=True)
+ file_msg = f"from {remote_filepath} to {local_filepath}"
+ self.log.info("Starting to transfer %s", file_msg)
+ self.hook.retrieve_file(remote_filepath, local_filepath)
+ else:
+ remote_folder = os.path.dirname(remote_filepath)
+ if self.create_intermediate_dirs:
+ self.hook.create_directory(remote_folder)
+ file_msg = f"from {local_filepath} to {remote_filepath}"
+ self.log.info("Starting to transfer file %s", file_msg)
+ self.hook.store_file(remote_filepath, local_filepath)
+ except Exception as e:
+ raise AirflowException(f"Error while transferring {file_msg}, error: {str(e)}")
Review Comment:
Minor nit:
Catching general `Exception` is a slight code-smell (most IDEs will yell at you for doing it, for exampole) and I'm not sure throwing a new `AirflowException` adds much here, you'll get the `During handling of the above exception, another exception occurred:` double exception output which I find just confuses a lot of people if the second exception isn't actually a problem.
If you just want to add more context you can log an error message in a `finally` block and then just let the original exception bubble up.
P.S. You could also use the py3 syntax `raise <new exception> from <old exception>` but you still get the double stacked exception output in that case, but it's now logged as `The above exception was the direct cause of the following exception:` which I still think causes more confusion than it solves.
##########
airflow/providers/ftp/operators/ftp.py:
##########
@@ -0,0 +1,135 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains FTP operator."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Any, Sequence
+
+from airflow.compat.functools import cached_property
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.ftp.hooks.ftp import FTPHook
+
+
+class FTPOperation:
+ """Operation that can be used with FTP"""
+
+ PUT = 'put'
+ GET = 'get'
+
+
+class FTPOperator(BaseOperator):
+ """
+ FTPOperator for transferring files from remote host to local or vice a versa.
+ This operator uses an FTPHook to open ftp transport channel that serve as basis
+ for file transfer.
+
+ :param ftp_conn_id: :ref:`ftp connection id<howto/connection:ftp>`
+ from airflow Connections.
+ :param local_filepath: local file path to get or put. (templated)
+ :param remote_filepath: remote file path to get or put. (templated)
+ :param operation: specify operation 'get' or 'put', defaults to put
+ :param create_intermediate_dirs: create missing intermediate directories when
+ copying from remote to local and vice-versa. Default is False.
+
+ Example: The following task would copy ``file.txt`` to the remote host
+ at ``/tmp/tmp1/tmp2/`` while creating ``tmp``,``tmp1`` and ``tmp2`` if they
+ don't exist. If the parameter is not passed it would error as the directory
+ does not exist. ::
+
+ put_file = FTPOperator(
+ task_id="test_ftp",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/file.txt",
+ remote_filepath="/tmp/tmp1/tmp2/file.txt",
+ operation="put",
+ create_intermediate_dirs=True,
+ dag=dag
+ )
+
+ """
+
+ template_fields: Sequence[str] = ('local_filepath', 'remote_filepath')
+
+ def __init__(
+ self,
+ *,
+ ftp_conn_id: str,
+ local_filepath: str | list[str],
+ remote_filepath: str | list[str],
+ operation: str = FTPOperation.PUT,
+ create_intermediate_dirs: bool = False,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.ftp_conn_id = ftp_conn_id
+ self.operation = operation
+ self.create_intermediate_dirs = create_intermediate_dirs
+
+ self.local_filepath_was_str = False
Review Comment:
This looks unused?
##########
docs/apache-airflow-providers-ftp/operators/index.rst:
##########
@@ -0,0 +1,64 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/operator:FTPOperator:
+
+FTPOperator
+===============
+
+
+Use the FTPOperator to get or
+pull files to/from an FTP server.
+
+Using the Operator
+^^^^^^^^^^^^^^^^^^
+
+| **ftp_conn_id**: id referencing an FTP Connection **str**
+| **local_filepath**: local file path to get or put. **str**
+| **remote_filepath**: remote file path to get or put. **str**
+| **operation**: specify operation 'get' or 'put' **str** - Default: **PUT**
+| **create_intermediate_dirs**: create missing intermediate directories when copying from remote to local and vice-versa. **bool** - Default: **False**
Review Comment:
Just link to the class using `:class:` (search around for examples in `airflow/docs` I can't link one easily since Github will just render the rst and not deep link to the code line) instead of hardcoding the current set of arguments here, this will quickly become out of date/sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1292538043
@potiuk @eladkal @kaxil I have fixed the static check test failure from the previously ran workflow. When you have the time, could you approve the workflow for my latest commit? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1020741355
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
Yeah i think its better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal merged pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
eladkal merged PR #26974:
URL: https://github.com/apache/airflow/pull/26974
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1287889969
@o-nikolas Thank you for your response! I have rebased with main.
@eladkal @kaxil When you have time, could you approve the workflow? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1024938999
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
@RachitSharma2001 can you modify the example dag to AIP-47 format?
see many PRs for reference https://github.com/apache/airflow/pulls?q=is%3Aopen+is%3Apr+label%3Aarea%3Asystem-tests
The example DAG needs to be in
`tests/system/providers/ftp`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1020802540
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
I have updated the operator to reflect this name change to `FTPFileTransfer`. Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1339716055
@o-nikolas Thank you! Although, it seems that the tests were reran and now all the tests are passing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1279788434
@vincbeck I have added documentation to docs/apache-airflow-providers-ftp/operators ([here](https://github.com/RachitSharma2001/airflow/blob/main/docs/apache-airflow-providers-ftp/operators/index.rst)) that specifies how to use the `FTPOperator`. Let me know if these changes look good and if there is anything else you think should be added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1296326259
Hi everyone, I was wondering if I could get some additional reviews on this PR? No rush, but was just wondering if these changes look good or if there is anything else that should be added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1328248870
@RachitSharma2001 - How about changing to AIP-47 and rebasing that one ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1273582305
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r992640561
##########
airflow/utils/db.py:
##########
@@ -308,6 +308,18 @@ def create_default_connections(session: Session = NEW_SESSION):
),
session,
)
+ merge_conn(
+ Connection(
+ conn_id="ftp_default",
+ conn_type="ftp",
+ host="localhost",
Review Comment:
Added an ftp_default connection, as it previously did not exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r996475195
##########
docs/apache-airflow-providers-ftp/operators/index.rst:
##########
@@ -0,0 +1,64 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/operator:FTPOperator:
+
+FTPOperator
+===============
+
+
+Use the FTPOperator to get or
+pull files to/from an FTP server.
+
+Using the Operator
+^^^^^^^^^^^^^^^^^^
+
+| **ftp_conn_id**: id referencing an FTP Connection **str**
+| **local_filepath**: local file path to get or put. **str**
+| **remote_filepath**: remote file path to get or put. **str**
+| **operation**: specify operation 'get' or 'put' **str** - Default: **PUT**
+| **create_intermediate_dirs**: create missing intermediate directories when copying from remote to local and vice-versa. **bool** - Default: **False**
+|
+
+Example: The following task would copy ``file.txt`` to the remote host
+at ``/tmp/tmp1/tmp2/`` while creating ``tmp``, ``tmp1`` and ``tmp2`` if they
+don't exist. If the ``create_immediate_dirs`` parameter is not passed it would error as the directory
+does not exist. ::
+
+ put_file = FTPOperator(
Review Comment:
Thank you for the suggestion. I have added an example dag ([here](https://github.com/RachitSharma2001/airflow/tree/main/airflow/providers/ftp/example_dags)) and have linked to this example in the docs ([here](https://github.com/RachitSharma2001/airflow/blob/main/docs/apache-airflow-providers-ftp/operators/index.rst#id1)). Let me know if I did something wrong or if there is any additional things I need to do in terms of linking the example dags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1284284170
@o-nikolas Sorry about that. I have fixed those failures. I was wondering if I should first wait to get the workflow approved first before rebasing, as if I rebase now, my branch may quickly become out of date again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1032689949
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
It should be, I think. All our examples should bin AIP-47 compatible form so one day we can automate running them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1010684836
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
I see, so do you mean that rather than defining the FTP Operator within `airflow.providers.ftp.operators`, instead define them in a class within the `airflow.providers.ftp.transfers` folder? Looking through examples of operators defined in transfers folder, like for the microsoft azure transfer operators, it seems like many of them are unidirectional in the sense that they only upload files from local to a remote server, or only download files from a remote to a local. I was thinking that maybe it would make more sense to make FTPOperator a pure operator rather than a transfer operator since it can do bidirectional file movements and the direction is determined by passed-in parameters from the user, similar to how the SFTPOperator is defined [here](https://github.com/apache/airflow/tree/219df18414185e426aab9bb8f297785d6f54f6a5/airflow/providers/sftp/operators).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1024938999
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
@RachitSharma2001 can you modify the example dag to AIP-47 format?
see many PRs for reference https://github.com/apache/airflow/pulls?q=is%3Aopen+is%3Apr+label%3Aarea%3Asystem-tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1329447788
@potiuk Sorry for late reply, got back from vacation today. That sounds good, I will change to AIP-47 and rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
vincbeck commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r994984846
##########
airflow/providers/ftp/operators/ftp.py:
##########
@@ -0,0 +1,146 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains FTP operator."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Any, Sequence
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.ftp.hooks.ftp import FTPHook
+
+
+class FTPOperation:
+ """Operation that can be used with FTP"""
+
+ PUT = 'put'
+ GET = 'get'
+
+
+class FTPOperator(BaseOperator):
+ """
+ FTPOperator for transferring files from remote host to local or vice a versa.
+ This operator uses ftp_hook to open ftp transport channel that serve as basis
+ for file transfer.
+
+ :param ftp_conn_id: :ref:`ftp connection id<howto/connection:ftp>`
+ from airflow Connections.
+ :param ftp_hook: predefined FTPHook to use
+ Either `ftp_hook` or `ftp_conn_id` needs to be provided.
+ :param local_filepath: local file path to get or put. (templated)
+ :param remote_filepath: remote file path to get or put. (templated)
+ :param operation: specify operation 'get' or 'put', defaults to put
+ :param create_intermediate_dirs: create missing intermediate directories when
+ copying from remote to local and vice-versa. Default is False.
+
+ Example: The following task would copy ``file.txt`` to the remote host
+ at ``/tmp/tmp1/tmp2/`` while creating ``tmp``,``tmp1`` and ``tmp2`` if they
+ don't exist. If the parameter is not passed it would error as the directory
+ does not exist. ::
+
+ put_file = FTPOperator(
+ task_id="test_ftp",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/file.txt",
+ remote_filepath="/tmp/tmp1/tmp2/file.txt",
+ operation="put",
+ create_intermediate_dirs=True,
+ dag=dag
+ )
+
+ """
+
+ template_fields: Sequence[str] = ('local_filepath', 'remote_filepath')
+
+ def __init__(
+ self,
+ *,
+ ftp_hook: FTPHook | None = None,
+ ftp_conn_id: str | None = None,
+ local_filepath: str | list[str],
+ remote_filepath: str | list[str],
+ operation: str = FTPOperation.PUT,
+ create_intermediate_dirs: bool = False,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.ftp_hook = ftp_hook
+ self.ftp_conn_id = ftp_conn_id
+ self.operation = operation
+ self.create_intermediate_dirs = create_intermediate_dirs
+
+ self.local_filepath_was_str = False
+ if isinstance(local_filepath, str):
+ self.local_filepath = [local_filepath]
+ self.local_filepath_was_str = True
+ else:
+ self.local_filepath = local_filepath
+
+ if isinstance(remote_filepath, str):
+ self.remote_filepath = [remote_filepath]
+ else:
+ self.remote_filepath = remote_filepath
+
+ if len(self.local_filepath) != len(self.remote_filepath):
+ raise ValueError(
+ f'{len(self.local_filepath)} paths in local_filepath '
+ f'!= {len(self.remote_filepath)} paths in remote_filepath'
+ )
+
+ if not (self.operation.lower() == FTPOperation.GET or self.operation.lower() == FTPOperation.PUT):
+ raise TypeError(
+ f"Unsupported operation value {self.operation}, "
+ f"expected {FTPOperation.GET} or {FTPOperation.PUT}."
+ )
+
+ def execute(self, context: Any) -> str | list[str] | None:
+ file_msg = None
+ try:
+ if self.ftp_conn_id and not self.ftp_hook:
+ self.log.info('ftp_hook not provided or invalid. Trying ftp_conn_id to create FTPHook.')
+ self.ftp_hook = FTPHook(ftp_conn_id=self.ftp_conn_id)
+
+ if self.ftp_hook and not isinstance(self.ftp_hook, FTPHook):
+ self.log.info('ftp_hook not provided or invalid. Trying ftp_conn_id to create FTPHook.')
+ self.ftp_hook = FTPHook(ftp_conn_id=self.ftp_conn_id)
+
+ if not self.ftp_hook:
+ raise AirflowException("Cannot operate without ftp_hook or ftp_conn_id.")
+
+ for local_filepath, remote_filepath in zip(self.local_filepath, self.remote_filepath):
+ if self.operation.lower() == FTPOperation.GET:
+ local_folder = os.path.dirname(local_filepath)
+ if self.create_intermediate_dirs:
+ Path(local_folder).mkdir(parents=True, exist_ok=True)
+ file_msg = f"from {remote_filepath} to {local_filepath}"
+ self.log.info("Starting to transfer %s", file_msg)
+ self.ftp_hook.retrieve_file(remote_filepath, local_filepath)
+ else:
+ remote_folder = os.path.dirname(remote_filepath)
+ if self.create_intermediate_dirs:
+ self.ftp_hook.create_directory(remote_folder)
+ file_msg = f"from {local_filepath} to {remote_filepath}"
+ self.log.info("Starting to transfer file %s", file_msg)
+ self.ftp_hook.store_file(remote_filepath, local_filepath)
+
+ except Exception as e:
+ raise AirflowException(f"Error while transferring {file_msg}, error: {str(e)}")
+
+ return self.local_filepath[0] if self.local_filepath_was_str else self.local_filepath
Review Comment:
For simplicity purposes I would return the list `self.local_filepath` regardless of the input. That way you also get rid of `local_filepath_was_str `
##########
airflow/providers/ftp/operators/ftp.py:
##########
@@ -0,0 +1,146 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains FTP operator."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Any, Sequence
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.ftp.hooks.ftp import FTPHook
+
+
+class FTPOperation:
+ """Operation that can be used with FTP"""
+
+ PUT = 'put'
+ GET = 'get'
+
+
+class FTPOperator(BaseOperator):
+ """
+ FTPOperator for transferring files from remote host to local or vice a versa.
+ This operator uses ftp_hook to open ftp transport channel that serve as basis
+ for file transfer.
+
+ :param ftp_conn_id: :ref:`ftp connection id<howto/connection:ftp>`
+ from airflow Connections.
+ :param ftp_hook: predefined FTPHook to use
+ Either `ftp_hook` or `ftp_conn_id` needs to be provided.
+ :param local_filepath: local file path to get or put. (templated)
+ :param remote_filepath: remote file path to get or put. (templated)
+ :param operation: specify operation 'get' or 'put', defaults to put
+ :param create_intermediate_dirs: create missing intermediate directories when
+ copying from remote to local and vice-versa. Default is False.
+
+ Example: The following task would copy ``file.txt`` to the remote host
+ at ``/tmp/tmp1/tmp2/`` while creating ``tmp``,``tmp1`` and ``tmp2`` if they
+ don't exist. If the parameter is not passed it would error as the directory
+ does not exist. ::
+
+ put_file = FTPOperator(
+ task_id="test_ftp",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/file.txt",
+ remote_filepath="/tmp/tmp1/tmp2/file.txt",
+ operation="put",
+ create_intermediate_dirs=True,
+ dag=dag
+ )
+
+ """
+
+ template_fields: Sequence[str] = ('local_filepath', 'remote_filepath')
+
+ def __init__(
+ self,
+ *,
+ ftp_hook: FTPHook | None = None,
Review Comment:
```suggestion
```
##########
airflow/providers/ftp/operators/ftp.py:
##########
@@ -0,0 +1,146 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains FTP operator."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Any, Sequence
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.ftp.hooks.ftp import FTPHook
+
+
+class FTPOperation:
+ """Operation that can be used with FTP"""
+
+ PUT = 'put'
+ GET = 'get'
+
+
+class FTPOperator(BaseOperator):
+ """
+ FTPOperator for transferring files from remote host to local or vice a versa.
+ This operator uses ftp_hook to open ftp transport channel that serve as basis
+ for file transfer.
+
+ :param ftp_conn_id: :ref:`ftp connection id<howto/connection:ftp>`
+ from airflow Connections.
+ :param ftp_hook: predefined FTPHook to use
+ Either `ftp_hook` or `ftp_conn_id` needs to be provided.
+ :param local_filepath: local file path to get or put. (templated)
+ :param remote_filepath: remote file path to get or put. (templated)
+ :param operation: specify operation 'get' or 'put', defaults to put
+ :param create_intermediate_dirs: create missing intermediate directories when
+ copying from remote to local and vice-versa. Default is False.
+
+ Example: The following task would copy ``file.txt`` to the remote host
+ at ``/tmp/tmp1/tmp2/`` while creating ``tmp``,``tmp1`` and ``tmp2`` if they
+ don't exist. If the parameter is not passed it would error as the directory
+ does not exist. ::
+
+ put_file = FTPOperator(
+ task_id="test_ftp",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/file.txt",
+ remote_filepath="/tmp/tmp1/tmp2/file.txt",
+ operation="put",
+ create_intermediate_dirs=True,
+ dag=dag
+ )
+
+ """
+
+ template_fields: Sequence[str] = ('local_filepath', 'remote_filepath')
+
+ def __init__(
+ self,
+ *,
+ ftp_hook: FTPHook | None = None,
+ ftp_conn_id: str | None = None,
+ local_filepath: str | list[str],
+ remote_filepath: str | list[str],
+ operation: str = FTPOperation.PUT,
+ create_intermediate_dirs: bool = False,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.ftp_hook = ftp_hook
+ self.ftp_conn_id = ftp_conn_id
+ self.operation = operation
+ self.create_intermediate_dirs = create_intermediate_dirs
+
+ self.local_filepath_was_str = False
+ if isinstance(local_filepath, str):
+ self.local_filepath = [local_filepath]
+ self.local_filepath_was_str = True
+ else:
+ self.local_filepath = local_filepath
+
+ if isinstance(remote_filepath, str):
+ self.remote_filepath = [remote_filepath]
+ else:
+ self.remote_filepath = remote_filepath
+
+ if len(self.local_filepath) != len(self.remote_filepath):
+ raise ValueError(
+ f'{len(self.local_filepath)} paths in local_filepath '
+ f'!= {len(self.remote_filepath)} paths in remote_filepath'
+ )
+
+ if not (self.operation.lower() == FTPOperation.GET or self.operation.lower() == FTPOperation.PUT):
Review Comment:
```suggestion
if self.operation.lower() not in [FTPOperation.GET, FTPOperation.PUT]:
```
##########
airflow/providers/ftp/operators/ftp.py:
##########
@@ -0,0 +1,146 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains FTP operator."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Any, Sequence
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.ftp.hooks.ftp import FTPHook
+
+
+class FTPOperation:
+ """Operation that can be used with FTP"""
+
+ PUT = 'put'
+ GET = 'get'
+
+
+class FTPOperator(BaseOperator):
+ """
+ FTPOperator for transferring files from remote host to local or vice a versa.
+ This operator uses ftp_hook to open ftp transport channel that serve as basis
+ for file transfer.
+
+ :param ftp_conn_id: :ref:`ftp connection id<howto/connection:ftp>`
+ from airflow Connections.
+ :param ftp_hook: predefined FTPHook to use
+ Either `ftp_hook` or `ftp_conn_id` needs to be provided.
+ :param local_filepath: local file path to get or put. (templated)
+ :param remote_filepath: remote file path to get or put. (templated)
+ :param operation: specify operation 'get' or 'put', defaults to put
+ :param create_intermediate_dirs: create missing intermediate directories when
+ copying from remote to local and vice-versa. Default is False.
+
+ Example: The following task would copy ``file.txt`` to the remote host
+ at ``/tmp/tmp1/tmp2/`` while creating ``tmp``,``tmp1`` and ``tmp2`` if they
+ don't exist. If the parameter is not passed it would error as the directory
+ does not exist. ::
+
+ put_file = FTPOperator(
+ task_id="test_ftp",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/file.txt",
+ remote_filepath="/tmp/tmp1/tmp2/file.txt",
+ operation="put",
+ create_intermediate_dirs=True,
+ dag=dag
+ )
+
+ """
+
+ template_fields: Sequence[str] = ('local_filepath', 'remote_filepath')
+
+ def __init__(
+ self,
+ *,
+ ftp_hook: FTPHook | None = None,
+ ftp_conn_id: str | None = None,
+ local_filepath: str | list[str],
+ remote_filepath: str | list[str],
+ operation: str = FTPOperation.PUT,
+ create_intermediate_dirs: bool = False,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.ftp_hook = ftp_hook
+ self.ftp_conn_id = ftp_conn_id
+ self.operation = operation
+ self.create_intermediate_dirs = create_intermediate_dirs
+
+ self.local_filepath_was_str = False
+ if isinstance(local_filepath, str):
+ self.local_filepath = [local_filepath]
+ self.local_filepath_was_str = True
+ else:
+ self.local_filepath = local_filepath
+
+ if isinstance(remote_filepath, str):
+ self.remote_filepath = [remote_filepath]
+ else:
+ self.remote_filepath = remote_filepath
+
+ if len(self.local_filepath) != len(self.remote_filepath):
+ raise ValueError(
+ f'{len(self.local_filepath)} paths in local_filepath '
+ f'!= {len(self.remote_filepath)} paths in remote_filepath'
+ )
+
+ if not (self.operation.lower() == FTPOperation.GET or self.operation.lower() == FTPOperation.PUT):
+ raise TypeError(
+ f"Unsupported operation value {self.operation}, "
+ f"expected {FTPOperation.GET} or {FTPOperation.PUT}."
+ )
+
+ def execute(self, context: Any) -> str | list[str] | None:
+ file_msg = None
+ try:
+ if self.ftp_conn_id and not self.ftp_hook:
+ self.log.info('ftp_hook not provided or invalid. Trying ftp_conn_id to create FTPHook.')
+ self.ftp_hook = FTPHook(ftp_conn_id=self.ftp_conn_id)
+
+ if self.ftp_hook and not isinstance(self.ftp_hook, FTPHook):
+ self.log.info('ftp_hook not provided or invalid. Trying ftp_conn_id to create FTPHook.')
+ self.ftp_hook = FTPHook(ftp_conn_id=self.ftp_conn_id)
+
+ if not self.ftp_hook:
+ raise AirflowException("Cannot operate without ftp_hook or ftp_conn_id.")
Review Comment:
Instead of all this I would created a method to get the hook using `self.ftp_conn_id`. I would also use @cached_property to get it only once
```suggestion
@cached_property
def hook(self) -> FTPHook:
"""Create and return an FTPHook."""
return FTPHook(ftp_conn_id=self.ftp_conn_id)
def execute(self, context: Any) -> list[str]:
file_msg = None
try:
# You can now use the hook by just using `self.hook`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r991485018
##########
scripts/docker/entrypoint_ci.sh:
##########
@@ -221,6 +221,12 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null
fi
+ sudo apt-get update && sudo apt-get install -y vsftpd
+ mkdir /ftphome
+ sudo chown airflow:airflow /ftphome
+ echo "write_enable=YES" >> /etc/vsftpd.conf
+ sudo service vsftpd restart
Review Comment:
Here I pull up the local FTP server for testing. I am not sure if this is the appropriate place to do this, but I did not find any other part of the codebase where this should be done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1020535745
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
That is a good point. Would a name like `FTPFileTransfer` make more sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1020491575
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
I think consistancy is important. Its very hard to work with Airflow when every provider sets its own unique behavior for the same action.
That said, If we accept it as is, at lease the class name should be more explanatory. It should contain some verb/action to better describe what it does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1340019976
> but I personaly do not see the need for consistency (and I consider the local_* a bit strange now when I think of it.
Consistency is important. Many users are using multiple providers lack of consistency affects development velocity as you always need to look up how specific provider/service name things. If all use same convention then you don't need to look it up.
If we are not happy with current convention then we can start the process of slowly changing. I have no strong feelings for either way I just want to keep things crystal clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
o-nikolas commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1285887214
> @o-nikolas Sorry about that. I have fixed those failures. I was wondering if I should first wait to get the workflow approved first before rebasing, as if I rebase now, my branch may quickly become out of date again?
You're code needs to at least be "rebase-able" before it's merged into main anyway, so doing it now or later doesn't matter too much (as long as main is stable).
You'll need a committer to approve the workflow (and merge if all tests are passing). CCing @eladkal and @kaxil who might be able to help if they have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
o-nikolas commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1290954178
> @o-nikolas Thank you for your response! I have rebased with main.
>
> @eladkal @kaxil When you have time, could you approve the workflow? Thank you!
CCing @potiuk as well, who might have time to approve the workflow.
You can also run static checks yourself locally, to ensure those at least pass, by enabling pre-commit checks ([readme](https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#installing-pre-commit-hooks)) or running
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1336652996
Yep. I woudl have rebased myself, but there are conflicts (so you need to rebase abway @RachitSharma2001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1025461324
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
Yeah. Would be great to get it to AIP-47 format already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1026899619
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
I added the example dag mainly as reference for documentation. However, I have not set up any system tests, as that would require setting up a local FTP server, which currently has not been added to the ci. Would it still be required to move the example dags to `tests/system/providers/ftp` and change it to AIP-47 even if it isn't meant to be a system test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1008920293
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
This should be a transfer operator.
See examples for LocalTo* operators.
Since operators dont share disk the source of temp folder doesnt seem right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r996471847
##########
airflow/providers/ftp/operators/ftp.py:
##########
@@ -0,0 +1,135 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains FTP operator."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Any, Sequence
+
+from airflow.compat.functools import cached_property
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.ftp.hooks.ftp import FTPHook
+
+
+class FTPOperation:
+ """Operation that can be used with FTP"""
+
+ PUT = 'put'
+ GET = 'get'
+
+
+class FTPOperator(BaseOperator):
+ """
+ FTPOperator for transferring files from remote host to local or vice a versa.
+ This operator uses an FTPHook to open ftp transport channel that serve as basis
+ for file transfer.
+
+ :param ftp_conn_id: :ref:`ftp connection id<howto/connection:ftp>`
+ from airflow Connections.
+ :param local_filepath: local file path to get or put. (templated)
+ :param remote_filepath: remote file path to get or put. (templated)
+ :param operation: specify operation 'get' or 'put', defaults to put
+ :param create_intermediate_dirs: create missing intermediate directories when
+ copying from remote to local and vice-versa. Default is False.
+
+ Example: The following task would copy ``file.txt`` to the remote host
+ at ``/tmp/tmp1/tmp2/`` while creating ``tmp``,``tmp1`` and ``tmp2`` if they
+ don't exist. If the parameter is not passed it would error as the directory
+ does not exist. ::
+
+ put_file = FTPOperator(
+ task_id="test_ftp",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/file.txt",
+ remote_filepath="/tmp/tmp1/tmp2/file.txt",
+ operation="put",
+ create_intermediate_dirs=True,
+ dag=dag
+ )
+
+ """
+
+ template_fields: Sequence[str] = ('local_filepath', 'remote_filepath')
+
+ def __init__(
+ self,
+ *,
+ ftp_conn_id: str,
Review Comment:
Good idea, thank you!
##########
airflow/providers/ftp/operators/ftp.py:
##########
@@ -0,0 +1,135 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains FTP operator."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Any, Sequence
+
+from airflow.compat.functools import cached_property
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.ftp.hooks.ftp import FTPHook
+
+
+class FTPOperation:
+ """Operation that can be used with FTP"""
+
+ PUT = 'put'
+ GET = 'get'
+
+
+class FTPOperator(BaseOperator):
+ """
+ FTPOperator for transferring files from remote host to local or vice a versa.
+ This operator uses an FTPHook to open ftp transport channel that serve as basis
+ for file transfer.
+
+ :param ftp_conn_id: :ref:`ftp connection id<howto/connection:ftp>`
+ from airflow Connections.
+ :param local_filepath: local file path to get or put. (templated)
+ :param remote_filepath: remote file path to get or put. (templated)
+ :param operation: specify operation 'get' or 'put', defaults to put
+ :param create_intermediate_dirs: create missing intermediate directories when
+ copying from remote to local and vice-versa. Default is False.
+
+ Example: The following task would copy ``file.txt`` to the remote host
+ at ``/tmp/tmp1/tmp2/`` while creating ``tmp``,``tmp1`` and ``tmp2`` if they
+ don't exist. If the parameter is not passed it would error as the directory
+ does not exist. ::
+
+ put_file = FTPOperator(
+ task_id="test_ftp",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/file.txt",
+ remote_filepath="/tmp/tmp1/tmp2/file.txt",
+ operation="put",
+ create_intermediate_dirs=True,
+ dag=dag
+ )
+
+ """
+
+ template_fields: Sequence[str] = ('local_filepath', 'remote_filepath')
+
+ def __init__(
+ self,
+ *,
+ ftp_conn_id: str,
+ local_filepath: str | list[str],
+ remote_filepath: str | list[str],
+ operation: str = FTPOperation.PUT,
+ create_intermediate_dirs: bool = False,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.ftp_conn_id = ftp_conn_id
+ self.operation = operation
+ self.create_intermediate_dirs = create_intermediate_dirs
+
+ self.local_filepath_was_str = False
Review Comment:
Good catch, sorry about that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1279504873
@vincbeck Thank you for your suggestions. I have integrated the suggested changes. As for documentation, do you mean that I should add some operator file to `docs/apache-airflow-providers-ftp` that outlines how to use the FTPOperator, in a similar way to the Tableau Operator from [here](https://github.com/RachitSharma2001/airflow/blob/main/docs/apache-airflow-providers-tableau/operators.rst)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r992575427
##########
scripts/docker/entrypoint_ci.sh:
##########
@@ -221,6 +221,12 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null
fi
+ sudo apt-get update && sudo apt-get install -y vsftpd
+ mkdir /ftphome
+ sudo chown airflow:airflow /ftphome
+ echo "write_enable=YES" >> /etc/vsftpd.conf
+ sudo service vsftpd restart
Review Comment:
I see, but I saw from the SFTPOperator tests that they also pull up an local ssh server and use it directly for testing as opposed to mocking the server (tests [here](https://github.com/apache/airflow/blob/main/tests/providers/sftp/operators/test_sftp.py)). I thought that, for consistency and accuracy purposes, it would make sense to do the same for the FTPOperator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r993718498
##########
airflow/utils/db.py:
##########
@@ -308,6 +308,18 @@ def create_default_connections(session: Session = NEW_SESSION):
),
session,
)
+ merge_conn(
+ Connection(
+ conn_id="ftp_default",
+ conn_type="ftp",
+ host="localhost",
Review Comment:
Even though the unit tests no longer use a local FTP Server, I still think that having this FTP Default connection would be useful, as without this, defining the following piece of code will throw an error indicating that the "connection with id 'ftp_default' does not exist": `hook = FTPHook()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r993238368
##########
scripts/docker/entrypoint_ci.sh:
##########
@@ -221,6 +221,12 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null
fi
+ sudo apt-get update && sudo apt-get install -y vsftpd
+ mkdir /ftphome
+ sudo chown airflow:airflow /ftphome
+ echo "write_enable=YES" >> /etc/vsftpd.conf
+ sudo service vsftpd restart
Review Comment:
I’d argue there’s a limit on being thorough and the SFTP tests are probably too much trouble to be worthwhile. They are already there so there’s less reason to rewrite them, but it’s better for new code to be tested in a more lightweight fashion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1020802540
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
I have updated the operator to reflect this name change to `FTPFileTransferOperator`. Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1020491575
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
I think consistancy is important. Its very hard to work with Airflow when every provider sets its own unique behavior for the same action.
That said, If we accept it as is, at least the class name should be more explanatory. It should contain some verb/action to better describe what it does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1009812933
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
Do you mean that the `local_filepath` should not be set to `/tmp/filepath`? I was looking at the example given for SFTP put [here](https://airflow.apache.org/docs/apache-airflow-providers-sftp/stable/_api/airflow/providers/sftp/operators/sftp/index.html), and it seems that they put a similar path. Do you mean that the `local_filepath` should instead be something like `"relative/path/to/file"` like from [here](https://airflow.apache.org/docs/apache-airflow-providers-amazon/3.3.0/operators/transfer/local_to_s3.html)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r993713609
##########
scripts/docker/entrypoint_ci.sh:
##########
@@ -221,6 +221,12 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null
fi
+ sudo apt-get update && sudo apt-get install -y vsftpd
+ mkdir /ftphome
+ sudo chown airflow:airflow /ftphome
+ echo "write_enable=YES" >> /etc/vsftpd.conf
+ sudo service vsftpd restart
Review Comment:
That make sense. I have replaced the tests with more lightweight unit tests (see [here](https://github.com/RachitSharma2001/airflow/blob/main/tests/providers/ftp/operators/test_ftp.py)). I modeled these tests after [test_sftp](https://github.com/apache/airflow/blob/main/tests/providers/sftp/operators/test_sftp.py), with the difference being that the FTP server is mocked. Let me know if these tests make sense, and if there are any other changes that should be made.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
o-nikolas commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1281165460
> @o-nikolas Thank you for your suggestions. I have made the requested changes. Specifically, I have changed [ftp.py](https://github.com/RachitSharma2001/airflow/blob/main/airflow/providers/ftp/operators/ftp.py), as well as the FTPOperator docs ([here](https://github.com/RachitSharma2001/airflow/blob/main/docs/apache-airflow-providers-ftp/operators/index.rst#id1)). I have also added [example dags](https://github.com/RachitSharma2001/airflow/tree/main/airflow/providers/ftp/example_dags) which use the FTP Operator. Let me know if there is any other changes you think should be added.
Great thanks! It looks like your PR is failing for the couple issues that are in main right now. You'll have to rebase your PR once those are fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r996470106
##########
airflow/providers/ftp/operators/ftp.py:
##########
@@ -0,0 +1,135 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains FTP operator."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Any, Sequence
+
+from airflow.compat.functools import cached_property
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.ftp.hooks.ftp import FTPHook
+
+
+class FTPOperation:
+ """Operation that can be used with FTP"""
+
+ PUT = 'put'
+ GET = 'get'
+
+
+class FTPOperator(BaseOperator):
+ """
+ FTPOperator for transferring files from remote host to local or vice a versa.
+ This operator uses an FTPHook to open ftp transport channel that serve as basis
+ for file transfer.
+
+ :param ftp_conn_id: :ref:`ftp connection id<howto/connection:ftp>`
+ from airflow Connections.
+ :param local_filepath: local file path to get or put. (templated)
+ :param remote_filepath: remote file path to get or put. (templated)
+ :param operation: specify operation 'get' or 'put', defaults to put
+ :param create_intermediate_dirs: create missing intermediate directories when
+ copying from remote to local and vice-versa. Default is False.
+
+ Example: The following task would copy ``file.txt`` to the remote host
+ at ``/tmp/tmp1/tmp2/`` while creating ``tmp``,``tmp1`` and ``tmp2`` if they
+ don't exist. If the parameter is not passed it would error as the directory
+ does not exist. ::
+
+ put_file = FTPOperator(
+ task_id="test_ftp",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/file.txt",
+ remote_filepath="/tmp/tmp1/tmp2/file.txt",
+ operation="put",
+ create_intermediate_dirs=True,
+ dag=dag
+ )
+
+ """
+
+ template_fields: Sequence[str] = ('local_filepath', 'remote_filepath')
+
+ def __init__(
+ self,
+ *,
+ ftp_conn_id: str,
+ local_filepath: str | list[str],
+ remote_filepath: str | list[str],
+ operation: str = FTPOperation.PUT,
+ create_intermediate_dirs: bool = False,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.ftp_conn_id = ftp_conn_id
+ self.operation = operation
+ self.create_intermediate_dirs = create_intermediate_dirs
+
+ self.local_filepath_was_str = False
+ if isinstance(local_filepath, str):
+ self.local_filepath = [local_filepath]
+ else:
+ self.local_filepath = local_filepath
+
+ if isinstance(remote_filepath, str):
+ self.remote_filepath = [remote_filepath]
+ else:
+ self.remote_filepath = remote_filepath
+
+ if len(self.local_filepath) != len(self.remote_filepath):
+ raise ValueError(
+ f'{len(self.local_filepath)} paths in local_filepath '
+ f'!= {len(self.remote_filepath)} paths in remote_filepath'
+ )
+
+ if self.operation.lower() not in [FTPOperation.GET, FTPOperation.PUT]:
+ raise TypeError(
+ f"Unsupported operation value {self.operation}, "
+ f"expected {FTPOperation.GET} or {FTPOperation.PUT}."
+ )
+
+ @cached_property
+ def hook(self) -> FTPHook:
+ """Create and return an FTPHook."""
+ return FTPHook(ftp_conn_id=self.ftp_conn_id)
+
+ def execute(self, context: Any) -> str | list[str] | None:
+ file_msg = None
+ try:
+ for local_filepath, remote_filepath in zip(self.local_filepath, self.remote_filepath):
+ if self.operation.lower() == FTPOperation.GET:
+ local_folder = os.path.dirname(local_filepath)
+ if self.create_intermediate_dirs:
+ Path(local_folder).mkdir(parents=True, exist_ok=True)
+ file_msg = f"from {remote_filepath} to {local_filepath}"
+ self.log.info("Starting to transfer %s", file_msg)
+ self.hook.retrieve_file(remote_filepath, local_filepath)
+ else:
+ remote_folder = os.path.dirname(remote_filepath)
+ if self.create_intermediate_dirs:
+ self.hook.create_directory(remote_folder)
+ file_msg = f"from {local_filepath} to {remote_filepath}"
+ self.log.info("Starting to transfer file %s", file_msg)
+ self.hook.store_file(remote_filepath, local_filepath)
+ except Exception as e:
+ raise AirflowException(f"Error while transferring {file_msg}, error: {str(e)}")
Review Comment:
That is a good point. I think I originally just wanted to add context for the reader by telling them that the error occurred during the `"transfer of {file_msg}"`. But I think this context is unnecessary as the function already logs the `file_msg` before the calls to `self.hook.store_file` or `self.hook.retrieve_file`. So now I just remove the `try-except`, as it is unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1340021520
Awesome work, congrats on your first merged pull request!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r993713609
##########
scripts/docker/entrypoint_ci.sh:
##########
@@ -221,6 +221,12 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null
fi
+ sudo apt-get update && sudo apt-get install -y vsftpd
+ mkdir /ftphome
+ sudo chown airflow:airflow /ftphome
+ echo "write_enable=YES" >> /etc/vsftpd.conf
+ sudo service vsftpd restart
Review Comment:
That make sense. I have replaced the tests with more lightweight unit tests (see [here](https://github.com/RachitSharma2001/airflow/blob/main/tests/providers/ftp/operators/test_ftp.py)). Let me know if these tests make sense, and if there are any other changes that should be made.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r991788170
##########
scripts/docker/entrypoint_ci.sh:
##########
@@ -221,6 +221,12 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null
fi
+ sudo apt-get update && sudo apt-get install -y vsftpd
+ mkdir /ftphome
+ sudo chown airflow:airflow /ftphome
+ echo "write_enable=YES" >> /etc/vsftpd.conf
+ sudo service vsftpd restart
Review Comment:
I’d prefer simply mocking `ftplib` for testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1280024401
@o-nikolas Thank you for your suggestions. I have made the requested changes. Specifically, I have changed [ftp.py](https://github.com/RachitSharma2001/airflow/blob/main/airflow/providers/ftp/operators/ftp.py), as well as the FTPOperator docs ([here](https://github.com/RachitSharma2001/airflow/blob/main/docs/apache-airflow-providers-ftp/operators/index.rst#id1)). I have also added [example dags](https://github.com/RachitSharma2001/airflow/tree/main/airflow/providers/ftp/example_dags) which use the FTP Operator. Let me know if there is any other changes you think should be added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
vincbeck commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1279521148
Exactly :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1015875274
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
I was wondering if it would be more appropriate to define the FTPOperator similarly to SFTPOperator, rather than having it as a specific transfer operator?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1020741355
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
Sounds ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #26974: Add FTP Operator
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1020479132
##########
airflow/providers/ftp/example_dags/example_ftp.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.ftp.operators.ftp import FTPOperation, FTPOperator
+
+with models.DAG(
+ "example_ftp_put_get",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_ftp_put]
+ ftp_put = FTPOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
Review Comment:
I think we have two different approaches for that. And we can decide either way. I personally think the "Local" is a bit artifficial for transfer. Having to define connection to download/upload files is not necessarily what you woul dlike ot do.
I think we are not currently in the spree of cleaning and unifying this, so I would be ok with merging it as is.
WDYT @eladkal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on a diff in pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
o-nikolas commented on code in PR #26974:
URL: https://github.com/apache/airflow/pull/26974#discussion_r1038436706
##########
tests/system/providers/ftp/example_ftp.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using the FTPFileTransmitOperator.
+"""
+from __future__ import annotations
+
+import os
+from datetime import datetime
+
+from airflow import DAG
+from airflow.providers.ftp.operators.ftp import FTPFileTransmitOperator, FTPOperation
+
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = "example_ftp_put_get"
+
+with DAG(
+ DAG_ID,
+ schedule_interval="@once",
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=["example", "Ftp", "FtpFileTransmit"],
+) as dag:
+ # [START howto_operator_ftp_put]
+ ftp_put = FTPFileTransmitOperator(
+ task_id="test_ftp_put",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
+ remote_filepath="/remote_tmp/filepath",
+ operation=FTPOperation.PUT,
+ create_intermediate_dirs=True,
+ )
+ # [END howto_operator_ftp_put]
+
+ # [START howto_operator_ftp_get]
+ ftp_get = FTPFileTransmitOperator(
+ task_id="test_ftp_get",
+ ftp_conn_id="ftp_default",
+ local_filepath="/tmp/filepath",
+ remote_filepath="/remote_tmp/filepath",
+ operation=FTPOperation.GET,
+ create_intermediate_dirs=True,
+ )
+ # [END howto_operator_ftp_get]
+
Review Comment:
I know you don't have any teardown tasks here, but I'd prefer that you still add the watcher task and a task chain, just to match other AIP-47 spec tests and so that future additions are more straightforward
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
RachitSharma2001 commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1337799402
@potiuk @o-nikolas I have added the requested changes and updated my branch. Let me know if the current changes look good and if there is anything else I should add.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on pull request #26974: Add `FTPFileTransmitOperator`
Posted by GitBox <gi...@apache.org>.
o-nikolas commented on PR #26974:
URL: https://github.com/apache/airflow/pull/26974#issuecomment-1335680099
Some checks also failed, do you mind re-running them @RachitSharma2001 (rebasing your branch will do it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org