You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/19 22:13:43 UTC
[GitHub] [airflow] ephraimbuddy opened a new pull request #13190: Add system test for LocalToAzureDataLakeStorageOperator.
ephraimbuddy opened a new pull request #13190:
URL: https://github.com/apache/airflow/pull/13190
This PR adds system test for LocalToAzureDataLakeStorageOperator.
To test this, create an azure data lake and provide the following connection attributes in JSON file named `azure_data_lake.json` inside `/files/airflow-breeze-config/keys`. The JSON should be like below:
{
"login": "<Client Id>",
"password": "Client secret",
"extra": {"account_name": "account-name", "tenant": "tenant"}
}
RUN: ` pytest tests/providers/microsoft/azure/transfers/test_local_to_adls_system.py --system azure -s`
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj merged pull request #13190: Add system test for LocalToAzureDataLakeStorageOperator.
Posted by GitBox <gi...@apache.org>.
mik-laj merged pull request #13190:
URL: https://github.com/apache/airflow/pull/13190
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #13190: Add system test for LocalToAzureDataLakeStorageOperator.
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #13190:
URL: https://github.com/apache/airflow/pull/13190#discussion_r546325124
##########
File path: airflow/providers/microsoft/azure/example_dags/example_local_to_adls.py
##########
@@ -18,11 +18,19 @@
import os
from airflow import models
+from airflow.operators.python import PythonOperator
+from airflow.providers.microsoft.azure.hooks.azure_data_lake import AzureDataLakeHook
from airflow.providers.microsoft.azure.transfers.local_to_adls import LocalToAzureDataLakeStorageOperator
from airflow.utils.dates import days_ago
LOCAL_FILE_PATH = os.environ.get("LOCAL_FILE_PATH", 'localfile.txt')
-REMOTE_FILE_PATH = os.environ.get("REMOTE_LOCAL_PATH", 'remote')
+REMOTE_FILE_PATH = os.environ.get("REMOTE_LOCAL_PATH", 'remote.txt')
+
+
+def remove_file():
Review comment:
Ok. I will add it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13190: Add system test for LocalToAzureDataLakeStorageOperator.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13190:
URL: https://github.com/apache/airflow/pull/13190#issuecomment-751303490
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest master or amend the last commit of the PR, and push it with --force-with-lease.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #13190: Add system test for LocalToAzureDataLakeStorageOperator.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #13190:
URL: https://github.com/apache/airflow/pull/13190#discussion_r546304126
##########
File path: airflow/providers/microsoft/azure/example_dags/example_local_to_adls.py
##########
@@ -18,11 +18,19 @@
import os
from airflow import models
+from airflow.operators.python import PythonOperator
+from airflow.providers.microsoft.azure.hooks.azure_data_lake import AzureDataLakeHook
from airflow.providers.microsoft.azure.transfers.local_to_adls import LocalToAzureDataLakeStorageOperator
from airflow.utils.dates import days_ago
LOCAL_FILE_PATH = os.environ.get("LOCAL_FILE_PATH", 'localfile.txt')
-REMOTE_FILE_PATH = os.environ.get("REMOTE_LOCAL_PATH", 'remote')
+REMOTE_FILE_PATH = os.environ.get("REMOTE_LOCAL_PATH", 'remote.txt')
+
+
+def remove_file():
Review comment:
What do you think to add the operator that will do this action? I think it would be good if the example DAGs did not include tricks with calling the Hook directly, but if possible they would use operators.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #13190: Add system test for LocalToAzureDataLakeStorageOperator.
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #13190:
URL: https://github.com/apache/airflow/pull/13190#issuecomment-751290289
Hi @mik-laj , please can you take a look once more
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #13190: Add system test for LocalToAzureDataLakeStorageOperator.
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #13190:
URL: https://github.com/apache/airflow/pull/13190#issuecomment-748804016
Depends on https://github.com/apache/airflow/pull/13206
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org