You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/31 08:20:24 UTC
[GitHub] [airflow] pyerbiz opened a new pull request #17355: [11912] [skip ci] Add how-to Guide for Azure operators
pyerbiz opened a new pull request #17355:
URL: https://github.com/apache/airflow/pull/17355
Adding documentation and example dags for missing cases in Azure Providers.
closes: #11912
related:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pyerbiz commented on pull request #17355: [11912] [skip ci] Add how-to Guide for Azure operators
Posted by GitBox <gi...@apache.org>.
pyerbiz commented on pull request #17355:
URL: https://github.com/apache/airflow/pull/17355#issuecomment-922596307
@kaxil Not right now. I'll try to this weekend and seek help if I need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pyerbiz commented on pull request #17355: [11912] [skip ci] Add how-to Guide for Azure operators
Posted by GitBox <gi...@apache.org>.
pyerbiz commented on pull request #17355:
URL: https://github.com/apache/airflow/pull/17355#issuecomment-927353297
@kaxil I had a question about the example_dag for [adx.py](https://github.com/apache/airflow/blob/main/airflow/providers/microsoft/azure/operators/adx.py).
I am using gcp's big query docs for reference and also the postgres operator doc. Based on them, to make a complete example I should be able to create table, load data, query data, and drop table. I have to tried to do that in kusto above. I am concerned about the connection_string_id used to connect to ADX cluster. When I access adx with python in my local machine, I have to do the browser pop-up method, credential based login -- other methods have caused me problems in the past. I'm going to try using the AAD token method again and see if that works without pop-ups. Is there a preferred way users do adx authentication on airflow? I ask because the other example_dag files for azure all use a default "default_connection_id" string to show the example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pyerbiz commented on pull request #17355: [11912] [skip ci] Add how-to Guide for Azure operators
Posted by GitBox <gi...@apache.org>.
pyerbiz commented on pull request #17355:
URL: https://github.com/apache/airflow/pull/17355#issuecomment-913381738
I am back from a work and travel hiatus. If still acceptable, maintainers may re-assign this issue to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #17355: [11912] [skip ci] Add how-to Guide for Azure operators
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #17355:
URL: https://github.com/apache/airflow/pull/17355#discussion_r680747378
##########
File path: airflow/providers/microsoft/azure/example_dags/example_adx_query.py
##########
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+
+from airflow import models
+from airflow.providers.microsoft.azure.operators.adx import AzureDataExplorerQueryOperator
+from airflow.utils.dates import days_ago
+
+LOCAL_FILE_PATH = os.environ.get("LOCAL_FILE_PATH", 'localfile.txt') #1
+REMOTE_FILE_PATH = os.environ.get("REMOTE_LOCAL_PATH", 'remote.txt') #2
+
+# Operator for querying Azure Data Explorer (Kusto).
+
+# :param query: KQL query to run (templated).
+# :type query: str
+# :param database: Database to run the query on (templated).
+# :type database: str
+# :param options: Optional query options. See:
+# https://docs.microsoft.com/en-us/azure/kusto/api/netfx/request-properties#list-of-clientrequestproperties
+# :type options: dict
+# :param azure_data_explorer_conn_id: Reference to the
+# :ref:`Azure Data Explorer connection<howto/connection:adx>`.
+# :type azure_data_explorer_conn_id: str
Review comment:
Maintaining the above parameters here would be problematic. For example, if there's an addition to the params, we may forget to update the param list here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #17355: [11912] [skip ci] Add how-to Guide for Azure operators
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #17355:
URL: https://github.com/apache/airflow/pull/17355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #17355: [11912] [skip ci] Add how-to Guide for Azure operators
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #17355:
URL: https://github.com/apache/airflow/pull/17355#issuecomment-913389332
You’re still assigned in the issue, please feel free to continue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #17355: [11912] [skip ci] Add how-to Guide for Azure operators
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #17355:
URL: https://github.com/apache/airflow/pull/17355#issuecomment-922569632
@pyerbiz Do you need any help with this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org